Is it impolite or rude if you refer to someone as "young madam"?

Is it improper grammar or wrong because a flight attendant once referred to me as young madam and I always wondered if he was trying to be nice or trying to be respectful of my age. Also, how do you know when to use the term "madam" when you don't know the person's age or if they are married? Or should you always use the term "miss"?