Why Is Saying “Ma’am” Offensive?

Growing up in Virginia, my parents always made sure that I always said "yes ma'am, no ma'am, yes sir, no sir".  It was a given, it was a must.  So, why is it that when I say "ma'am" today, it's considered offensive?

I remember when I first moved to New Jersey.  I walked into a grocery store in Collingswood, and a gentleman held the door for me.  My immediate response was:  "Thank You Sir".

He turned around right away, with a look colder then the Phillies off-season.  With a firm voice he said, "Don't call me sir---ever".  I immediately apologized and he repeated the word "ever" to me, just in case I was too deaf from listening to Kid N Play at the highest volume.

It was my first culture shock in South Jersey.  What do you think?  Is saying the word "Ma'am" or "Sir" offensive?