I have the exact same problems when playing with others, or hearing a song in a concert which has been transposed. I freak out when someone thinks he's just about right with the key, but is way off chart.

I also recognise notes in relative to C though, so it might just be really good relative pitch