There is definitely some graphical identification. You should try Google+ image search (if you have any images on there), it's really incredible. I searched "water" on my friend's images and got pictures of water glasses, the ocean, etc. None of the pictures had comments or metadata. Also worked searching for things like "soccer", got a bunch of pictures of him playing soccer.
edit: I'm sure some of theirs is from metadata, but I thought I read a while back that they were doing some graphical identification also.