"It turns out I'm not alone in adding these types of restrictions. Yelp blocks everybody but Google, Bing, ia_archiver (archive.org), ScoutJet (Blekko) and Yandex. LinkedIn also has a similar opt-in robots.txt, though they have whitelisted a larger number of bots than yelp."
At least we can contact/email Yelp and LinkedIn regarding to the crawlers if one can crawl or not according to their robots.txt. It's more generous than just allowing the big search engines such as Google and Bing. I'm not quite sure what's actually happening if we ask them though. I'll try that.
At least we can contact/email Yelp and LinkedIn regarding to the crawlers if one can crawl or not according to their robots.txt. It's more generous than just allowing the big search engines such as Google and Bing. I'm not quite sure what's actually happening if we ask them though. I'll try that.