I've got a calendar site that's used by <50 people. 99% of my requests look like... | Hacker News

Hacker Timesnew | past | comments | ask | show | jobs | submit

		bryanlarsen on Oct 18, 2013 \| parent \| context \| favorite \| on: Who exactly is crawling my site? I've got a calendar site that's used by <50 people. 99% of my requests look like: "GET /calls?month=2&year=7206 HTTP/1.1" 200 2423 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" It wouldn't be hard to stop (404 nonsensical years & disable the previous / next links that will get them there), but it is amusing that Googlebot has clicked on that "next month" link over 60000 times...

RKearney on Oct 18, 2013 | [–]

I would think implementing something like this[0] would help.

[0] https://support.google.com/webmasters/answer/96569?hl=en

kolinko on Oct 19, 2013 | [–]

aside from nofollow, that someone mentioned, you might want to implement http://en.wikipedia.org/wiki/Canonical_link_element or put a block on year= into the robots.txt file

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact