Hacker Times
new
|
past
|
comments
|
ask
|
show
|
jobs
|
submit
login
webhat
on Oct 19, 2013
|
parent
|
context
|
favorite
| on:
Who exactly is crawling my site?
The first thing I did after reading this was look at
http://danbirken.com/robots.txt
I guess he though of that.
# (This is not the SEO site in question) User-Agent: * Allow: /
Guidelines
|
FAQ
|
Lists
|
API
|
Security
|
Legal
|
Apply to YC
|
Contact
Search:
