Hacker Timesnew | past | comments | ask | show | jobs | submitlogin

Why do people want to block LLM crawlers? Everyone wants to be visible in GPT, Claude, and others, don’t they?


Exactly! With many people moving to AI-driven search, if you are not in their indexes you traffic will reduce even further.

And I say this as someone who built a search engine with no AI: I know my audience for that service is very niche, with the vast majority of people using AI search because it's more convenient.


If someone doesn't want to visit my website then why would I provide my website to them


No


One issue is just the sheer amount of such traffic.


You're a funny guy.


- If LLM knows about your content, people don't really need to visit your site - LLM crawlers can be pretty aggressive and eat up a lot of traffic - Google will not suggest its misleading "summaries" from its search for your site - Some people just hate LLMs that much ¯\_(ツ)_/¯




Consider applying for YC's Fall 2026 batch! Applications are open till July 27.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: