As Rob says, some crawlers are necessary for search engine indexing. We need a better version of robots.txt to exclude unauthorised LLM crawlers.
13 Likes
As Rob says, some crawlers are necessary for search engine indexing. We need a better version of robots.txt to exclude unauthorised LLM crawlers.
