AI crawlers run up a website's hosting bill

No, Hundreds of sites put current Anthropic scrapers on their blocklist, and then Anthropic changed their agent string. :roll_eyes:

It would be interesting to survey major sites’ robots.txt to see if there’s a particular blocking threshold that triggers Anthropic’s morphs.

6 Likes