It’s still up to the user-agent to respect robots.txt (or not). And given the ethical track record of AI companies, do we really expect them to heed any limitations specified in the file?
16 Likes
It’s still up to the user-agent to respect robots.txt (or not). And given the ethical track record of AI companies, do we really expect them to heed any limitations specified in the file?
