Internet Archive to ignore robots.txt directives

Ah, thanks.
I tend to ignore the Wayback Machine’s web-crawling aspect, using in cases of Someone being Wrong on the Internet to record the craziness of the scam or whatever it is, when there is a likelihood that the culprit will later scrub the page. For that purpose, I am disappointed by the Archive and its adherence to the no-robots tag even for manual archiving.

1 Like

I don’t know if it will go that far, I think people will still care about privacy. But they will learn that once you’ve published something, it’s public, and you can’t put that genie back in the bottle. And people will reach a point where we’ve all published something we shouldn’t have or that we no longer agree with, and we’ll learn not judge each other so harshly based on what we wrote in our livejournal/tumblr/whatever during our angsty teen years.

If anything, that seems to be the direction - things that were taboo a couple of generations ago, younger people are more than willing to talk about openly and accept in each other.

1 Like

or we get data standards with inherent (configurable) expiration dates.

This topic was automatically closed after 5 days. New replies are no longer allowed.