Reddit reportedly plans to block Google's search crawler

Originally published at: Reddit reportedly plans to block Google's search crawler | Boing Boing


Reddit does have the option of picking the level where it blocks crawlers. i.e. leave the tops of the sub-Reddits, but block anything below that…


Yeah I don’t see how that can work for their bottom line. Advertisers would kick their ass eventually.

Plus a few spiteful coders could probably whip up something in like a week to get around that and keep scraping


This move seems designed to eliminate people like me… I have a reddit account but I rarely post or comment and 99% of my interactions with the site are via google search results. If they block google spiders I’ll probably never visit the site, not out of malice or protest, simply because I’ll never get directed there.


It would be a big self-own, especially since their internal search functionality isn’t great.


If it helps kill Google it will be a good thing.

That shit is so over.


Yeah, and next month we’ll bankrupt IKEA by not giving back the little pencils at the checkout.


Google hasn’t managed to kill Google. I doubt this will do anything.

But it might do a serious number on Reddit.


Serious question, what are you using instead?


Not much to love about google these days, but at least it still gives you the odd link outside walled gardens. I see no reason to cheer those putting their walls that much higher.


Have you considered not limiting your web to where google directs you?


Between the recent API debacle, the continuing enshittification of the web site, and now this, it seems like Reddit is determined to follow the Twitter model of self-destruction.

Web search as a whole is broken these days. It’s almost always SEO crap, click farms, and AI generated junk that gets priority. Search engine capability is even getting worse. I can put a literal string in quotes and decorate it all kinds of ways, and in the past I’d either get results or nothing. These days search engines happily ignore my optimizations and return whatever. (Bing is far worse about this than Google though.) Scoping to Reddit has long been a great way to sift through the noise and find what real people are saying.

Reddit’s brass should know this, and if they can’t figure out how to monetize the huge amount of search traffic that ends up there, then that doesn’t really speak very highly of their competence. Instead they want to sell NFTs for avatar customization, block third party integrations, bombard you with ads in an effort to make you sign up for a subscription, and take away things like Reddit coins because they couldn’t figure out how to monetize that properly either.

Reddit is a wonderful place (yes, it has many dark corners but it’s very easy to avoid). There’s some great communities there where people love to be good humans and help one another. Even a lot of highly niche places are very lively. On any one visit I can get shit post memes about The Beach Boys, discuss the Yakuza series of games, read people’s deepest and darkest confessions, get interesting perspectives on if someone was an asshole or not, help identify obscure objects/cars/bugs/plants, get Sailor Moon themed pro-communist memes, help people with programming problems, chat about my favorite bands, and so many other things all in one place.

Yet, they seem determined to fuck it all up.


A week? Hahahahahaha

If the crawling code doesn’t already exist, take existing code and remove the robots.txt check. We’re talking less than 5 minutes.


That’s being generous. Reddit search is notoriously awful, breaking in bizarre ways if you try to tailor your results, and not bringing back things that match your query. Better search has been the #1 requested feature from the users for a decade, but it’s still as bad as ever.


Mostly serious question, can Reddit turn into a non-profit? Its function is so important to a well functioning internet, it is very hard to make profitable, and it continues to enshitify. I don’t know if or how this would work, but somehow wikipedia seems to thrive with this model. Once it stops trying to grow and make a profit, reddit can focus on just being a good space for its users.

Is it? I’ve quit it about a decade ago over all the toxicity and have done perfectly well since.

(For the record, before that, I was a mod of a very big and notoriously well-moderated subreddit, so it’s not like I don’t know what I’m missing)


I’m not a heavy user, but when I have questions about the city I live in, obscure movies and books, traveling, gardening, etc I can usually find something that I need (even if it doesn’t 100% answer my question). I usually stay away from the large subreddits, unless someone specifically shares a link with me. So, I never go there just to hang out. I’m only there when I have a purpose. In that way, I’m sure I miss all of the toxicity.


Can you recommend some entry points?

(Lemmy has a toxic developer problem.)


They’re in the middle of trying to IPO, so I’m going to go with “no”