Earlier this year Reddit announced it was making changes to its API that prohibited any scraping of its data without payment.
Earlier this year Reddit announced it was making changes to its API that prohibited any scraping of its data without payment. The change resulted in an uproar on the social media platform and then total darkness as subreddits were abandoned to stand in solidarity with third-party app designers who could no longer access Reddit's data unless they paid for it.
Reddit took the same stance against third-party apps with search engines and any other company that was looking to acquire its wealth of data to improve proprietary products. Reports then surfaced about a $60 million annual deal being struck with Google for direct access to Reddit data, along with a similar deal being struck with OpenAI for Reddit posts to appear in responses by the company's chatbots such as ChatGPT.
But now the boot has finally come down as a new report from The Verge reveals Reddit has rolled out an update that prohibits all web crawling, or data gathering, on its website. The change has banned search engines such as Microsoft's Bing, DuckDuckGo, and others from crawling Reddit, meaning Reddit posts will no longer appear as search results within the now-banned search engines.
Microsoft has confirmed the change in a recent statement posted in Search Engine Land.
"We respect the robots.txt standard. Bing stopped crawling Reddit after they implemented their updated robots.txt file on July 1, which prohibits all crawling of their site."
The only search engine that is allowed to display Reddit content is Google as it pays $60 million a year for it.