In the coming weeks, Reddit will begin blocking access to its public data by most automated bots. You’ll need to enter into a licensing agreement, as Google and OpenAI have done, to use Reddit content for model training and other commercial purposes.
Although this has technically summer Reddit’s policy is already in effect, now the company is enforcing it in update your robots.txt filea core part of the web that dictates how web crawlers are allowed to access a site. “It’s a signal to those who disagree with us that they shouldn’t be accessing Reddit’s data,” the company’s general counsel said. Ben LeeI said. “It’s also a signal to bad actors that the word ‘allow’ in robots.txt does not, and never has, meant that they can use the data however they want.”
