Cloudflare-enabled websites are blocking RSS feed generation
Certain website owners have enabled Cloudflare on their website to prevent abuse by malicious bots. But Open RSS is being blocked as well. As a result, the Open RSS feeds that rely on having access to the website's content can't be populated.
Cloudflare blocks Open RSS from accessing the website with a page that requires human interaction to bypass. And because Open RSS is a service—not a human that can interact with the page—it gets blocked.
-
It appears that Cloudflare has a Verified Bots policy. We're researching this policy to ensure we're compliant, which we suspect may resolve this issue for a number of feeds.
-
Submitted information to Cloudflare for Open RSS to be added to its Verified Bots program.
-
Re-submitted information to Cloudflare for Open RSS to be added to its Verified Bots program.
-
Cloudflare gives website owners the option to enable bot fighting modes along with a Block AI Scrapers and Crawlers option. Enabling the features allows Cloudflare to use some algorithm to detect and block RSS readers and services from accessing the website to obtain RSS feed content. Affected sites likely have either of these options enabled, which is causing Open RSS to be blocked from populating RSS feeds.
Blocking could also be triggered by these sites that have JavaScript bot detection, which is enabled by default for all Cloudflare users.
Once blocked, Cloudflare appears to add a cookie to persist the block indefinitely.
-
Discovered Cloudflare gives each bot visit a bot score, and if it is lower than their threshold they expect, it triggers the blocking game. We suspect whatever bot score Cloudflare has given Open RSS may be contributing to the likelihood of being blocked. We're looking into what score has been given to Open RSS and how we can improve the number, if possible.
-
We were able to get some feeds working again.
-
Confirmed RSS feeds for the following websites are being affected by this issue:
Note: the feeds above are only based on active feeds, those that are being used. There could be more feeds that are affected, but are no longer in use.
-
Published a blog article to inform website owners about issues like this one that are stopping RSS readers and services from accessing Cloudflare-enabled websites for their users.
-
We've been informed that a Cloudflare employee responded on a HackerNews thread of the article:
My email is [redacted]. I'd like to hear from the owners of RSS readers directly on what they are experiencing. Going to ask team to take a closer look.
If Cloudflare is serious about solving this issue, then there should be full transparency. Instead of users contacting individual employees privately via email, Cloudflare should open a public thread (perhaps on their forum). That way, we can openly discuss solutions, leave comments on what is being done to resolve the issue, and refer to it for updates on the progress.
-
We've worked with Cloudflare to successfully get Open RSS added as a Cloudflare verified bot. But it's still unclear of what this means or if Cloudflare websites will continue to block Open RSS feed generation based on this new verified bots status. So we've reached out to Cloudflare for clarification.
-
All feeds that were previously blocked by Cloudflare appear to be fully operable again. We'll keep an eye on them over the next few days and mark this issue as resolved if there are no more problems.