Using Cloudflare on your website could be blocking RSS users

Many users prefer to use an RSS feed reader to stay up to date with the content on the websites they visit. But if you've enabled Cloudflare on your website, you're likely blocking these RSS users from accessing your website content without realizing it.

In Cloudflare's dashboard, you'll find tools designed to block bot traffic to your website. Particularly, the Bot Fight Mode and block all "AI scrapers and crawlers" options below. When enabled, these features end up blocking users who access your website through RSS readers, even though RSS readers are legitimate and aren't malicious bots.

When enabling the tools, Cloudflare will evaluate each visit to your website and determine whether the visit is from an AI scraper or "bot" based on a score , which ironically Cloudflare uses AI to generate.

Then, when a user's RSS reader attempts to read your website, Cloudflare presents it with a number of challenges that the reader would never be able to fulfill.

Here's an example of the Human Verification challenge that an RSS reader would be shown when it tries to visit your website. The challenge requires a human to solve and, because an RSS reader is not a human, it can never complete them.

In other cases, Cloudflare will simply block the RSS reader from accessing your website without a reason.

The only way to resolve when Cloudflare blocks an RSS reader from accessing your website is by contacting you directly and asking you to make a custom rule to unblock it. But Cloudflare shouldn't expect people to contact every owner of every Cloudflare website that blocks their RSS reader. And you shouldn't have to waste time logging into Cloudflare to add an exception every time they block an RSS reader, either.

Even though Cloudflare blocks RSS readers from your website, you can whitelist RSS readers as a workaround. This would at least unblock RSS readers without having to turn off any security features that you may have already been enabled until Cloudflare better addresses the issue.

First, find the user agent of any blocked RSS reader in Cloudflare's analytics dashboard. The User-Agent of most good RSS readers usually include the name of the reader, it's URL, or a word like "RSS" or "feed" that makes it obvious that it's an RSS reader.

Once you've identified an RSS reader's user agent, you can create a custom rule that explicitly whitelists and allows all traffic by the reader's IP address or by it's user agent string. Note that user agents can be disguised, so it's often better to whitelist the reader's IP address instead of the user agent. If you'd like to whitelist Open RSS, please contact us for the required information.

Cloudflare offers a bot verification program to which RSS readers owners can manually apply to avoid being blocked by websites, but this program isn't guaranteed to work and it suffers from quite a few problems.

• The verification process is flimsy — They're using a Google form for applications to the program. Then after applying, no notification is sent that they're working on it or even received the application successfully (we've tried applying twice), with no progress updates or expected timeframe for completion.

• Verified RSS readers are still being blocked — There are reports that RSS readers Cloudflare has verified as "good bots" are still being blocked from websites. If Cloudflare has successfully approved an RSS reader as a "good bot", it shouldn't be blocked or still require website owners to add any custom exception rules.

• Unblocking RSS readers across multiple websites is cumbersome — Cloudflare's only resolution to unblocking RSS readers is for the owners of the readers to contact each website owner directly and ask for an exception to be made. While that may work for one-off cases, this is unreasonable for RSS readers that have to access thousands of different Cloudflare-enabled websites each day. It's also overwhelming for website owners to configure exceptions for each and every RSS reader.

To be clear, there's nothing wrong with using Cloudflare's security tools on your website to help deal with malicious AI bots, scrapers, and potential attacks. But Cloudflare needs to ensure that people who use RSS tools aren't blocked from accessing your website content, and make it easier to resolve when they are.

If you're interested in following this Cloudflare issue as it relates to Open RSS, we're tracking it in issue 144, which has its own RSS feed you can subscribe to for updates.