Developer's Guide to Open RSS feeds

The Open RSS service gets hundreds of thousands of requests from all sorts of applications for feed content each day. But we're a small, nonprofit organization relying solely on donations from the public, which limits the funds available to invest in a large amount of servers and resources. So it's important that every app that interacts with the service follows the recommendations in this guide to avoid overloading our limited servers, slowing down feeds, degrading the service, or causing issues for other users.

The following practices are the very same practices we use when consuming content from other websites that power Open RSS feeds, and they've been extremely successful in preventing us from being blocked. So we encourage you to use them not just when your app is interacting with the Open RSS service, but for any other website your app may request feeds from.

When consuming Open RSS, your app will likely need to make requests to a large number of feeds (maybe even thousands) all from Open RSS servers. This is very different from just requesting one or two RSS feeds from a single website. The impact of making too many unnecessary requests to the service can significantly deplete our resources, making feeds extremely slow for everyone else. So please follow the guidelines below to ensure your app isn't requesting feed content in a way that will ruin the experience for other users.

Each response to an Open RSS feed includes a Cache-Control header with a max-age value that tells your app when there is likely new content to be retrieved from a feed. To avoid making unnecessary requests, please do not make any more requests to the feed until the max-age expires. The Cache-Control header isn't new or specific to Open RSS. It's a standard header field used all over the web to avoid making too many requests to a server— even web browsers use it.

Here's an example of a response to an Open RSS feed with the Cache-Control header.

HTTP/1.1 200 OK
Content-Type: application/xml
Content-Length: 124
Date: Mon, 21 Apr 2022 12:23:45 GMT
Cache-Control: max-age=300

As illustrated above, the Cache-Control header in the response has a max-age, which represents the amount of time (in seconds) before there's likely new content. This means that you should allow at least 5 minutes (300s / 60s) to elapse before making another request to this same feed. The number of seconds in max-age will depend on the feed. If the feed publishes new content very frequently, the max-age will be shorter, as new content is likely to be available in a shorter amount of time.

In addition to using the max-age above, you can also optionally use conditional requests when polling for feed updates to avoid using resources unnecessarily. With conditional requests, you have two options, both of which require your app to constantly check with the service for new feed updates:

1. The Last-Modified header (less efficient, less accurate, and potentially slower updates)
2. The Etag header (increased accuracy, slightly faster updates)

These headers are explained below.

Last-Modified header

When making a request to a feed, its response will include a Last-Modified header. The header will contain the most recent date and time the feed was updated with new content from its source website.

HTTP/1.1 200 OK
Content-Type: application/xml
Last-Modified: Tue 22 Feb 2022 12:23:45 GMT

Then, the next time your app needs to make another request to this same feed, take the Last-Modified value and set it on the If-Modified-Since header in the subsequent request. The service will return a 200 response with the updated feed if there is new content or a 304 response if there's no new content to be had.

Note, If-Unmodified-Since headers have no effect and are not applicable to the service.

ETag header

If your app has conditional request implemented and accepts the ETag header, your app can use this header to save bandwidth and avoid using unnecessary resources for feed updates. When making a request to a feed, it will respond with an ETag header.

HTTP/1.1 200 OK
Content-Type: application/xml
Etag: ab2bC9lbxGGwlmvcMGmMwYpn+99uig

When your app is ready to recheck the feed for updates, send another request with the If-None-Match header set to the ETag header given in the previous request. If the feed has new updates, the service will respond with a 200 response. If there have been no updates, a 304 response is returned with the original ETag value.

If-Match and If-Range headers aren't supported.

Occasionally, your app may receive a response with a redirect code signaling for it to make a request to some alternative location for the feed's content. Here's an example.

HTTP/1.1 301 Moved Permanently
Content-Type: text/html
Content-Length: 178
Date: Tue, 22 Feb 2022 12:23:45 GMT
Location: https://openrss.org/abcfeed.com/222

When receiving this response, this means that the URL used isn't correct, and your app will need to update the feed URL to point to the URL in the Location header.

Some apps—usually for security reasons—prefer not to redirect in these cases to prevent a feed owner from changing an authenticated feed URL in a way that would unauthenticate its users, for example. But all externally-facing Open RSS feeds are publicly available and will never require authentication or login credentials, so there is no need for this solution when requesting Open RSS feeds.

If your app makes a request for a feed that doesn't exist, you'll get a 404 response code, as illustrated below. Please don't continue to make requests after receiving this response code.

HTTP/1.1 404 Not Found
Content-Type: text/html
Content-Length: 326
Date: Wed, 23 Feb 2022 5:08:32 GMT

A 404 can be rare, because most Open RSS feeds users add to your application will initially work. But your app will especially need to pay attention to a 404 response in cases where a feed once existed but no longer does.

If you need to make multiple requests to different Open RSS feeds, making them all at the same time is a sure way to get blocked from the service. Instead, space out your requests, such that each request is initiated at least 1-2 seconds after the previous request has been kicked off, as shown below.

If making requests sequentially would make your users wait too long, we recommend making requests in the background while showing some loading indicator so your users can continue on with other tasks while the requests are being fulfilled.

If your app is an RSS reader, two areas where we see many readers get rate limited very quickly are when:

1. A user opens your app after having it closed for some time, and a lot of feeds need to be refreshed or
2. A new user is importing a large number of Open RSS feeds into your app with an OPML file

You'll want to make sure your app is handling requests sequentially in these cases, especially.

We enforce rate limits for every application that attempts to access Open RSS feed content, including RSS readers. When rate limited, the server simply responds with a 429 status code to indicate that we'd like for your app to slow down the frequency of its requests. The response will contain a Retry-After header with a value that tells you how long (in seconds) you'll need to wait before making another request to an Open RSS feed.

For instance, here's an example response indicating too many requests to a feed are being made, but can be retried after 300 seconds.

HTTP/1.1 429 Too Many Requests
Content-Type: application/xml
Content-Length: 1024
Date: Tue, 22 Feb 2022 12:23:45 GMT
Retry-After: 300

Please don't continue to make any other requests after getting a 429 response until the amount of seconds in the Retry-After header has elapsed. If you find that you're getting a 429 quite frequently, it's usually because the Cache-Control header above isn't being adhere to and you're likely overloading our server and increasing your chances of your app getting banned.

To avoid being flagged as suspicious or associated with malicious or abusive activity, your app needs to be explicitly clear about itself when making requests to the service by taking the following precautions.

Having an inappropriate or inaccurate user agent makes your application unidentifiable when making feed requests to the service. To avoid being blocked or restricted, please make sure your app sends a consistent user agent in its feed requests that clearly and uniquely identifies it. Your app's user agent should include:

1. The app name
2. The app's version (for debugging purposes), and
3. Some way for us to contact you or the app owners if necessary (usually a link to the app's website or their support email address)

Here's an example of a good user agent for a fictitious My RSS Bot.

While we understand that some apps may spoof the user agent mainly as a precaution (and not to be malicious and abusive), there's no need to do this when making requests to Open RSS feed content. Spoofing or using user agents that are not representative of your application is not only unsupported, but it's a violation of our Terms and can lead to your app being banned.

If your application is using a third-party library for its requests, the default user agent needs to be changed to the name of your app, or it will be flagged as suspicious. For instance, if your app is using the Python Request library, the user agent of requests should not use the library's default of python-requests, but changed to the recommendation above.

To ensure a proper response and to avoid your app being blocked or restricted, each of your requests must explicitly include one of the following XML mime types in the Accept header.

• application/xml
• text/xml
• application/rss+xml
• application/atom+xml

Please ensure your app doesn't make requests to Open RSS feeds using IP addresses that are known to be blacklisted or from companies that engage in fraudulent activity. Many of these IPs have already been banned from Open RSS, so using them is likely to give you unexpected results.

Note that using different IPs to circumvent a ban, rate limits, and/or any other restrictions is against our Terms and can result in legal action.

Last Updated: 2 weeks ago