Developer's Guide to Open RSS feeds

All sorts of feed readers, RSS bots, and services are making thousands of requests to Open RSS feeds each day. It's important that each of them are using the Open RSS service in a way that won't compromise the service for other users. If you're using or planning to use the Open RSS service in an app you're developing, following the guidelines below will ensure that your app integrates well and won't cause any issues for other users.

As a small, nonprofit organization only relying on donations from the public, funds to invest in a large amount of servers are limited. Because of these infrastructure limitations, we must preserve our resources so that Open RSS remains reliable for all users. Therefore, you'll find that when making requests to Open RSS, the service will have restrictions that are different from other traditional APIs or services you may be used to.

Our servers receive extraordinary large amounts of traffic from all sorts of RSS readers and applications. Therefore, the frequency and rates at which RSS feed content can be retrieved must be limited to avoid degrading performance for other users. After your app receives RSS feed content, we assume that it will be stored, and no subsequent requests will be made until after the time frame included in the headers of our responses.

Feeds aren't updated with new content immediately and will likely have delays due to our caching to avoid overwhelming the web sites we rely on. The service shouldn't be used for time-sensitive tasks or app's that require content to be available in real-time.

To ensure feeds stay fast and responsive, Open RSS doesn't keep a full history of feed content. So feeds aren't guaranteed to have content that has been posted beyond the past few months.

Because the websites and sources feeds rely on are unpredictable, responses for feed content can be also. Although, we'll do our best to resolve any inconsistencies, we expect that your app is always prepared to handle them gracefully as well to avoid unexpected behavior.

Requesting RSS feeds from Open RSS is different from requesting feeds from other websites. Requests for feeds on other sites are usually pretty straightforward. Your app makes a request to an RSS feed on a server and displays the content and does this repetitively across a number of different servers.

But when consuming Open RSS, your app will likely need to make requests to a number of feeds (maybe even thousands) all from the same openrss.org server. This is a very important distinction and the primary reason we strongly advise you to follow the guidelines below when making requests to Open RSS feeds.

Each response to an Open RSS feed includes a Cache-Control header with a max-age that tells you when there is likely new content to be retrieved from a feed. Please use this header to avoid making unnecessary requests. Here's an example of a response to an Open RSS feed with the Cache-Control header.

HTTP/1.1 200 OK
Content-Type: application/xml
Content-Length: 124
Date: Mon, 21 Apr 2022 12:23:45 GMT
Cache-Control: max-age=300

You'll notice that the Cache-Control header in the response has a max-age, which represents the amount of time (in seconds) before there's likely new content. This means that you should allow at least 5 minutes (300s / 60s) to elapse before making another request to this same feed.

The amount of seconds in max-age will depend on the feed. If the feed is a popular one, the max-age will be shorter. This is because new content is likely to be available in a shorter amount of time, so we expect requests to them to be more frequent, to ensure you're always getting the newest data.

We enforce rate limits for every application that attempts to access Open RSS feed content, including RSS readers, but we don't use any custom rate limiting responses. Instead, the server simply responds with a 429 status code to indicate that we'd like for your app to slow down the frequency of its requests. The response will contain a Retry-After header with a value that tells you how long (in seconds) you'll need to wait before making another request to an Open RSS feed.

For instance, here's an example response indicating too many requests to a feed are being made, but can be retried after 300 seconds.

HTTP/1.1 429 Too Many Requests
Content-Type: application/xml
Content-Length: 1024
Date: Tue, 22 Feb 2022 12:23:45 GMT
Retry-After: 300

Please don't continue to make any other requests after getting a 429 response until the amount of seconds in the Retry-After header has elapsed. If you find that you're getting a 429 quite frequently, it's usually because the Cache-Control header above isn't being adhere to and you're likely overloading our server and increasing your chances of getting banned.

To ensure a feed response is XML, you'll need to explicitly include one of the following XML mime types in the Accept header of your requests.

• application/xml
• text/xml
• application/rss+xml
• application/atom+xml

If your request doesn't include any of the mime types above in its Accept headers, you may receive the HTML preview that a user would see when they access the feed from a web browser. This is particularly the case if you're using a headless browser or similar for requests. To avoid this altogether, it's always just safer to set the header every time a request is made.

If you need to make multiple requests to different Open RSS feeds, avoid making them all at the same time. Instead, we recommend that you space out your requests, such that each request is initiated at least 1-2 seconds after the previous request as been kicked off, as shown below.

If making requests sequentially would make your users wait too long, we recommend making requests in the background while showing some loading indicator, so your users can continue on with other tasks while the requests are being fulfilled.

If your app is an RSS reader, two areas where we see many readers get rate limited very quickly are when:

1. A user opens your app after having it closed for some time and a lot of feeds need to be refreshed or
2. A new user is importing a large number of Open RSS feeds into your app with an OPML file

You'll want to make sure your app is handling requests sequentially in these cases, especially.

Your app will likely need to show its users content from the same Open RSS feed multiple times. There's no need to make a request each of those times. Making unnecessary requests to the same feed not only makes your app slower, but it also consumes unnecessary resources from Open RSS servers that can be used for other traffic.

Instead, after requesting a feed's content, your app should store it in a database. Then, reuse it when a user attempts to access the contents of the same feed. Meanwhile, hold off on making any new requests to the feed until the max-age in the Cache-Control header has elapsed.

Sometimes your app may receive a response with a redirect code signaling for it to make a request to some alternative location for the feed's content. Here's an example.

HTTP/1.1 301 Moved Permanently
Content-Type: text/html
Content-Length: 178
Date: Tue, 22 Feb 2022 12:23:45 GMT
Location: https://openrss.org/abcfeed.com/222

When receiving this response, this means that the URL used isn't correct, and your app will need to update the feed URL to point to the URL in the Location header.

Some apps—usually for security reasons—prefer not to redirect in these cases to prevent a feed owner from changing an authenticated feed URL in a way that would unauthenticate its users, for example. But all externally-facing Open RSS feeds are publicly available and will never require authentication or login credentials, so there is no need for this solution when requesting Open RSS feeds.

To avoid being banned, please use a consistent user agent in your requests that clearly and uniquely identifies your application. Your app's user agent should include the unique name of your app, its version (for debugging purposes), and a URL that contains a way for us to contact you if necessary. Here's an example of a good user agent for a fictitious My RSS Bot.

While we understand that some apps may spoof the user agent mainly as a precaution (and not to be malicious and abusive), there's no need to do this when making requests to Open RSS feed content. Spoofing or using user agents that are not representative of your application is not only unsupported, but it's a violation of our Terms and can lead to your app being banned.

Please ensure your app does not make requests to Open RSS feeds using IP addresses that are known to be blacklisted or from companies that engage in fraudulent activity. Many of these IPs have already been banned from Open RSS, so using them when making requests to the server is likely to give you unexpected results.

Note that using different IPs in order to circumvent a ban, rate limits, and/or any other restrictions is against our Terms and can result in legal action.

Last Updated: 7 months ago