Continued abuse of the service from automated apps, AI tools, web scrapers and bots
Ever since Open RSS has existed, the service has suffered from bots, automated systems, and scanners repeatedly probing the servers, making malicious requests to try to exploit weaknesses in login pages and other endpoints.
The code that powers the service is not currently open source or available to the public, and unfortunately, these attacks are exactly the kind that continue to shape that decision. They cost the organization time and server resources to mitigate, and pull effort away from providing new feeds.
This tracks these ongoing abuse patterns and documents the protections in place. Bringing more awareness to these issues gives the public full transparency into the challenges the organization faces and how its time is spent.
-
A certain number of suspicious IPs were sending abusive amounts of traffic to the service, causing outages and major disruptions. Requests from the offending addresses are now blocked.
-
To resolve this issue, feed generation and updates were restricted to only verified and known applications. If you are a feed reader developer or using a self-hosted feed reader, please get in touch for verification.
Even if you are not the developer of an unverified application you'd like to use with Open RSS, you can still get in touch so the organization can work with the developer to get the application verified.
-
Due to bots also using the feed preview pages to abuse the service, previewing a feed in a web browser will now likely show a heavily cached version of the feed. The feed will not be updated with new content unless it's used in a verified feed reader or if previewing the feed after logging in as a monthly donor.
-
Automated bots were found ignoring the site's crawling rules and scraping feed preview pages beyond normal use, slowing down the service for users. Rules were added to identify and restrict clients that behave that way.
-
Clients were firing rapid bursts of requests at the service, well above normal usage. Limits were put in place to catch and slow that behavior.
-
Since Cloudflare has been adopted as an addition to the abuse protection, many of the issues related to excessive requests from bots and AI appear to have been resolved. As a result, all cache restrictions have been lifted, and a login or usage of a verified reader is no longer required to view updated feed content.
-
Reopening this issue, as malicious activity is still a problem the organization must continue to address.
-
Refused bots that were found to be making requests to probe for injection weaknesses.
-
Requests were being made targeting internal network addresses to try to probe systems on the server that shouldn't be reachable from outside, and those requests are now refused.
-
Blocked automated systems that were caught sending a burst of malicious requests in an attempt to exploit weaknesses in login pages and other endpoints.