From Detection to Prevention - Managing Poorly Behaved Bots

It seems these tech giants are aggressively trying to amass as much data as possible to fuel their large language models (LLMs) in an effort to keep pace with the advancements in AI, like those seen with ChatGPT. Unfortunately, this relentless data gathering appears to be conducted without regard for the collateral damage inflicted on other web services.

Assuming others are facing similar challenges, we decided to share some insights here that might prove helpful.

Our choice to resolve this challenge was our old friend Fail2Ban, a tool we have used since the beginning and found invaluable in blocking threats. Fail2Ban works by monitoring log files and banning IPs that show malicious signs—such as too many password failures, seeking exploits, etc.—and now, in our case, it helps manage excessive bot traffic.

Our Implementation with Fail2Ban

Today, we noticed performance degradation in some of our services, and upon analysis, we found that the majority of non-user-generated traffic was caused by the Bytedance, Facebook, and Amazon bots. Here are the links to their respective bot pages:

• AmazonBot
• facebookexternalhit

Upon further inspection, we checked the IP ranges from which the traffic was originating and confirmed that all addresses were legitimate and listed on the respective bots' pages. So, instead of it being a DDoS attack, it was legitimate bot traffic, unfortunately. Since our clients do not benefit much from these bots, and the offensive traffic was generating more than 1 GB of access log data per day, we decided to reduce the availability of our servers to them to manage the traffic and lighten the load.

Below is the sample from our access logs for each bot:

Steps to Throttle Bot Traffic Using Fail2Ban

1. Create a Jail Configuration:
Create a new jail configuration file in the jail.d directory.

Add the following content to the file:

• maxretry = 20 sets the limit to 20 requests within the findtime period.
• findtime = 600 sets the window to 10 minutes (600 seconds).
• bantime = 3600 sets the ban duration to 1 hour (3600 seconds).

This will allow us to grant the bot access, but limit the amount of requests the bot can make to 20 within 10 minutes, and if the limit is exceded it will ban the specific IP from accesing the server for the period of 1 hour. What you need to keep in mind is that facebook, and other companies use a large ammount of servers with different ips to do the crawling, so we are just preventing this specific ip for a short period of time. It will not fully block facebookg bot fro accesing the page, and it will make much more then 20 requests within 10 minutes using multiple ips.

2. Create a Filter for the Bots:
Create a filter configuration file in the filter.d directory.

Add the following content to the file:

3. Restart Fail2Ban & Verify the configuration is working::
After creating or modifying the configuration files, restart the Fail2Ban service to apply the changes. And check the status of Fail2Ban to ensure the webbots jail is active and correctly configured.

The output should look something like this:

By implementing these steps, we were able to manage and throttle the bot traffic effectively, ensuring that our clients’ services remained stable and performant. This approach can help others facing similar issues with excessive bot traffic, providing a practical solution to maintain server health and user experience.