As soon as a server becomes reachable from the Internet, it usually does not take long before the first unfriendly visitors arrive. This is no longer limited to occasional SSH login attempts. Automated scanners continuously check whether WordPress, phpMyAdmin, forgotten .env files, Git repositories, PHP debugging tools, known vulnerabilities or other exploitable services are available.
And when I say "check", I am talking about fairly impressive numbers.
On my small Raspberry Pi, I observed one scanner sending 272 HTTP requests within the same logged second. Another scanner generated 300 requests within roughly two seconds.
These were not requests for existing pages. The scanners systematically searched for files such as:
/.env
/.git/config
/.aws/credentials
/.ssh/id_rsa
/wp-config.php
/phpinfo.php
/docker-compose.yml
/composer.json
The small Raspberry Pi handled the traffic surprisingly well. Still, there is no reason to allow remote systems to test hundreds or thousands of URLs without restriction.
This is where Fail2Ban comes in.
Fail2Ban monitors system log files and detects repeated suspicious activity. If an IP address exceeds predefined limits, it is automatically blocked at the firewall level for a specified period of time.
Fail2Ban does not replace proper server configuration, regular updates or secure passwords. It also cannot protect against a real volumetric DDoS attack.
What it does very effectively is prevent automated scanners from spending hours testing their complete dictionaries against a server.
The resource requirements are minimal, configuration is straightforward and it runs perfectly well even on a small Raspberry Pi. For publicly accessible Linux servers, Fail2Ban has therefore become part of my standard setup.
My server runs Debian with Apache. At the router, only TCP ports 80 and 443 are forwarded to the web server. SSH and other administrative services are not accessible from the Internet at all.
That remains the most important security measure:
If a service does not need to be accessible from the Internet, do not expose it to the Internet.
Before starting with the configuration, here is a simplified overview of what happens:
flowchart TB
A([Internet])
B["Router<br/>only TCP 80 and 443"]
C["Apache Web Server"]
D{"Type of request"}
DN["normal"]
DF["file missing"]
DX["exploit"]
E["Serve normal request"]
F["File does not exist<br/>403 or 404"]
G["apache 404 scan<br/>counts matches"]
H{"Threshold<br/>reached?"}
HN["not yet"]
HY["reached"]
I["IP remains allowed"]
J["Exploit attempt"]
K["Apache processes or<br/>rejects request"]
L["apache hacker<br/>detects pattern"]
M{"Threshold<br/>reached?"}
MN["not yet"]
MY["reached"]
N["IP remains allowed"]
O["Firewall<br/>blocks IP address"]
P(["Further requests from this IP<br/>no longer reach Apache"])
A --> B
B --> C
C --> D
D --> DN
DN --> E
D --> DF
DF --> F
F --> G
G --> H
H --> HN
H --> HY
HN --> I
HY --> O
D --> DX
DX --> J
J --> K
K --> L
L --> M
M --> MN
M --> MY
MN --> N
MY --> O
O --> P
classDef internet fill:#f00,stroke:#3578e5,stroke-width:4px,color:#fff
classDef router fill:#fff4d6,stroke:#d89000,stroke-width:2px,color:#111
classDef apache fill:#e4f7e8,stroke:#209447,stroke-width:2px,color:#111
classDef normal fill:#eef8ee,stroke:#45a049,stroke-width:1.5px,color:#111
classDef warning fill:#fff0e6,stroke:#e07020,stroke-width:2px,color:#111
classDef fail2ban fill:#f0eaff,stroke:#7257c8,stroke-width:2px,color:#111
classDef decision fill:#f8f4ff,stroke:#7257c8,stroke-width:2px,color:#111
classDef label fill:#ffffff,stroke:#888,stroke-width:1px,color:#111
classDef blocked fill:#ffe5e5,stroke:#b42318,stroke-width:3px,color:#111
classDef endnode fill:#eeeeee,stroke:#555,stroke-width:2px,color:#111
class A internet
class B router
class C,D apache
class E normal
class F,J,K warning
class G,L fail2ban
class H,M decision
class DN,DF,DX,HN,HY,MN,MY label
class O blocked
class P endnodeInstalling Fail2Ban
On Debian or Raspberry Pi OS, installation is straightforward:
sudo apt update
sudo apt install fail2ban
sudo systemctl enable --now fail2ban
A quick check confirms whether the service is running:
sudo systemctl status fail2ban
Configured jails can be displayed with:
sudo fail2ban-client status
Fail2Ban already includes several filters for Apache.
However, for my web server I also use a few simple custom rules. This allows me to define exactly what I consider suspicious behaviour on this particular server.
Detecting scanners using 403 and 404 responses
A single 404 response is perfectly normal.
Someone may mistype an address, a browser may request an outdated favicon, or an old link may point to a page that no longer exists.
If the same IP address requests ten, twenty or one hundred different nonexistent files within a few minutes, the situation looks rather different.
For this purpose I create a custom filter:
sudo nano /etc/fail2ban/filter.d/apache-404-scan.conf
Contents:
[Definition]
failregex = ^<HOST> .* "(?:GET|POST|HEAD|OPTIONS) .*" (?:403|404) (?:\d+|-)
ignoreregex =
Then create the corresponding jail:
sudo nano /etc/fail2ban/jail.d/apache-404-scan.local
Contents:
[apache-404-scan]
enabled = true
port = http,https
filter = apache-404-scan
logpath = /var/log/apache2/*access.log
findtime = 300
maxretry = 10
bantime = 86400
With this configuration, an IP address may generate ten matching errors within five minutes.
Once the threshold is exceeded, the address is blocked for 24 hours.
These values are not universally correct for every website.
A large website with many outdated links may need more generous limits. On my small server, however, it is highly unlikely that a legitimate visitor will generate ten different 403 or 404 responses within five minutes.
Handling obvious attack patterns separately
Some requests are much less ambiguous.
There is no legitimate reason for an external visitor to search my server for files such as:
/.env
/.git/config
/.aws/credentials
/.ssh/id_rsa
/wp-config.php
/phpinfo.php
/cgi-bin/../../../../bin/sh
Someone requesting such paths is probably not trying to read my homepage.
For these patterns I use an additional filter:
sudo nano /etc/fail2ban/filter.d/apache-hacker.conf
Example configuration:
[Definition]
failregex = ^<HOST> .* "(?:GET|POST|HEAD|OPTIONS) .*?(?:\.env|\.git/|\.aws/|\.ssh/|wp-config|phpinfo|/cgi-bin/).*" (?:200|301|400|403|404) .*
ignoreregex =
The corresponding jail can react much more aggressively:
sudo nano /etc/fail2ban/jail.d/apache-hacker.local
[apache-hacker]
enabled = true
port = http,https
filter = apache-hacker
logpath = /var/log/apache2/*access.log
findtime = 600
maxretry = 3
bantime = 604800
In this example, three matching requests within ten minutes result in a seven day ban.
For a general 404 filter, that would be too aggressive for my taste.
But if someone repeatedly requests .aws/credentials, .ssh/id_rsa or wp-config.php, they can spend the next week looking for something else to do.
Testing filters before enabling them
Custom filters should always be tested against the existing Apache log before they are allowed to block addresses.
For example:
sudo fail2ban-regex /var/log/apache2/access.log /etc/fail2ban/filter.d/apache-404-scan.conf
And for the second filter:
sudo fail2ban-regex /var/log/apache2/access.log /etc/fail2ban/filter.d/apache-hacker.conf
Fail2Ban reports how many log entries were matched.
This makes it possible to check whether the regular expression is actually detecting the intended requests.
Afterwards, reload the configuration:
sudo fail2ban-client reload
The status of an individual jail can be displayed with:
sudo fail2ban-client status apache-404-scan
or:
sudo fail2ban-client status apache-hacker
What happens when 272 requests arrive in one second?
This is where things become interesting.
Fail2Ban does not sit in front of Apache.
The basic process looks like this:
Request arrives
↓
Apache processes request
↓
Apache writes access log entry
↓
Fail2Ban detects the log entry
↓
Threshold is exceeded
↓
Fail2Ban creates firewall rule
↓
Further connections are blocked
This inevitably introduces a small delay.
Modern scanners, however, can be extremely fast.
In my Apache log, one IP address appeared 272 times within the same logged second.
The scanner searched for Git files, WordPress configuration files, Composer files and other common configuration files.
That represents a burst in the range of:
272 requests per second.
Another scanner generated a total of approximately 300 requests within roughly two seconds.
This means something like the following can happen:
00.000 s Scanner starts
00.050 s Many requests are already in flight
00.200 s Apache writes log entries
00.400 s Fail2Ban detects matches
00.600 s Threshold is exceeded
00.800 s Firewall rule is installed
These timings are only an illustration. The important part is the principle.
If the scanner has already sent hundreds of requests in parallel, many of them may still reach Apache even though maxretry is configured as only 10.
So:
maxretry = 10
does not mean:
Apache will see no more than 10 requests.
It means:
After enough matching requests have been detected,
the IP address will be blocked.
With slow bots there may be almost no practical difference.
With scanners sending several hundred requests per second, the difference can be substantial.
Fail2Ban still performs its job.
Once the firewall rule has been installed, the next thousand or ten thousand requests from that IP address no longer reach Apache.
Results after only a few days
After only a few days of operation, the status on my server already looked like this:
| Jail | Detected matches | Currently banned | Total bans |
|---|---|---|---|
apache-hacker |
253 | 51 | 51 |
apache-php-scan |
82 | 8 | 8 |
apache-404-scan |
847 | 8 | 17 |
Fail2Ban had already triggered 76 ban actions.
Because some particularly active scanners were detected by several filters, these numbers cannot simply be added together to calculate the number of unique attackers.
Some IP addresses appeared simultaneously in the 404 filter, the PHP scanner and the hacker filter.
That makes sense.
A scanner requesting .git/config, followed by wp-config.php, followed by known PHP shell filenames, qualifies for several categories at once.
The change in Apache log file size was even more interesting.
Before enabling Fail2Ban, individual daily logs contained several thousand requests.
Afterwards, the numbers dropped considerably.
This does not mean the Internet suddenly became friendlier.
The bots are still there.
They simply do not get very far anymore.
The Raspberry Pi hardly notices
At first, 272 requests within a single second sounds like something that should make a small Raspberry Pi struggle.
For this type of scanning, however, most requests are extremely cheap to process.
Usually Apache only has to do something like:
GET /.env
File does not exist
Return 404
Done
Or:
GET /wp-config.php
File does not exist
Return 404
Done
There is no large database query, no computationally expensive application and no large document being generated.
A short burst of several hundred simple requests is therefore surprisingly undramatic for a Raspberry Pi.
Once Fail2Ban blocks the IP address, things become even easier.
Further connections are rejected at the firewall level.
Apache never sees them and therefore does not write additional access log entries.
This is also why Apache logs can become dramatically smaller after enabling Fail2Ban.
Fake bots and misleading user agents
Another interesting detail becomes obvious when looking through access logs.
One of the aggressive scanners identified itself as:
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
This obviously does not mean that Google was trying to download my AWS credentials or .git/config.
The HTTP User Agent is simply text supplied by the remote client.
Anyone can write anything into it.
Other attack requests used the names of well known search engines, AI bots and normal browsers while simultaneously requesting paths such as:
../../../../etc/passwd
/@fs/proc/self/environ
/@fs/root/.aws/credentials
Using the User Agent as a security criterion would therefore be a poor idea.
Behaviour is what matters.
Do not accidentally ban your own network
Trusted local networks can be excluded from banning.
For example:
sudo nano /etc/fail2ban/jail.local
Then:
[DEFAULT]
ignoreip = 127.0.0.1/8 ::1 192.168.2.0/24
Reload afterwards:
sudo fail2ban-client reload
Naturally, only networks that are actually trusted should be added here.
The router firewall is still more important
Fail2Ban is only an additional layer.
On my server, the router exposes only:
80/tcp
443/tcp
SSH is not reachable from the Internet.
The database is not reachable.
Samba is not reachable.
There is no externally accessible administration port.
This significantly reduces the attack surface.
A closed SSH port is better than a perfectly secured SSH service that I do not need remotely in the first place.
Fail2Ban then deals with the unavoidable noise hitting the two ports that actually need to remain publicly accessible because they provide the web service.
Conclusion
A publicly accessible web server will be scanned.
Not perhaps someday.
It starts almost immediately and continues permanently.
And we are not just talking about an occasional request for /wp-admin.
Modern scanners can test several hundred different URLs within a single second.
On my small Raspberry Pi, one real example produced:
272 requests within the same second.
Apache handled it without difficulty.
Fail2Ban detected the source.
The IP address was blocked.
From that point on, the firewall handled the rest.
Fail2Ban cannot turn an insecure server into a secure one.
But combined with an up to date Debian installation, a properly configured firewall, only the necessary port forwards and a clean Apache configuration, it provides a remarkably effective additional layer of protection.
And its resource requirements are so small that even a Raspberry Pi has plenty of time left to tell automated visitors from around the world:
404. Not here. And that is enough for today.
This article was originally published by DEV Community and written by Frank Wisniewski.
Read original article on DEV Community