System: fail2ban and iptables
source: http://www.the-art-of-web.com/system/fail2ban/
Around the beginning of 2005 we saw an increase in brute-force ssh attacks – people or robots trying different combinations of username and password to log into remote servers. A quick search on this topic returns many references to iptables and ipchains but noone really explained how they work.
Having just gone through this learning curve myself, and found a satisfactory solution in the fail2ban package, I’m going to try and explain how to achieve the simple goal of banning IP addresses that make repeated failed ssh login attempts.
If you want more technical information regarding firewalls and iptables in particular, see the References section at the bottom of this page.
Note: The following relates to the default config file provided with fail2ban 0.6.0 in the Debian unstable distribution and should be used for information purposes only. We take no responsibility for any consequences of following the instructions on this page.
1. What is/are iptables?
iptables is the firewall administration program for the Netfilter firewall mechanism which is built into the Linux kernel.
To view your current ‘firewall chains’ you need to run the following command as root:
# iptables -L
This will show a list of ‘chains’ called INPUT, FORWARD and OUTPUT – these are always present – followed by any custom chains. The default (blank) settings will look something like:
Chain INPUT (policy ACCEPT) target prot opt source destination Chain FORWARD (policy ACCEPT) target prot opt source destination Chain OUTPUT (policy ACCEPT) target prot opt source destination
If that’s what you’re seeing then there are no existing firewall rules and it’s probably safe to proceed. Otherwise you may want to check with the sysadmin who installed any existing firewall rules before making changes.
Note: you can also use iptables -L INPUT to list just the INPUT chain. Other useful options are to add -n to display values in numerical format or -v for verbose mode. Finally iptables -h will display all command-line options.
2. Installing and configuring fail2ban
The fail2ban package is available under Debian/unstable and also as a download for other Linux systems. See the Fail2Ban website linked under Resources at the bottom of the page for details. To install on Debian:
# apt-get -t unstable install fail2ban
If you run this command then fail2ban will be installed and already running as a daemon. However you might want to edit the configuration file and stop/start the daemon to get it running how you want. The configuration file can be found at /etc/fail2ban.conf.
The configuration file is broken up into sections. Most entries don’t need to be changed but there are a few that you might want to edit. The DEFAULT settings apply to all sections:
[DEFAULT]
maxfailures = 3 bantime = 900 findtime = 600
This says that after three failed access attempts are detected from a single IP address within 600 seconds or 10 minutes (findtime), then that address will be automatically blocked for 900 seconds (bantime). In most cases you won’t see them again as the bots will move on to another server.
[MAIL]
enabled = true to = root@localhost
Needs to be set to true for emails to be sent – to the root user in this case. Only do this if you’re actually going to check the root mailbox otherwise you’re just wasting disc space.
[Apache]
enabled = false
For now we’re not going to worry about monitoring the Apache log files.
[SSH]
enabled = true logfile = /var/log/auth.log
fail2ban will monitor the auth.log file for failed access attempts. As soon as the daemon is running your ssh port (22) will be protected from brute-force attacks – preventing more than a small number of attempts at one time.
To check if it’s running:
# /etc/init.d/fail2ban status Status of fail2ban: fail2ban is running.
Note: these commands will differ for different flavours of Linux.
To stop/start the daemon after making configuration changes:
# /etc/init.d/fail2ban stop Stopping fail2ban: .done # /etc/init.d/fail2ban start Starting fail2ban: .done
Actions taken by the daemon are logged by default in /var/log/fail2ban.log and you can change the verbosity in the conf file to one of: 0 – WARN, 1 – INFO or 2 – DEBUG.
3. What does fail2ban do with iptables?
This code runs when the daemon is started and adds new firewall rules using iptables:
fwstart = iptables -N fail2ban-ssh iptables -A fail2ban-ssh -j RETURN iptables -I INPUT -p tcp --dport ssh -j fail2ban-ssh
Translation:
- create a NEW chain called fail2ban-ssh;
- APPEND a RETURN command to the end of the fail2ban-ssh chain (same as return in PHP or other languages);
- INSERT a rule at the start of the built-in INPUT chain that redirects SSH packets to the fail2ban-ssh chain.
This adds the following to the output of iptables -L:
Chain INPUT (policy ACCEPT) target prot opt source destination fail2ban-ssh tcp -- anywhere anywhere tcp dpt:ssh Chain fail2ban-ssh (1 references) target prot opt source destination RETURN all -- anywhere anywhere
Because there are no actual rules in the fail2ban-ssh chain, connection attempts using SSH are simply redirected from the INPUT chain to the fail2ban-ssh chain and then sent straight back. What it does mean however is that we (or the fail2ban daemon to be precise) can insert new rules at any time and they will be applied to all incoming SSH traffic.
The fwend commands simply reverse this process, removing the JUMP command and and then FLUSHing and DELETEing the fail2ban-ssh chain:
fwend = iptables -D INPUT -p tcp --dport ssh -j fail2ban-ssh iptables -F fail2ban-ssh iptables -X fail2ban-ssh
Note: You should always stop fail2ban before editing the config file – so that it cleans up existing rules.
To ban an IP address the following command is run, with <ip> replaced by the actual IP address or hostname captured by the failregex regular expression (see below):
fwban = iptables -I fail2ban-ssh 1 -s <ip> -j DROP
Translation:
- Insert a rule as line 1 of the fail2ban-ssh chain to DROP all packets from the listed IP address or hostname.
When a fwban event is triggered, the output of iptables -L fail2ban-ssh --line-numbers becomes:
Chain fail2ban-ssh (1 references) num target prot opt source destination 1 DROP all -- <host> anywhere 2 RETURN all -- anywhere anywhere
where <host> is the problem ip address or domain. While this rule is in effect all ssh requests from <host> will be silently dropped. If a second fwban even occurs while the first is in place it will appear as line 1 and the other commands moved down the chain.
The fwunban command simply removes the DROP command from the chain. It’s called after 10 minutes or whatever time you have set for the bantime in fail2ban.conf:
fwunban = iptables -D fail2ban-ssh -s <ip> -j DROP
4. The failregex regular expression
The most complicated part of the configuration file is the regular expression that is applied to auth.log to determine what to count as an access failure.
failregex = : (?:(?:Authentication failure|Failed [-/\w+]+) for(?: illegal user)?|Illegal user|Did not receive identification) .* from (?P<host>\S*)
Note: the regular expression used here has been optimised for the auth.log in Debian Linux (Sarge) and may need to be modified slightly for other platforms.
In short, this matches any/all of the following auth.log entries and triggers a fwban event with the ip address or domain (host) that’s been captured by the regular expression:
Authentication failure for username from <host> Failed password for illegal user username from <host> Failed password for username from <host> Failed keyboard-interactive/pam for username from <host> Illegal user username from <host> Did not receive identification string from <host>
For anyone trying to reverse-engineer the regular expression:
- (?:.*) is a non-grouping version of regular parentheses; and
- (?P<name>.*) is similar to regular parentheses, but the substring matched by the group is accessible via the symbolic group name name.
The single matched variable <host> will be added to the fail2ban-ssh chain if maxfailures or more matches occur within findtime. It’s that simple.
You will see a lot of different (and usually simpler) failregex patterns that don’t contain the <host> capture. The reason for having the named capture in the regexp is to counter a bug that allows malicious users to trick your server into blocking non-hostile addresses:
« fail2ban’s approach to identifying an IP address in a login failure line is to scan the line for all IP addresses.
Since it is possible to generate false logins from accounts such as 10.2.28.2, it is possible to force fail2ban to block access to addresses which are not attempting to connect to the system. For each IP address available to the attacker, a desired ip address may be blocked. »
5. Customising the Config file
From reading more about iptables in the Linux Firewalls book (very informative but tough reading) and the online resources listed below it appears that some of the iptables commands contain options that are redundant:
- the RETURN command in fwstart is not necessary as a user-defined chain will automatically return to the parent chain after processing all commands (unless another JUMP is encountered). The only advantage I see in including the RETURN command is that iptables with the -v option can then show you both the number of packets/bytes sent to and returned from fail2ban-ssh – the difference telling you how many were dropped.
- the rule number 1 in fwban is also redundant as commands will by default be INSERTed at the top of the chain.
The fwstart command then becomes just:
fwstart = iptables -N fail2ban-ssh iptables -I INPUT -p tcp --dport ssh -j fail2ban-ssh
It’s also possible to use the LOG feature in iptables rather than relying on the fail2ban program. This gives the advantage of recording the details of packets that are DROPped while a ban rule is in place.
This involves a change to the fwban command:
fwban = iptables -A fail2ban-ssh -s <ip> -j LOG --log-prefix "Fail2Ban: " --log-ip-options --log-tcp-sequence --log-tcp-options iptables -A fail2ban-ssh -s <ip> -j DROP
Note: with the RETURN command removed we can use APPEND instead of INSERT here.
Translation:
- add a DROP command for <ip> at the start of the chain;
- add a LOG command with all options enabled at the start of the chain (before the DROP command).
and to fwunban:
fwunban = iptables -D fail2ban-ssh -s <ip> -j DROP iptables -D fail2ban-ssh -s <ip> -j LOG --log-prefix "Fail2Ban: " --log-ip-options --log-tcp-sequence --log-tcp-options
Note: DROP commands must exactly match the INSERT command syntax (or specify a rule number). The order isn’t really important but you’ll notice that we add the LOG command first and remove it last.
Now your logfiles will record all details of the dropped packages – typically in kern.log or syslog depending on your server configuration. Be aware that this can cause those files to grow quite quickly so only add the LOG commands if you’re going to use the data collected.
6. Does it work?
Does it ever!?!
Here you can see the results for a single server – you can tell when fail2ban was installed and why it was necessary:
$ zcat /var/log/auth.log* | grep 'Failed password' | grep sshd | awk '{print $1,$2}' | sort -k 1,1M -k 2n | uniq -c 408 Dec 11 77 Dec 12 12153 Dec 13 9390 Dec 14 1033 Dec 18 3743 Dec 19 4167 Dec 20 2789 Dec 25 9200 Dec 26 15742 Dec 27 281 Dec 28 7 Dec 29 17 Dec 30 3 Dec 31 4 Jan 1 5 Jan 2 4 Jan 4 9 Jan 5 11 Jan 6 9 Jan 7
There’s word of some big changes in the fail2ban package including ‘better configuration files’ and ‘templates for common services’ among other things but it’s not clear when that might make it into package form.
Note: if you just want to rate-limit SSH connections without installing new software then you can use the recent module but be aware that this can use a lot of system resources on busy networks as it has to store state information for all connections.
7. Related Articles
- Fail2Ban 0.8.3 Howto
- fail2ban and iptables
- fail2ban and sendmail
- Monitoring the fail2ban log
8. References
- Fail2Ban
- iptables Tutorial
- Manpage of IPTABLES
- Linux Firewalls, Third Edition – ISBN: 0672327716
- Python: Regular Expression HOWTO
Note: The following relates to the default config file provided with fail2ban 0.6.0 in the Debian unstable distribution and should be used for information purposes only. We take no responsibility for any consequences of following the instructions on this page.