When the Great Firewall of China starts hosing your server with unexpected and unrelated traffic, how do you deal with it?
Discovering a problem
Three times in the last week I’ve had email reports from my Linode’s automatic warning system, informing me that the server had exceeded an average 8Mb/s output for a two hour period. Each time I logged on the traffic had gone right back down, and my website analytics never showed unusual traffic. By the third occurrence I wanted to get to the bottom of it, and I already had suspicions.
Those spikes are not normal.
Earlier in the day I’d stumbled across Craig Hockenberry’s post Fear China, where he was seeing a similar (but larger) problem over a longer period than I was. I looked into my access logs… and discovered I did indeed have the same problem, though it looks like I caught it earlier., or it was less severe.
Being DDOS’d via the Great Firewall of China
Distributed Denial of Service attacks flood a server with pointless requests from many computers all at once.
My logs showed requests for services and URLs that had nothing to do with my server, including an awful lot of BitTorrent URLs. Checking the geolocation of the requesting IPs showed they were all inside China. As Craig’s post covered – it looks a lot like there’s a mis-configuration with China’s state controlled firewall, and people’s normal traffic is sometimes being sent to entirely the wrong servers.
I wondered how bad my server was getting hit, as it didn’t seem to be in the same league as Craig’s:
Almost 27Mb/s out is roughly 95 times greater than normal for that server – close to two orders of magnitude increase, and I didn’t like that – I could imagine this getting worse rapidly.
Blocking China
As Craig discusses, there’s really no option but to block everyone from China. Unfortunately for me, I wasn’t using ipfw as a firewall so I couldn’t follow his advice. Having finally figured out how to do this I thought I’d write a step-by-step guide assuming you’ve not got a firewall already set up.
If you want to find out where a given IP address is physically located on earth, there are quite a few online GeoIP lookup services you can try (e.g. geoiptool.com). These online services are mostly powered by freely available GeoIP databases such as those from MaxMind. Besides using such web-based services, there are different ways to query the GeoIP databases, notably via the Linux command line.
In this tutorial, I am going to describe how to geolocate an IP address from the command line in Linux.
Method One
The first method is to use geoiplookup tool which is a command-line client for MaxMind’s GeoIP databases. geoiplookup allows you to look up the geography or network information of an IP address (or hostname). You can install the tool (along with the free GeoIP database used by the tool) as follows.
To install geoiplookup on Debian, Ubuntu or Linux Mint:
$ sudo apt-get install geoip-bin
To install geoiplookup on Fedora:
$ sudo yum install geoip
To install geoiplookup on CentOS, first enable EPEL repository, and then use yum command:
$ sudo yum install geoip
The default installation of geoiplookup comes with GeoIP.dat database file which is located in /usr/share/GeoIP. With this database, you can look up the country information only.
$ geoiplookup 23.66.166.151
GeoIP Country Edition: US, United States
You can download additional GeoIP databases from MaxMind, which give you more detailed information about IP addresses beyond country info. You can also download more up-to-date GeoIP.dat from the site. This is recommended because GeoIP.dat may have already been outdated by the time you install it from Linux repositories. The GeoIP databases available on MaxMind website are updated every month.
To install additional GeoIP databases from MaxMind, do the following. You may want to set up a monthly cronjob to automate this process.
Now if you re-run geoiplookup, you will see the additional AS number information of an IP address. This basically tells you which administrative domain the IP address belongs to.
$ geoiplookup 128.112.119.209
GeoIP Country Edition: US, United States
GeoIP ASNum Edition: AS88 Princeton University
When run without any parameter, geoiplookup tool automatically uses GeoIP.dat and GeoIPASNum.dat only, but not use GeoLiteCity.dat. The latter can give you city-level information.
To obtain city-level geolocation information, explicitly tell geoiplookup to use GeoLiteCity.dat database.
The output includes state, city, zipcode, latitude and longitude. The accuracy of the inferred location varies across different countries and networks. For example, the geolocation result tends to be more accurate for broadband IP addresses, but not as accurate for mobile networks.
Method Two
If you want to avoid the hassle of installing and updating GeoIP databases, you can try ipinfo.io online service. Unlike other services, ipinfo.io provides JSON-based geolocation API, so you can easily look up geolocation from the command line, using tools like curl.
$ curl ipinfo.io/23.66.166.151
Note that the access to their API is rate-limited at 1,000 API requests per day.
As a system admin who maintains production Linux servers, there are circumstances where you need to selectively block or allow network traffic based on geographic locations. For example, you are experiencing denial-of-service attacks mostly originating from IP addresses registered with a particular country. In other cases, you want to block SSH logins from unknown foreign countries for security reasons. Or your company has a distribution right to online videos, which allows it to legally stream to particular countries only. Or you need to prevent any local host from uploading documents to any non-US remote cloud storage due to geo-restriction company policies.
All these scenarios require an ability to set up a firewall which does country-based traffic filtering. There are a couple of ways to do that. For one, you can use TCP wrappers to set up conditional blocking for individual applications (e.g., SSH, NFS, httpd). The downside is that the application you want to protect must be built with TCP wrappers support. Besides, TCP wrappers are not universally available across different platforms (e.g., Arch Linux dropped its support). An alternative approach is to set up ipset with country-based GeoIP information and apply it to iptables rules. The latter approach is more promising as the iptables-based filtering is application-agnostic and easy to set up.
In this tutorial, I am going to present another iptables-based GeoIP filtering which is implemented with xtables-addons. For those unfamiliar with it, xtables-addons is a suite of extensions for netfilter/iptables. Included in xtables-addons is a module called xt_geoip which extends the netfilter/iptables to filter, NAT or mangle packets based on source/destination countries. For you to use xt_geoip, you don’t need to recompile the kernel or iptables, but only need to build xtables-addons as modules, using the current kernel build environment (/lib/modules/`uname -r`/build). Reboot is not required either. As soon as you build and install xtables-addons, xt_geoip is immediately usable with iptables.
As for the comparison between xt_geoip and ipset, the official source mentions that xt_geoip is superior to ipset in terms of memory foot print. But in terms of matching speed, hash-based ipset might have an edge.
In the rest of the tutorial, I am going to show how to use iptables/xt_geoip to block network traffic based on its source/destination countries.
Install Xtables-addons on Linux
Here is how you can compile and install xtables-addons on various Linux platforms.
To build xtables-addons, you need to install a couple of dependent packages first.
Install Dependencies on Debian, Ubuntu or Linux Mint
Download the latest xtables-addons source code from the official site, and build/install it as follows.
$ wget http://downloads.sourceforge.net/project/xtables-addons/Xtables-addons/xtables-addons-2.10.tar.xz
$ tar xf xtables-addons-2.10.tar.xz
$ cd xtables-addons-2.10
$ ./configure
$ make
$ sudo make install
Note that for Red Hat based systems (CentOS, RHEL, Fedora) which have SELinux enabled by default, it is necessary to adjust SELinux policy as follows. Otherwise, SELinux will prevent iptables from loading xt_geoip module.
The next step is to install GeoIP database which will be used by xt_geoip for IP-to-country mapping. Conveniently, the xtables-addons source package comes with two helper scripts for downloading GeoIP database from MaxMind and converting it into a binary form recognized by xt_geoip. These scripts are found in geoip folder inside the source package. Follow the instructions below to build and install GeoIP database on your system.
According to MaxMind, their GeoIP database is 99.8% accurate on a country-level, and the database is updated every month. To keep the locally installed GeoIP database up-to-date, you want to set up a monthly cron job to refresh the local GeoIP database as often.
I use fail2ban on my servers to protect them from would-be attackers, if you don’t your either insanely nieve to the fact that somebody wants in your system, or your just wanting to see if you can get hacked. Most of the attackers I would assume are just after another « bot » in their « net », or maybe a place to host files.
Durzo hosts a script that allows you to log the attacks on you into a mysql database with geocoding, I thought this would be cool to use as I could see from where I was being attacked. I then got this working and another script to display the table in a web page so I could view the data easily.
I then found some scripts from Google to pull data from MySQL in a geolocation table and generate an XML file used to import into Google Maps. With some tweaking and customizing, I now have a map with the geolocation data as markers on the map. Not all the markers are right on a building, but they are close enough for me to see the areas from which attacks are coming.
