When the Great Firewall of China starts hosing your server with unexpected and unrelated traffic, how do you deal with it?
Discovering a problem
Three times in the last week I’ve had email reports from my Linode’s automatic warning system, informing me that the server had exceeded an average 8Mb/s output for a two hour period. Each time I logged on the traffic had gone right back down, and my website analytics never showed unusual traffic. By the third occurrence I wanted to get to the bottom of it, and I already had suspicions.
Those spikes are not normal.
Earlier in the day I’d stumbled across Craig Hockenberry’s post Fear China, where he was seeing a similar (but larger) problem over a longer period than I was. I looked into my access logs… and discovered I did indeed have the same problem, though it looks like I caught it earlier., or it was less severe.
Being DDOS’d via the Great Firewall of China
Distributed Denial of Service attacks flood a server with pointless requests from many computers all at once.
My logs showed requests for services and URLs that had nothing to do with my server, including an awful lot of BitTorrent URLs. Checking the geolocation of the requesting IPs showed they were all inside China. As Craig’s post covered – it looks a lot like there’s a mis-configuration with China’s state controlled firewall, and people’s normal traffic is sometimes being sent to entirely the wrong servers.
I wondered how bad my server was getting hit, as it didn’t seem to be in the same league as Craig’s:
Almost 27Mb/s out is roughly 95 times greater than normal for that server – close to two orders of magnitude increase, and I didn’t like that – I could imagine this getting worse rapidly.
Blocking China
As Craig discusses, there’s really no option but to block everyone from China. Unfortunately for me, I wasn’t using ipfw as a firewall so I couldn’t follow his advice. Having finally figured out how to do this I thought I’d write a step-by-step guide assuming you’ve not got a firewall already set up.
The mod_geoip2 module embeds GeoIP database lookups into the Apache web server. It is only capable of looking up the IP of a client that connects to the web server, as opposed to looking up arbitrary addresses.
This module works with Apache 2. Please use mod_geoip with Apache 1.
Installation
You may download the latest release of mod_geoip2 or get the latest development version from GitHub. See the included INSTALL file in the tarball for installation details.
Overview
The mod_geoip2 module uses the libGeoIP library to look up geolocation information for a client as part of the http request process. This module is free software, and is licensed under the Apache license.
To compile and install this module, you must first install libGeoIP 1.4.3 or newer.
The mod_geoip2 module takes effect either during request header parsing phase or the post read request phase, depending on whether it is configured for server-wide use or for a specific location/directory.
When enabled, the module looks at the incoming IP address and sets some variables which provide geolocation information for that IP. The variables it set depend on the specific GeoIP database being used (Country, City, ISP, etc.). These variables can be set in either the request notes table, the environment or both depending on the server configuration.
Anyone who has administered even a moderately high traffic server will have noticed that certain unwelcome traffic such as port scans and probes tend to come from IP addresses belonging to a certain group of countries. If your application or service does not cater to users in these countries, it might be a safe bet to block these countries off entirely.
This is especially true for email servers. The average email server, based on anecdotal evidence of servers for around 20 domains, rejects about 30% of incoming email every day as spam. Some servers on some days reject up to as much as 97% of incoming email as spam. Most of these originate in a certain subset of countries. That is a lot of wasted CPU cycles being expended on scanning these undesired emails for spam and viruses. Although tools such as amavisd and spamassasin do a good job of keeping the vast majority of spam out of users’ inboxes, when the rare well crafted and targeted phishing email does get through, it wrecks havoc in the enterprise.
If you want to find out where a given IP address is physically located on earth, there are quite a few online GeoIP lookup services you can try (e.g. geoiptool.com). These online services are mostly powered by freely available GeoIP databases such as those from MaxMind. Besides using such web-based services, there are different ways to query the GeoIP databases, notably via the Linux command line.
In this tutorial, I am going to describe how to geolocate an IP address from the command line in Linux.
Method One
The first method is to use geoiplookup tool which is a command-line client for MaxMind’s GeoIP databases. geoiplookup allows you to look up the geography or network information of an IP address (or hostname). You can install the tool (along with the free GeoIP database used by the tool) as follows.
To install geoiplookup on Debian, Ubuntu or Linux Mint:
$ sudo apt-get install geoip-bin
To install geoiplookup on Fedora:
$ sudo yum install geoip
To install geoiplookup on CentOS, first enable EPEL repository, and then use yum command:
$ sudo yum install geoip
The default installation of geoiplookup comes with GeoIP.dat database file which is located in /usr/share/GeoIP. With this database, you can look up the country information only.
$ geoiplookup 23.66.166.151
GeoIP Country Edition: US, United States
You can download additional GeoIP databases from MaxMind, which give you more detailed information about IP addresses beyond country info. You can also download more up-to-date GeoIP.dat from the site. This is recommended because GeoIP.dat may have already been outdated by the time you install it from Linux repositories. The GeoIP databases available on MaxMind website are updated every month.
To install additional GeoIP databases from MaxMind, do the following. You may want to set up a monthly cronjob to automate this process.
Now if you re-run geoiplookup, you will see the additional AS number information of an IP address. This basically tells you which administrative domain the IP address belongs to.
$ geoiplookup 128.112.119.209
GeoIP Country Edition: US, United States
GeoIP ASNum Edition: AS88 Princeton University
When run without any parameter, geoiplookup tool automatically uses GeoIP.dat and GeoIPASNum.dat only, but not use GeoLiteCity.dat. The latter can give you city-level information.
To obtain city-level geolocation information, explicitly tell geoiplookup to use GeoLiteCity.dat database.
The output includes state, city, zipcode, latitude and longitude. The accuracy of the inferred location varies across different countries and networks. For example, the geolocation result tends to be more accurate for broadband IP addresses, but not as accurate for mobile networks.
Method Two
If you want to avoid the hassle of installing and updating GeoIP databases, you can try ipinfo.io online service. Unlike other services, ipinfo.io provides JSON-based geolocation API, so you can easily look up geolocation from the command line, using tools like curl.
$ curl ipinfo.io/23.66.166.151
Note that the access to their API is rate-limited at 1,000 API requests per day.
I use fail2ban on my servers to protect them from would-be attackers, if you don’t your either insanely nieve to the fact that somebody wants in your system, or your just wanting to see if you can get hacked. Most of the attackers I would assume are just after another « bot » in their « net », or maybe a place to host files.
Durzo hosts a script that allows you to log the attacks on you into a mysql database with geocoding, I thought this would be cool to use as I could see from where I was being attacked. I then got this working and another script to display the table in a web page so I could view the data easily.
I then found some scripts from Google to pull data from MySQL in a geolocation table and generate an XML file used to import into Google Maps. With some tweaking and customizing, I now have a map with the geolocation data as markers on the map. Not all the markers are right on a building, but they are close enough for me to see the areas from which attacks are coming.
