Preventing a DDoS from China, a Great Firewall of China gone rogue?
Source: defendagainstddos.wordpress.com
On the 25th of January one of my sites was struggling to stay up, my “Dos Deflate” emails were popping into my inbox at a great frequency.
A quick run of the following command:
netstat -tn 2>/dev/null | grep :80 | awk ‘{print $5}’ | cut -d: -f1 | sort | uniq -c | sort -nr | awk ‘{print $2}’Gave me a list of addresses connected to the site. The number of addresses was way way above the number normally connected to the site. A site which although modestly doing 100,000s of page views per day will normally have only 300-500 port 80 connections at any one instance. This time however we had a 2 or 3 thousand.
A quick copy and paste into the following tool http://software77.net/geo-ip/multi-lookup/ (max 2000 ips per lookup) which is very usefully for bulk ip address to location lookups, yielded the following
1.26.199.253 # CN China 1.81.157.62 # CN China 1.83.168.34 # CN China 1.180.57.96 # CN China 1.26.201.167 # CN China 1.62.140.66 # CN China 1.189.82.78 # CN China 1.193.76.146 # CN China 1.26.241.209 # CN China 1.190.209.36 # CN China 1.189.162.213 # CN China 1.31.57.105 # CN China 1.62.17.196 # CN China 1.58.137.111 # CN China 1.83.229.4 # CN China 1.180.10.45 # CN China 1.180.5.200 # CN China 1.26.169.201 # CN China 1.180.187.10 # CN China 1.190.2.102 # CN China 1.25.134.125 # CN China 1.62.161.233 # CN China 1.182.28.181 # CN China (And on and on)
We clearly had an issue with China, but a quick log in to Google Analytics indicated almost no real-time traffic visiting the site.
Due to the sheer volume of connections for the next 45 mins, I was regularly running this command pasting into excel, then bulk iptabling the the ip address as well as the 65,000 neighbouring addresses.
sudo iptables -A INPUT -s 59.56.0.0/16 -m comment –comment “China IP” -j DROP sudo iptables -A INPUT -s 1.26.0.0/16 -m comment –comment “China IP” -j DROP
etc etc.
It didn’t take to long until around 100 million ip addresses had been blocked by the rules. This activity brought the site back into a more functioning state, with page load times drastically improved. Clearly a perpetual activity such as this was not the solution. So I got the code editor up and knocked together in a hurried fashion the following python file, which blocks non whitelisted visitors from china, can block “all” of china, block project honey pot threats and also block ip addresses with over 200 connections.
Feel free to use/modify as you wish, it is still a bit rough and ready but works well(remember to get a project honeypotAPI key):
https://s3-eu-west-1.amazonaws.com/codedumphere/blockChina.0.1dev.py
I set that to run as a cron job every 15 mins.
*/15 * * * * root /usr/bin/python /root/blockChina.0.1dev.py >/dev/null 2>&1
This removed the manual element of the process and allowed me to sleep at night.
The reality of the siltation was that although our site was back and running with almost no performance issues, the China IP blocklist wasn’t all encompassing and we were still getting ip address connections from China hitting the site. So yesterday (about two weeks after the inital issues) I enabled Apache logging (we have it turned off for performance reasons) to see what was still getting through.
Even with huge ranges of ip addresses blocked we were will getting traffic from China. The example shows three page requests per second:
180.212.91.60 - - [05/Feb/2015:16:45:43 +0000] "GET /announce.php?info_hash=%92%5F%1C%F3%E2%0Fj%2D%BD%C7ux%E0%B8t%F0%28%60%E5%CA&peer_id=%2DSD0100%2D%1E%D5%60u%40%AF%8A7%B2vB%0E&ip=180.212.91.60&port=20065&uploaded=20447232&downloaded=20447232&left=310902784&numwant=200&key=21597&compact=1 HTTP/1.0" 301 1186 "-" "Bittorrent" 222.161.250.167 - - [05/Feb/2015:16:45:43 +0000] "GET /announce.php?info_hash=%AC%F7%9E%EF%DC%AE%94%25%0D%9C%E9%0A%D6%D5R%27%F8Tj%DE&peer_id=%2DSD0100%2DA%2B%29%5D%2B%C3%2E%2C%D1%D5I%CE&ip=49.223.38.183&port=11091&uploaded=2111667829&downloaded=2111667829&left=49590208783&numwant=200&key=10003&compact=1 HTTP/1.0" 301 1222 "-" "Bittorrent" 180.212.91.60 - - [05/Feb/2015:16:45:43 +0000] "GET /announce?info_hash=%92%5F%1C%F3%E2%0Fj%2D%BD%C7ux%E0%B8t%F0%28%60%E5%CA&peer_id=%2DSD0100%2D%1E%D5%60u%40%AF%8A7%B2vB%0E&ip=180.212.91.60&port=20065&uploaded=20447232&downloaded=20447232&left=310902784&numwant=200&key=13577&compact=1 HTTP/1.0" 301 1178 "-" "Bittorrent"
So it looks to be that for some reason China is incorrect sending Bittorrent traffic to the site, effectively weaponising the Great Firewall of China or that the site’s IP address has been added to 1000s of torrent trackers to initiate a denial of service attack.
Solution #1 – Block “Bittorrent” user agent requests in .htaccess
As things currently stood any request for a page on our site that didn’t exist would send the visitor to the homage via a 301 redirect. This was putting load on our server. With almost all the bad traffic having the “Bittorrent” user agent lets send the 301 redirect elsewhere using “.htaccess”. See the code below:
<IfModule mod_rewrite.c>
RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} ^Bittorrent [NC]
RewriteRule ^ http://bittorrent.com [R=301,L]
</IfModule>This is better than the default behaviour or redirecting to our homepage or a 404 as it shifts the page serving load off the site. In this case we are sending Bittorrent traffic to the bittorrent site, as it seems more appropriate. Ideally we should not look to block a denial of service at layer 7 of the OSI model, so lets see what we can do with iptables.
Solution #2 – Block “GET /announce.php” tracking requests in ip tables
Almost all bad traffic were seeing was a request for the trackers “announce.php” file. By using the string searching capabilities of packets in iptables, we were able to block the request at Layer 3/4 of the OSI model
sudo iptables -A INPUT -p tcp -m tcp –dport 80 -m string –string “GET /announce.php” –algo bm –to 65535 -j DROP
In an ideal world we would not want to be searching all incoming port 80 packets for a specific string, but it is probably the most effective method to address what has been a consistent and ongoing issue.
In summary, if you have a similar issue you are welcome to use any one or all of those three methods to address the issue.
The good news is that the site is performing as normal. The one downside is that after nearly two weeks of massive volumes of traffic, ultimately it may be that completely moving ip address might be the only way to completely stop the traffic hitting the server.