Recently, the anti-spam organization Spamhaus has come under yet another distributed denial-of-service attack. With some help from our good friends at myNetWatchman we were able to obtain a sample of the malware used in the attack. This one is particularly nasty, starting up 1500 threads to send randomized HTTP requests to Spamhaus’ webserver in a loop. This attack tool doesn’t have a command-and-control mechanism, so it was likely force-installed on all the infected systems of an existing botnet.
This kind of attack is particularly troublesome to deal with. While simplistic packet-based attacks can be more easily mitigated upstream, with an HTTP-based attack it is often difficult to distinguish attack traffic from legitimate HTTP requests. And because the attack consumes resources from the webserver, not just the system TCP/IP stack, it can quickly bring even a well-tuned webserver to its knees unless the target has better-than-average resources at its disposal to help weather the storm (like Spamhaus does).
Fortunately, most HTTP-based DoS attacks we have seen have a particular weakness – they are vulnerable to a technique known as “tarpitting”. This idea was first proposed by Tom Liston many years ago when the first scanning worms hit the Internet, and was implemented in his program LaBrea (which he no longer offers for download, due to legal concerns – however, the source code can still be found elsewhere and should work on Linux or BSD-based operating systems). The quickest way to implement tarpitting (if your webserver runs on Linux) is in the Linux netfilter source code. If the tarpit module is compiled for your Linux kernel, the operation becomes as simple as “iptables -A INPUT -s x.x.x.x -p tcp -j TARPIT“.
Tarpitting works by taking advantage of TCP/IP’s idea of window size and state. In a tarpitted session, we respond to the connection initiation as normal, but we immediately set our TCP window size to just a few bytes in size. By design, the connecting system’s TCP/IP stack must not send any more data than will fit in our TCP window before waiting for us to ACK the packets sent. This is to allow connections to deal with packet loss that might occur in a normal session. If the sending system doesn’t get an ACK to a packet sent, it will resend the packet at increasingly longer intervals. In our tarpitted session, we simply don’t ack any of the post-handshake packets at all, forcing the remote TCP/IP stack to keep trying to send us those same few bytes, but waiting longer each time. With this, bandwidth usage falls off quickly to almost nothing.
It might seem reasonable to simply use iptables -j DROP to not respond to connections at all – this will certainly prevent the webserver from seeing the connection, but because the connection may have a timeout set, it is likely we’ll keep seeing connection attempts at a fairly regular rate. Essentially a DROP rule turns our HTTP flood into a SYN flood.
Below is a chart made by running the Spamhaus DDoS malware on a single host in a sandnet, and comparing the bandwidth usage of each iptables mitigation technique with no mitigation. Here we extrapolated the bandwidth usage of the full non-mitigated attack (the blue line in the chart) from just the first 30 seconds of the attack, rather than let it fully DDoS the server the entire two hours. Under load, our untuned Apache server probably would have slowed down, forcing the attack bandwidth to decrease with it as it became unresponsive. In this simulation we are assuming the server can withstand a single attacker with no slowdown.
We see that although with tarpitting, there are momentary spikes as the sessions finally time out and try again, overall the average bandwidth usage is substantially lower with tarpitting as opposed to simply dropping the attacker’s packets.
There seems to be an added side-effect to mitigation by tarpit. When the attacked host mitigates with an iptables DROP (no response), the attacker’s CPU load is fairly minimal and the system is responsive. However, as demonstated by the graphic below, under tarpitting the CPU load in our test system quickly rose to 100% as the attacking system’s kernel tried to maintain a large number of open connections in a retry state.
In the case of our test system, the user interface became nearly unresponsive. Of course, this may depend on the overal system CPU speed and RAM (our test was done in a VMware environment which also may be a factor), but if the system becomes unacceptibly slow, it serves as an impetus for the unknowing owner of the attacking system to have the computer cleaned of its botnet infection.
Unfortunately, the TARPIT iptables module is not part of every Linux distribution. It is possible to obtain it by getting the netfilter Patch-o-Matic-NG source and compiling it for your kernel (don’t forget the iptables source must also be patched to support the TARPIT option).
At some point, the scalability of having thousands of iptables rules may come into play. At the University of Florida, Chuck Logan and Jordan Wiens were able to successfully mitigate a DDoS attack from the Netsky virus by writing customized software to detect the attacking hosts and create intelligent chains of iptables rules that were more scalable. Details and source code can be found here.
No matter how you implement it, tarpitting isn’t going to completely stop an attack – it can only slow it down. However, it could be an effective stop-gap measure until upstream providers can effectively null-route an attack on your website, or until the attack stops on its own.
Source: Dell SecureWorks – Author: Joe Stewart
