Few days ago, my friend’s WordPress website went down. After investigation, I have figured out that it was receiving massive amount of posts requests to the xmlrpc.php file, which brings the apache and mysql to eat up all the system resources and the website crashed. Fortunately, I have figured out the way to mitigate this attack using Fail2Ban, which I’ll share in this post.
Install the Fail2Ban package using the following command:
apt-get install fail2ban iptables
Make a local copy of jail.conf file for configuration change:
hping is a free packet generator and analyzer for the TCP/IP protocol distributed by Salvatore Sanfilippo (also known as Antirez). Hping is one of the de facto tools for security auditing and testing of firewalls and networks, and was used to exploit the idle scan scanning technique (also invented by the hping author), and now implemented in the Nmap Security Scanner. The new version of hping, hping3, is scriptable using the Tcl language and implements an engine for string based, human readable description of TCP/IP packets, so that the programmer can write scripts related to low level TCP/IP packet manipulation and analysis in very short time.
On ubuntu hping can be installed from synaptic manager.
$ sudo apt-get install hping3
Syn flood
To send syn packets use the following command at terminal
$ sudo hping3 -i u1 -S -p 80 192.168.1.1
The above command would send TCP SYN packets to 192.168.1.1 sudo is necessary since the hping3 create raw packets for the task , for raw sockets/packets root privilege is necessary on Linux.
S – indicates SYN flag p 80 – Target port 80 i u1 – Wait for 1 micro second between each packet
.IptabLex & .IptabLes ELF DDoS malware is the malware made by China DDoSer crime group, designed to infect multiple architecture of Linux distribution, was aiming for Linux boxes in the internet with the low security and authentication flaw in SSH as vector of infection, was an emerged ELF threat in 2014.
Historically, MalwareMustDie, NPO (MMD) is the first entity who detected this malware around May last year and named it as Linux .Iptablesx|s on our last year’s alert MMD-0025-2014 [link] released on June 15, 2014. And we build malware repository for this ELF family for sharing samples and trend for researchers and industries on kernelmode started from September 4th 2014 [link], since the threat was gone so wild at the time and there was so few information about this malware that causing low awareness and detection ratio, so we managed all we can to suppress the growth of infection rate.
The DDoS attacks originated from this malware, in quantity of incidents and traffic used, was so massive in 2014 causing some warning was released from important security entities in September 2014, as per announced by Prolexic (thank you for mentioning MalwareMustDie) [link] in their Threat Advisory with « High Risk » level, following by Akamai‘s warning referred to the Prolexic’s advisory announcing the world wide warning [link] of .IptableS|X.
Afterward, Linux .IptableS / .IptablesX ELF malware was still be detected in the wild until the end of October 2014, but since November 2014 we did not find any significant wave of infection using these family, wiped by the emerge of many other China DDoS new malware families that we detected also afterwards. From the early this year (January 2015), we started to assume the malware popularity and development of .IptabLes|x was stopped..
However, on June 27th 2015 I was informed in the twitter by a friend @TinkerSec for what was suspected as Linux/ChinaZ infection. I supported him with ELF binary sample’s « real time » analysis in twitter as per shown in his report below:
Today, our team mate @benkow has detected a shellshock attack with having the same payload as sample, and curiousity made me taking a deeper analysis this time, to find and feel so surprised to realize that the payload is a Linux IptableS or .IptablesX variant actually. I can not believe this myself so I checked many times until I am very positive with this conclusion and after understanding why we were thinking it was Linux/ChinaZ I wrote this information as the follow up, the return of 2014’s DDoS disaster, the IptableS|X threat. Below is the detail.
Last week during a casual conversation I overheard a colleague saying: « The Linux network stack is slow! You can’t expect it to do more than 50 thousand packets per second per core! »
That got me thinking. While I agree that 50kpps per core is probably the limit for any practical application, what is the Linux networking stack capable of? Let’s rephrase that to make it more fun:
On Linux, how hard is it to write a program that receives 1 million UDP packets per second?
Hopefully, answering this question will be a good lesson about the design of a modern networking stack.
First, let us assume:
Measuring packets per second (pps) is much more interesting than measuring bytes per second (Bps). You can achieve high Bps by better pipelining and sending longer packets. Improving pps is much harder.
Since we’re interested in pps, our experiments will use short UDP messages. To be precise: 32 bytes of UDP payload. That means 74 bytes on the Ethernet layer.
For the experiments we will use two physical servers: « receiver » and « sender ».
They both have two six core 2GHz Xeon processors. With hyperthreading (HT) enabled that counts to 24 processors on each box. The boxes have a multi-queue 10G network card by Solarflare, with 11 receive queues configured. More on that later.
A couple of explicitly defined IP addresses will later become handy:
receiver$ for i in `seq 1 20`; do
ip addr add 192.168.254.$i/24 dev eth2;
done
sender$ ip addr add 192.168.254.30/24 dev eth3
1. The naive approach
To start let’s do the simplest experiment. How many packets will be delivered for a naive send and receive?
The sender pseudo code:
fd = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
fd.bind(("0.0.0.0", 65400)) # select source port to reduce nondeterminism
fd.connect(("192.168.254.1", 4321))
while True:
fd.sendmmsg(["x00" * 32] * 1024)
While we could have used the usual send syscall, it wouldn’t be efficient. Context switches to the kernel have a cost and it is be better to avoid it. Fortunately a handy syscall was recently added to Linux: sendmmsg. It allows us to send many packets in one go. Let’s do 1,024 packets at once.
With the naive approach we can do between 197k and 350k pps. Not too bad. Unfortunately there is quite a bit of variability. It is caused by the kernel shuffling our programs between cores. Pinning the processes to CPUs will help:
Now, the kernel scheduler keeps the processes on the defined CPUs. This improves processor cache locality and makes the numbers more consistent, just what we wanted.
In recent times our network has seen a lot of attempts to brute-force ssh passwords. A method to hamper such attacks by blocking attacker’s IP addresses using iptables ‘recent’ matching is presented in this text:
When the amount of connection attempts from a certain IP address exceeds a defined threshold, this remote host is blacklisted and further incoming connection attempts are ignored. The host is only removed from the blacklist after it has been stopped connecting for a certain time.
Edit: The fail2ban scripts offer a more sophisticated (but also more heavy-weighted) solution for this problem.
Software requirements
Linux kernel and iptables with ‘recent’ patch. (It seems that this patch has entered the mainline some time ago. ‘Recent’ matching e.g. is known to be included with kernels 2.4.31 and 2.6.8 of DebianSarge 4.0.)
Implementation
We begin with empty tables…
iptables -F
and add all the chains that we will use:
iptables -N ssh
iptables -N blacklist
Setup blacklist chain
One chain to add the remote host to the blacklist, dropping the connection attempt:
iptables -A blacklist -m recent --name blacklist --set
iptables -A blacklist -j DROP
The duration that the host is blacklisted is controlled by the match in the ssh chain.
Setup ssh chain
In the ssh chain, incoming connections from blacklisted hosts are dropped. The use of --update implies that the timer for the duration of blacklisting (600 seconds) is restarted every time an offending packet is registered. (If this behaviour is not desired, --rcheckmay be used instead.)
iptables -A ssh -m recent --update --name blacklist --seconds 600 --hitcount 1 -j DROP
These rules are just for counting of incoming connections.
iptables -A ssh -m recent --set --name counting1
iptables -A ssh -m recent --set --name counting2
iptables -A ssh -m recent --set --name counting3
iptables -A ssh -m recent --set --name counting4
With the following rules, blacklisting is controlled using several rate limits. In this example, a host is blacklisted if it exceeds 2 connection attempts in 20 seconds, 14 in 200 seconds, 79 in 2000 seconds or 399 attempts in 20000 seconds.
