What is the main objective of this entire topology?
Redundancy and Load Sharing! Imagine a scenario where your single web server is receiving millions and millions of HTTP requests per second, the CPU load is going insane, as well as the memory usage, when suddenly “crash!”, the server dies without saying good-bye (probably because of some weird hardware out-stage that you certainly won’t have time to debug). Well, this simple scheme might lead you into a brand new world of possibilities
What is this going to solve?
Hardware Failures! We are going to have redundant hardware all over the place, if one goes down, another one will be immediately ready for taking its place. Also, by using load sharing schemes, this is going to solve our High Usage! issue. Balancing the load among every server on our “farm” will reduce the amount of HTTP request per server (but you already figured that out, right?).
Let’s set it up! Firstly, we’re not going to use a domain scheme (let’s keep it simple), make sure your /etc/hosts file looks exactly like the picture below on every machine:
Here are a few notes about how to set-up a high-availability web server farm using Ubuntu 12.04 LTS using a whole load of awesome software (HAProxy, HeartBeat, Watchdog and Nginx)
The setup
In my setup I have five virtual machines, these are named and used for the following:-
haproxy1 – Our first proxy (master)/load-balancer (running HAProxy, HeartBeat and Watchdog) [IP address: 172.25.87.190] haproxy2 – Our second proxy (failover)/load-balancer (running HAProxy, HeartBeat and Watchdog) [IP address: 172.25.87.191] web1 – Our first web server node (running nginx) [IP address: 172.25.87.192] web2 – Our second web server node (running nginx) [IP address: 172.25.87.193] web3 – Our third web server node (running nginx) [IP address: 172.25.87.194]
The servers are connected in the following way:-
In my next post I will also explain how to configure the web servers to point to a backend shared storage cluster (using NFS) and a MySQL cluster server to have a truly highly available web hosting platform.
To demonstrate how iptables can perform network address translation this how-to shows how to use it to implement a over-simplified load balancer. In practice we would use a daemon such as HAProxy allowing IP tables to check packets before forwarding them.
Using the method presented in this tutorial packets get forwarded without going through the INPUT, FORWARD and OUTPUT chains.
iptables is a powerful tool that is used to create rules for how incoming or outgoing packets are handled. It keeps track of a packets state – there is NEW, ESTABLISHED, RELATED, INVALID and UNTRACKED. It can make filtering decisions based on the packets header data and the payload section of the packet, for these purposes iptables even has regular expression matching.
On top of that iptables has extensions that can be used to filter packets based on a packets history so we can keep track of packets and sessions. We can set filters to only trigger at specific times, parse the packet contents and header information searching for specific patterns, differentiate protocols such as tcp, udp, icmp, etc.
For load balancing behavior we want the incoming packets on one machine to be routed to another machine. iptables has extentions that helps us achieve this aim but we also need to muck around with its internal PREROUTING and POSTROUTING table, which is not recommended as this could potentially pose a security risk. lets use iptables to route all traffic coming in on an interface eth0 with a destination port 80 and route it to another IP address:
Allow IP forwarding
(Note: if your testing this on the same box your doing this on it won’t work, you need at least 3 machines to test this out, virtual ones work nicely)
First we enable ipv4 forwarding or this will not work: # echo "1" > /proc/sys/net/ipv4/ip_forward
XOR
# sysctl net.ipv4.ip_forward=1
next we add a filter that changes the packets destination ip and allows us to masquerade:
The above filter gets added to iptables PREROUTING chain. The packets first go through the filters in the PREROUTING chain before iptables decides where they go. The above filter says all packets input into eth0 that use tcp protocol and have a destination port 80 will have their destination address changed to 1.2.3.4 port 80. The DNAT target in this case is responsible for changing the packets Destination IP address. Variations of this might include mapping to a different port on the same machine or perhaps to another interface all together, that is how one could implement a simple stateful vlan (in theory).
The masquerade option acts as a one to many NAT server allowing one machine to route traffic with one centralized point of access. This is similar to how many commercial firewalls and network routers function.
The above ruleset results in all incoming packets to dport 80 traversing the iptables chains in a straight line from INCOMING to OUTGOING in the image below, effectively bypassing any rules we might have had in our INPUT chain. If we were to choose to implement nat like this we would need to implement those – our desired INPUT filter rules – on the machines where traffic is forwarded OR add them to the FORWARD chain if we want to block things before they are forwarded (Note: packets might go through FORWARD chain in both directions so direction needs to be considered when writing filters for this chain).
Path incoming packets take through iptables chains
In fight against DDoS through the years, i’ve compiled a list of useful iptables commands which may come handy in time of trouble.
Here is how to stop DDoS attack with iptables
To block small SYN floods: iptables -A INPUT -p tcp --syn -m limit --limit 1/s --limit-burst 3 -j RETURN
To block small UDP floods: iptables -N udp-flood iptables -A OUTPUT -p udp -j udp-flood iptables -A udp-flood -p udp -m limit --limit 50/s -j RETURN iptables -A udp-flood -j LOG --log-level 4 --log-prefix 'UDP-flood attempt: ' iptables -A udp-flood -j DROP
To limit rate of connections on a port: iptables -A INPUT -p tcp --dport (port-here) -m state --state NEW -m recent --set --name DDOS --rsource iptables -A INPUT -p tcp --dport (port-here) -m state --state NEW -m recent --update --seconds 5 --hitcount 5 --name DDOS --rsource -j DROP
Limit connections per ip on a port iptables -A INPUT -p tcp --syn --dport (port-here) -m connlimit --connlimit-above (limit-here) --connlimit-mask 32 -j REJECT --reject-with tcp-reset
To block an ip using iptables iptables -A INPUT -s (ip-here) -j DROP
If you use haproxy as load balancer use this to limit concurrent connections: stick-table type ip size 200k expire 30s store conn_cur tcp-request connection reject if { src_conn_cur ge 6 } tcp-request connection track-sc1 src
We’ve seen recently more and more DOS and DDoS attacks. Some of them were very big, requiring thousands of computers… But in most cases, this kind of attacks are made by a few computers aiming to make a service or website unavailable, either by sending it too many requests or by taking all its available resources, preventing regular users to use the service. Some attacks targets known vulnerabilities of widely used applications.
In the present article, we’ll explain how to take advantage of an application delivery controller to protect your website and application against DoS, DDoS and vulnerability scans.
Why using a LB for such protection since a firewall and a Web Application Firewall (aka WAF) could already do the job? Well, the Firewall is not aware of the application layer but would be useful to pretect against SYN flood attacks. That’s why we saw recently application layer firewalls: Web Application Firewalls, also known as WAF. Well, since the load balancer is in front of the platform, it can be a good partner for the WAF, filtering out 99% of the attacks, which are managed by script kiddies. The WAF can then happily clean up the remaining attacks. Well, maybe you don’t need a WAF and you want to take advantage of your Aloha and save some money ;).
Note that you need an application layer load-balancer, like Aloha or OpenSource HAProxy to be efficient.
TCP syn flood attacks
The syn flood attacks consist in sending as many TCP syn packets as possible to a single server trying to saturate it or at least, saturating its uplink bandwith.
If you’re using the Aloha load-balancer, you’re already protected against this kind of attacks: the Aloha includes mechanism to protect you. The TCP syn flood attack mitigation capacity may vary depending on your Aloha box.
It you’re running your own LB based on HAProxy or HAPEE, you should have a look at the sysctl below (edit /etc/sysctl.conf or play with sysctl command):
Note: If the attack is very big and saturates your internet bandwith, the only solution is to ask your internet access provider to null route the attackers IPs on its core network.
