Accueil > Réseau, Sécurité > Use iptables to monitor network usage

Use iptables to monitor network usage

24/11/2015 Categories: Réseau, Sécurité Tags: , , ,
Print Friendly, PDF & Email

Iptables is a powerful firewall/packet filtering framework inside Linux, and obviously used for firewalls on desktop, servers, and even embedded Linux devices such as most home internet routers. I was asked to write a script that could monitor and report network usage on one of our machines at work.

I took on the challenge and after searching package repositories and Google for cool Linux console apps that will report network usage, I came came across the idea of using iptables.. seeing as I love iptables, and it is installed by default on most machines it was the perfect solution for us.

The Idea
Iptables can be thought of a bunch of tables each containing some lists of rules called “chains”. There are some default chains which packets must progress through depending on the packets origin and destination. The main and default table that most people use is the ‘filter’ table, the default chains are:

  • INPUT – Packets coming to the machine from the network.
  • OUTPUT – Packets leaving your machine,
  • FORWARD – Packets passing through your machine, if your machine routes packets.

Each of these chains have a default policy, that is what should happen if there is no rules or no rules matching the packet, this is either:

  • ACCEPT – Allow the packet into the machine.
  • DROP – Drop the packet,

Now the default chains cannot be changed, the packets will work through one of those chains, we can add any rules we want to filter these packets. Netfilter/iptables tracks the amount of data running through chains. So if you want to track all your incoming network usage you can just use the INPUT chain, but if we want to track more specific traffic, we can create a custom chain, add a rule to pass the specific packets to this new chain, and thus monitor the specific traffic! Easy huh!

Before I go into the script and specific iptables configuration I’ll show you readers some useful itptables commands:

  • To see the manual page on iptables: man iptables
  • To list the rules on the default (filter) table: iptables -L
  • To list rules on other tables: iptables -t <tablename> -L

NOTE: If you add a -v you can see packet and byte counts.

Now we move onto what I did.

Network script and setup

I mentioned some iptables commands in the last section, so now I will describe the iptables command I use in the script for reporting:
iptables -L -n -x -v --line-numbers

The options mean:

  • -L = List the rules
  • -n = Do not do a DNS lookup, just show numbers
  • -x = use exact byte values, don’t convert to M or G, this is needed to ease the maths.
  • -v = verbose output, to actually show the counts
  • –line-numbers = The script inserts rules as to not disrupt other iptables rules that it doesn’t control so we need to know the rule number.

With the reporting explained let now talk about how we setup iptables, this is just the theory, the script actually sets it up for you, but as you will have different requirements you’ll need to know.

In this example we will only be only worried about monitoring things going through a proxy, which we’ll call 192.168.1.10 and traffic not coming from our local network, not via the proxy (not on 192.168.1.0/24). As the we get the required byte counts from the rule on the INPUT chain, we can use 1 custom chain for both types of traffic. So the first step is to create the custom chain and then add rules to match these packets:

iptables -N DOWNLOADED

Then we add a rule for each of the traffic conditions we want to track:

# Proxy rule
iptables -I INPUT 1 -s 192.168.1.10 -j DOWNLOADED

# Not our network rule
iptables -I INPUT 1 ! -s 192.168.1.0/24 -j DOWNLOADED

The above rules break down like:

  • -I INPUT 1 = Insert into the INPUT chain at index 1 (1 based).
  • -s <ip address or network> = Source is from <ip address>, the ‘!’ means negate (read as ‘not’)
  • -j DOWNLOADED = Jump or push this packet over to the DOWNLOADED chain.

See simple huh… ok maybe not, it is quite easy once you’ve used iptables for a while. Anyway, now that we have iptables set up I can talk about the script.

When ever the machine is rebooted or the chains flushed the counts will be zero’d out again, and as the chains only store the totals we need to keep track of the previous values so we can do a calculation. So I log the entries as three values (columns) separated by tabs:
date proxy bytes non-network bytes

The report I then generate says to usage since last check and current total, but the current total since when? In stead of having to parse the file since the last flush/reboot I simply have another file storing the last run with the following structure, similar to the log but containing the date of the last reset.

date proxy bytes non-network bytes total start date

Anyway without further adieu I’ll now present my script, it contains the reporting, and I have my own function that makes the report counts human readable:

  1. #!/usr/bin/env python
  2. import sys
  3. import os
  4. import datetime
  5. from send_email import send_email
  6. # Global Variables
  7. PROXY = « 192.168.1.10 »
  8. NETWORK = « 192.168.1.0/24 »
  9. IPTABLES_CUSTOM_CHAIN = « DOWNLOADED »
  10. IPTABLES_CREATE_CHAIN = « iptables -N «  + IPTABLES_CUSTOM_CHAIN
  11. IPTABLES_DELETE_CHAIN = « iptables -X «  + IPTABLES_CUSTOM_CHAIN
  12. IPTABLES_PROXY_RULE = « INPUT %s -s «  + PROXY +  » -j «  + IPTABLES_CUSTOM_CHAIN
  13. IPTABLES_NOT_NETWORK_RULE = « INPUT %s ! -s «  + NETWORK +  » -j «  + IPTABLES_CUSTOM_CHAIN
  14. IPTABLES_REPORT_CMD = « iptables -L -n -x -v –line-numbers »
  15. # Result column indexes
  16. TIMESTAMP_IDX = 0
  17. PROXY_IDX = 1
  18. NOT_NETWORK_IDX = 2
  19. TOTAL_START_IDX = 3
  20. # Format of the folling files: date     proxy bytes     non-network bytes
  21. # NOTE: Seperated by tabs (\t)
  22. LAST_RESULT = « /home/dpadmin/matt/bin/netmon.last »
  23. RESULT_LOG = « /home/dpadmin/matt/bin/netmon.log »
  24. # Email reporting variables
  25. EMAIL_TO = [’email@address.goes.here’]
  26. EMAIL_FROM = ’email.from@address.goes.here’
  27. EMAIL_SUBJECT = ‘Network Usage Report – %s’
  28. EMAIL_ATTACHMENTS = []
  29. EMAIL_SERVER = ‘localhost’
  30. EMAIL_MSG = «  » »Network usage between: %s and %s
  31. Proxy Traffic:
  32.   Usage: %s
  33.   Current Total: %s
  34. Non Network Traffic:
  35.   Usage: %s
  36.   Current Total: %s
  37. Total since: %s
  38. «  » »
  39. def human_readable(bytes):
  40.         if bytes < 1024:
  41.                 return str(bytes)
  42.         for x in ‘K’‘M’,‘G’:
  43.                 bytes /= 1024.0
  44.                 if bytes < 1024:
  45.                         return « %.2f%s » % (bytes, x)
  46.         if bytes > 1024:
  47.                 return « %.2f%s » % (bytes, ‘G’)
  48. def make_human_readable(results):
  49.         return (results[0], human_readable(results[1]), human_readable(results[2]))
  50. def get_totals():
  51.         timestamp = generate_timestamp()
  52.         result = os.popen(IPTABLES_REPORT_CMD)
  53.         proxy_bytes = 0
  54.         network_bytes = 0
  55.         # Parse the output. 
  56.         # 1. Find « Chain INPUT » that way we know we have the right chain.
  57.         # 2. Look for 1 and 2 in the first column, as they are our rules.
  58.         # 3. Find out which one is the proxy one.
  59.         # 4. return totals.
  60.         start = False
  61.         for line in result:
  62.                 if line.startswith(« Chain INPUT »):
  63.                         start = True
  64.                 elif line.startswith(« Chain »):
  65.                         start = False
  66.                 elif start:
  67.                         cols = line.split()
  68.                         if len(cols) != 0:
  69.                                 if cols[0] == ‘1’ or cols[0] == ‘2’:
  70.                                         # Found our rules
  71.                                         if cols[8] == PROXY:
  72.                                                 proxy_bytes = int(cols[2])
  73.                                         else:
  74.                                                 network_bytes = int(cols[2])
  75.         return (timestamp, proxy_bytes, network_bytes)
  76. def generate_timestamp():
  77.         d = datetime.datetime.now()
  78.         datestr = « %d/%.2d/%.2d-%.2d:%.2d:%.2d » % (d.year, d.month, d.day, d.hour, d.minute, d.second)
  79.         return datestr
  80. def get_last():
  81.         if os.path.exists(LAST_RESULT):
  82.                 lstFile = file(LAST_RESULT).readlines()
  83.                 result = lstFile[0].strip().split()
  84.                 result[PROXY_IDX] = int(result[PROXY_IDX])
  85.                 result[NOT_NETWORK_IDX] = int(result[NOT_NETWORK_IDX])
  86.                 return tuple(result)
  87.         else:
  88.                 timestamp = generate_timestamp()
  89.                 return (timestamp, 00, timestamp)
  90. def _cleanup_iptables():
  91.         os.system(« iptables -D %s » % (IPTABLES_PROXY_RULE % («  »)))
  92.         os.system(« iptables -D %s » % (IPTABLES_NOT_NETWORK_RULE % («  »)))
  93.         os.system(IPTABLES_DELETE_CHAIN)
  94. def start():
  95.         # Incase the rules alread exist lets remove them
  96.         _cleanup_iptables()
  97.         # Now we can add them
  98.         os.system(IPTABLES_CREATE_CHAIN)
  99.         os.system(« iptables -I %s » % (IPTABLES_PROXY_RULE % (« 1 »)))
  100.         os.system(« iptables -I %s » % (IPTABLES_NOT_NETWORK_RULE % (« 1 »)))
  101. def stop():
  102.         # Delete the rules TOTAL_START_IDX
  103.         _cleanup_iptables()
  104. def report():
  105.         last = get_last()
  106.         # Now we need to get the byte totals from iptables.
  107.         new_totals = get_totals()
  108.         reset_detected = False
  109.         proxy_usage = 0
  110.         not_network_usage = 0
  111.         total_start = last[TOTAL_START_IDX]
  112.         if last[PROXY_IDX] > new_totals[PROXY_IDX]:
  113.                 # Counters must have been reset.
  114.                 reset_detected = True
  115.                 proxy_usage = new_totals[PROXT_IDX]
  116.                 not_network_usage = new_totals[NOT_NETWORK_IDX]
  117.                 total_start = new_totals[TIMESTAMP_IDX]
  118.         else:
  119.                 # Do the calc
  120.                 proxy_usage = new_totals[PROXY_IDX] – last[PROXY_IDX]
  121.                 not_network_usage = new_totals[NOT_NETWORK_IDX] – last[NOT_NETWORK_IDX]
  122.         result = (new_totals[TIMESTAMP_IDX],proxy_usage, not_network_usage)
  123.         result_str = « Timestamp: %s Proxied: %s Off Network: %s »
  124.         # Write out the new last totals to the log and last.
  125.         last_file = file(LAST_RESULT, ‘w’)
  126.         tmp_list = []
  127.         tmp_list.extend(new_totals)
  128.         tmp_list.append(total_start)
  129.         last_file.write(« %s\t%d\t%d\t%s\n » % tuple(tmp_list))
  130.         last_file.close()
  131.         log = file(RESULT_LOG, ‘a’)
  132.         log.write(« %s\t%d\t%d\n » % new_totals)
  133.         log.close()
  134.         last = make_human_readable(last)
  135.         new_totals = make_human_readable(new_totals)
  136.         result = make_human_readable(result)
  137.         print « Last Total – «  + result_str % last
  138.         print « New Total – «  + result_str % new_totals
  139.         print « New Usage – «  + result_str % result
  140.         if reset_detected:
  141.                 msg =  » == RESET DETECTED! == \n »
  142.         else:
  143.                 msg = «  »
  144.         # Send the email report
  145.         msg += EMAIL_MSG % (last[TIMESTAMP_IDX],result[TIMESTAMP_IDX], result[PROXY_IDX], new_totals[PROXY_IDX], result[NOT_NETWORK_IDX], new_totals[NOT_NETWORK_IDX], total_start)
  146.         send_email(EMAIL_FROM, EMAIL_TO, EMAIL_SUBJECT % (result[TIMESTAMP_IDX]), msg, EMAIL_ATTACHMENTS, EMAIL_SERVER)
  147. def main(args):
  148.         if len(args) == 0:
  149.                 # Run report
  150.                 report()
  151.         elif str(args[0]).upper() == « CLEAR »:
  152.                 stop()
  153.         elif str(args[0]).upper() == « FLUSH »:
  154.                 stop()
  155.         elif str(args[0]).upper() == « STOP »:
  156.                 stop()
  157.         elif str(args[0]).upper() == « INITIATE »:
  158.                 start()
  159.         elif str(args[0]).upper() == « START »:
  160.                 start()
  161.         elif str(args[0]).upper() == « INITIALISE »:
  162.                 start()
  163.         elif str(args[0]).upper() == « REPORT »:
  164.                 report()
  165. if __name__ == « __main__ »:
  166.         main(sys.argv[1:])

The send email code looks like:

  1. import smtplib
  2. import os
  3. from email.MIMEMultipart import MIMEMultipart
  4. from email.MIMEBase import MIMEBase
  5. from email.MIMEText import MIMEText
  6. from email.Utils import COMMASPACE, formatdate
  7. from email import Encoders
  8. def send_email(send_from, send_to, subject, text, files=[], server=« localhost »):
  9.   assert type(send_to)==list
  10.   assert type(files)==list
  11.   msg = MIMEMultipart()
  12.   msg[‘From’] = send_from
  13.   msg[‘To’] = COMMASPACE.join(send_to)
  14.   msg[‘Date’] = formatdate(localtime=True)
  15.   msg[‘Subject’] = subject
  16.   msg.attach( MIMEText(text) )
  17.   for f in files:
  18.     part = MIMEBase(‘application’« octet-stream »)
  19.     part.set_payload( open(f,« rb »).read() )
  20.     Encoders.encode_base64(part)
  21.     part.add_header(‘Content-Disposition’‘attachment; filename= »%s »‘ % os.path.basename(f))
  22.     msg.attach(part)
  23.   smtp = smtplib.SMTP(server)
  24.   smtp.sendmail(send_from, send_to, msg.as_string())
  25.   smtp.close()

The script will setup the iptables setup by:

network_monitor.py start
network_monitor.py initiate
network_monitor.py initialise

To clean up iptables:

network_monitor.py clear
network_monitor.py flush
network_monitor.py stop

and finally to report:

network_monitor.py report
network_monitor.py

If you wish to graph the log then using higher and higher totals might not be what you want, so here is another script which parses the totals log and turns each entry into the daily usage, in MB, rather then totals:

  1. #!/usr/bin/env python
  2. import sys
  3. import os
  4. OUT_FILE = « netmon_graph.dat »
  5. def main(netmon_log):
  6.         if not os.path.exists(netmon_log):
  7.                 print « Error %s doesn’t exist! » % (netmon_log)
  8.                 sys.exit(1)
  9.         inFile = file(netmon_log)
  10.         outFile = file(OUT_FILE, ‘w’)
  11.         outFile.write(« %s\t%s\t%s\n » % (« Date »« Proxy »« Non-Network »))
  12.         line = inFile.readline()
  13.         lastProxyValue = 0
  14.         lastNetValue = 0
  15.         while len(line) > 0:
  16.                 #process
  17.                 cols = line.strip().split()
  18.                 if len(cols) == 3:
  19.                         date = cols[0]
  20.                         proxy = long(cols[1])
  21.                         net = long(cols[2])
  22.                         if proxy < lastProxyValue or net < lastNetValue:
  23.                                 lastProxyValue = 0
  24.                                 lastNetValue = 0
  25.                         # Calc
  26.                         newProxy = proxy – lastProxyValue
  27.                         newNet = net – lastNetValue
  28.                         lastProxyValue = proxy
  29.                         lastNetValue = net
  30.                         # Convert to MBs 
  31.                         newProxy = float(newProxy) / 1024.0 / 1024.0
  32.                         newNet = float(newNet) / 1024.0 / 1024.0
  33.                         outFile.write(« %s\t%.2f\t%.2f\n » % (date, newProxy, newNet))
  34.                 line = inFile.readline()
  35.         inFile.close()
  36.         outFile.close()
  37. if __name__ == « __main__ »:
  38.         main(sys.argv[1])

Happy network monitoring!

 

Source: Arrdino

Related Post

Les commentaires sont fermés.