Archive

Archives pour 11/2015

Monitor TCP Traffic on specific port

30/11/2015 un commentaire

Source: superuser.com

I’ve searched quite extensively for this, but cannot seem to come up with a working example.

My objective is to monitor TCP traffic on a specific port to see incoming connections and write them to a text file. The catch is I also need a timestamp on each row to show exactly when the client connected down to the second.

I’ve already exhausted netstat, nmap, and tcptrack, but none support timestamp.

I was thinking a linux shell script might work if I monitored a specific local port and wrote text to a file when a connection is made then just concatenate the date on each line.

I was playing with this:

netstat -ano|grep 443|grep ESTABLISHED

as well as this:

tcptrack -i eth0 port 443

but neither suit my needs as I need the time the connection comes in at.

Categories: Réseau Tags:

Communication Networks/IP Tables

30/11/2015 Comments off

Operational summary

The netfilter framework, of which iptables is a part of, allows the system administrator to define rules for how to deal with network packets. Rules are grouped into chains—each chain is an ordered list of rules. Chains are grouped into tables—each table is associated with a different kind of packet processing.

Each rule contains a specification of which packets match it and a target that specifies what to do with the packet if it is matched by that rule. Every network packet arriving at or leaving from the computer traverses at least one chain, and each rule on that chain attempts to match the packet. If the rule matches the packet, the traversal stops, and the rule’s target dictates what to do with the packet. If a packet reaches the end of a predefined chain without being matched by any rule on the chain, the chain’s policy target dictates what to do with the packet. If a packet reaches the end of a user-defined chain without being matched by any rule on the chain or the user-defined chain is empty, traversal continues on the calling chain (implicit target RETURN). Only predefined chains have policies.

Rules in iptables are grouped into chains. A chain is a set of rules for IP packets, determining what to do with them. Each rule can possibly dump the packet out of the chain (short-circuit), and further chains are not considered. A chain may contain a link to another chain – if either the packet passes through that entire chain or matches a RETURN target rule it will continue in the first chain. There is no limit to how many nested chains there can be. There are three basic chains (INPUT, OUTPUT, and FORWARD), and the user can create as many as desired. A rule can merely be a pointer to a chain.

Tables

There are three built-in tables, each of which contains some predefined chains. It is possible for extension modules to create new tables. The administrator can create and delete user-defined chains within any table. Initially, all chains are empty and have a policy target that allows all packets to pass without being blocked or altered in any fashion.

  • filter table — This table is responsible for filtering (blocking or permitting a packet to proceed). Every packet passes through the filter table. It contains the following predefined chains, and any packet will pass through one of them:
    • INPUT chain — All packets destined for this system go through this chain (hence sometimes referred to as LOCAL_INPUT)
    • OUTPUT chain — All packets created by this system go through this chain (aka. LOCAL_OUTPUT)
    • FORWARD chain — All packets merely passing through the system (being routed) go through this chain.
  • nat table — This table is responsible for setting up the rules for rewriting packet addresses or ports. The first packet in any connection passes through this table: any verdicts here determine how all packets in that connection will be rewritten. It contains the following predefined chains:
    • PREROUTING chain — Incoming packets pass through this chain before the local routing table is consulted, primarily for DNAT (destination-NAT).
    • POSTROUTING chain — Outgoing packets pass through this chain after the routing decision has been made, primarily for SNAT (source-NAT).
    • OUTPUT chain — Allows limited DNAT on locally-generated packets
  • mangle table — This table is responsible for adjusting packet options, such as quality of service. All packets pass through this table. Because it is designed for advanced effects, it contains all the possible predefined chains:
    • PREROUTING chain — All packets entering the system in any way, before routing decides whether the packet is to be forwarded (FORWARD chain) or is destined locally (INPUT chain).
    • INPUT chain — All packets destined for this system go through this chain
    • FORWARD chain — All packets merely passing through the system go through this chain.
    • OUTPUT chain — All packets created by this system go through this chain
    • POSTROUTING chain — All packets leaving the system go through this chain.

In addition to the built-in chains, the user can create any number of user-defined chains within each table, which allows them to group rules logically.

Each chain contains a list of rules. When a packet is sent to a chain, it is compared against each rule in the chain in order. The rule specifies what properties the packet must have for the rule to match, such as the port number or IP address. If the rule does not match then processing continues with the next rule. If, however, the rule does match the packet, then the rule’s target instructions are followed (and further processing of the chain is usually aborted). Some packet properties can only be examined in certain chains (for example, the outgoing network interface is not valid in the INPUT chain). Some targets can only be used in certain chains, and/or certain tables (for example, the SNAT target can only be used in the POSTROUTING chain of the nat table).

Lire la suite…

SynFlood

29/11/2015 Comments off

1 – Le concept

L’attaque SynFlood est basée sur l’envoi massif de demande d’ouverture de session TCP. Les buts recherchés peuvent être :

  • Le Buffer Overflow du process écoutant le port TCP de destination.
  • La saturation du nombre d’ouverture de session TCP en cours.

2 – Le fonctionnement

2.1 – Schéma

2.2 – Envoi du SYN

Le fonctionnement est de générer une trame TCP de demande de synchronisation à destination de la cible. Cette demande de synchronisation SYN est la première étape d’une ouverture de session TCP. Voici le schéma de l’entête TCP avec ce fameux flag SYN basé sur 1 bit :

 

Les 5 autres flags doivent être positionnés à 0.

2.3 – Réception par la cible A

La cible recevant la synchronisation TCP mémorise cette demande nécessitant donc de la mémoire et du processeur. Voici l’état des connexions d’un Windows XP avant la réception d’un Synflood :

 

Et voici après la réception des demandes de SYN :

 

La cible passe les requêtes reçues en SYN_RECEIVED. Cet état est temporaire, le temps de durée de vie est variable en fonction de la pile IP.

Dans mon exemple, la cible tourne sur une station XP limitée en nombre de sessions simultanée. Ceci ayant pour conséquence l’indisponibilité temporaire du port ciblé.

* testé depuis la machine ayant effectuée le Synflood

Lire la suite…

Categories: Réseau, Sécurité Tags: , , ,

Simple Stateful Load Balancer with iptables and NAT

27/11/2015 Comments off

To demonstrate how iptables can perform network address translation this how-to shows how to use it to implement a over-simplified load balancer. In practice we would use a daemon such as HAProxy allowing IP tables to check packets before forwarding them.

Using the method presented in this tutorial packets get forwarded without going through the INPUT, FORWARD and OUTPUT chains.

iptables is a powerful tool that is used to create rules for how incoming or outgoing packets are handled. It keeps track of a packets state – there is NEW, ESTABLISHED, RELATED, INVALID and UNTRACKED. It can make filtering decisions based on the packets header data and the payload section of the packet, for these purposes iptables even has regular expression matching.

On top of that iptables has extensions that can be used to filter packets based on a packets history so we can keep track of packets and sessions. We can set filters to only trigger at specific times, parse the packet contents and header information searching for specific patterns, differentiate protocols such as tcp, udp, icmp, etc.

For load balancing behavior we want the incoming packets on one machine to be routed to another machine. iptables has extentions that helps us achieve this aim but we also need to muck around with its internal PREROUTING and POSTROUTING table, which is not recommended as this could potentially pose a security risk. lets use iptables to route all traffic coming in on an interface eth0 with a destination port 80 and route it to another IP address:

Allow IP forwarding

(Note: if your testing this on the same box your doing this on it won’t work, you need at least 3 machines to test this out, virtual ones work nicely)

First we enable ipv4 forwarding or this will not work:
# echo "1" > /proc/sys/net/ipv4/ip_forward

XOR

# sysctl net.ipv4.ip_forward=1

next we add a filter that changes the packets destination ip and allows us to masquerade:

# iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 80 -j DNAT --to-destination 10.0.0.3:80
# iptables -t nat -A POSTROUTING -j MASQUERADE

The above filter gets added to iptables PREROUTING chain. The packets first go through the filters in the PREROUTING chain before iptables decides where they go. The above filter says all packets input into eth0 that use tcp protocol and have a destination port 80 will have their destination address changed to 1.2.3.4 port 80. The DNAT target in this case is responsible for changing the packets Destination IP address. Variations of this might include mapping to a different port on the same machine or perhaps to another interface all together, that is how one could implement a simple stateful vlan (in theory).

The masquerade option acts as a one to many NAT server allowing one machine to route traffic with one centralized point of access. This is similar to how many commercial firewalls and network routers function.

The above ruleset results in all incoming packets to dport 80 traversing the iptables chains in a straight line from INCOMING to OUTGOING in the image below, effectively bypassing any rules we might have had in our INPUT chain. If we were to choose to implement nat like this we would need to implement those – our desired INPUT filter rules – on the machines where traffic is forwarded OR add them to the FORWARD chain if we want to block things before they are forwarded (Note: packets might go through FORWARD chain in both directions so direction needs to be considered when writing filters for this chain).

560x260xbuilt-in-chains-in-filter-table.png.pagespeed.ic.CVAYVeFJ96

Path incoming packets take through iptables chains

Lire la suite…

IP leak affecting VPN providers with port forwarding

27/11/2015 Comments off

Vulnerability “Port Fail” reveals real IP address

We have discovered a vulnerability in a number of providers that allows an attacker to expose the real IP address of a victim. “Port Fail” affects VPN providers that offer port forwarding and have no protection against this specific attack. Perfect Privacy users are protected from this attack.

This IP leak affects all users: The victim does not need to use port forwarding, only the attacker has to set it up.

We have tested this with nine prominent VPN providers that offer port forwarding. Five of those were vulnerable to the attack and have been notified in advance so they could fix this issue before publication. However, other VPN providers may be vulnerable to this attack as we could not possibly test all existing VPN providers.

Details about the leak

The attacker needs to meet the following requirements:

  • Has an active account at the same VPN provider as the victim
  • Knows victim’s VPN exit IP address (can be obtained by various means, e.g. IRC or torrent client or by making the victim visit a website under the attackers control)
  • The attacker sets up port forwarding. It makes no difference whether the victim has port forwarding activated or not.

The IP leak can then be triggered as follows:

  1. Victim is connected to VPN server 1.2.3.4
  2. Victim’s routing table will look something like this:
    0.0.0.0/0 -> 10.0.0.1 (internal vpn gateway ip)
    1.2.3.4/32 -> 192.168.0.1 (old default gateway)
  3. Attacker connects to same server 1.2.3.4 (knows victim’s exit through IRC or other means)
  4. Attacker activates Port Forwarding on server 1.2.3.4, example port 12345
  5. Attacker gets the victim to visit 1.2.3.4:12345 (for example via embedding <img src=”http://1.2.3.4:12345/x.jpg”> on a website)
  6. This connection will reveal the victim’s real IP to the attacker because of the “1.2.3.4/32 -> 192.168.0.1” vpn route.

The crucial issue here is that a VPN user connecting to his own VPN server will use his default route with his real IP address, as this is required for the VPN connection to work. If another user (the attacker) has port forwarding activated for his account on the same server, he can find out the real IP addresses of any user on the same VPN server by tricking him into visiting a link that redirects the traffic to a port under his control.

Also note that due to the nature of this attack all VPN protocols (IPSec, OpenVPN, PPTP, etc.) and all operating systems are affected.

Mitigation

Affected VPN providers should implement one of the following:

  • Have multiple IP addresses, allow incoming connections to ip1, exit connections through ip2-ipx, have portforwardings on ip2-ipx
  • On Client connect set server side firewall rule to block access from Client real ip to portforwardings that are not his own.

 

Source: Perfect Privacy

Categories: Réseau Tags: , ,

Pipes and redirection

26/11/2015 Comments off

Many system administrators seem to have problems with the concepts of pipes and redirection in a shell. A coworker recently asked me how to deal with log files. How to find the information he was looking for. This article tries to shed some light on it.

Input / Output of shell commands

Many of the basic Linux/UNIX shell commands work in a similar way. Every command that you start from the shell gets three channels assigned:

  • STDIN (channel 0):
    Where your command draws the input from. If you don’t specify anything special this will be your keyboard input.
  • STDOUT (channel 1):
    Where your command’s output is sent to. If you don’t specify anything special the output is displayed in your shell.
  • STDERR (channel 2):
    If anything wrong happens the command will send error message here. By default the output is also displayed in your shell.

Try it yourself. The most basic command that just passes everything through from STDIN to STDOUT is the ‘cat’ command. Just open a shell and type ‘cat’ and press Enter. Nothing seems to happen. But actually ‘cat’ is waiting for input. Type something like “hello world”. Every time you press ‘Enter’ after a line ‘cat’ will output your input. So you will get an echo of everything you type. To let ‘cat’ know that you are done with the input send it an ‘end-of-file’ (EOF) signal by pressing Ctrl-D on an empty line.

The pipe(line)

A more interesting application of the STDIN/STDOUT is to chain commands together. The output of the first command becomes the input of the second command. Imagine the following chain:

grep-pipeline

The contents of the file /var/log/syslog are sent (as input) to the grep command. grep will filter the stream for lines containing the word ‘postfix’ and output that. Now the next grep picks up what was filtered and filter it further for the word ‘removed’. So now we have only lines containing both ‘postfix’ and ‘removed’. And finally these lines are sent to ‘wc -l’ which is a shell command counting the lines of some input. In my case it found 27 of such lines and printed that number to my shell. In shell syntax this reads:

cat /var/log/syslog | grep 'postfix' | grep 'removed' | wc -l

The ‘|’ character is called pipe. A sequence of such commands joined together with pipes are called pipeline.

Useless use of ‘cat’

Actually ‘cat’ is supposed to be used for concatenating files. Like “cat file1 file2”. But some administrators abuse the command to put something into a pipeline. That’s bad style and the reason why Randal L. Schwartz (a seasoned programmer) used to hand out virtual “Useless use of cat” awards. Shell commands usually can take a filename as the last argument as an input. So this would be right:

grep something /var/log/syslog | wc -l

While this works but is considered bad style:

cat /var/log/syslog | grep something | wc

Or if you knew that grep even has a “-c” option to count lines the whole task could be done with just grep:

grep -c something /var/log/syslog

Lire la suite…

Categories: Système Tags: , ,

Layer 7 DDOS – Blocking HTTP Flood Attacks

26/11/2015 Comments off

There are many types of Distributed Denial of Service (DDOS) attacks that can affect and bring down a website, and they vary in complexity and size. The most well known attacks are the good old SYN-flood, followed by the Layer 3/4 UDP and DNS amplification attacks.

Today though, we’re going to spend a little time looking at Layer 7, or what we call an HTTP Flood Attack.

An HTTP flood attack is a type of Layer 7 application attack that utilizes the standard valid GET/POST requests used to fetch information, as in typical URL data retrievals (images, information, etc.) during SSL sessions. An HTTP GET/POST flood is a volumetric attack that does not use malformed packets, spoofing or reflection techniques. – DDoSAttacks.biz

If you’re wondering, yes, we deal with these every day, and we protect our client websites via our Website Firewall.

Today I’m going to share with you some details on a rather large DDoS attack that leveraged the following HTTP request flood attack to wreak havoc on a clients website. I’ll also share the steps we took to mitigate the issue.

Layer 7 DDoS – HTTP Flood Attacks

The first thing to understand about Layer 7 attacks is that they require more understanding about the website and how it operates. The attacker has to do some homework and create a specially crafted attack to achieve their goal. Because of this, these types of DDoS attacks require less bandwidth to take the site down and are harder to detect and block.

Layer 7 DDoS – Part 1: Random URLs

This specific client came to us after his site was down for almost a week. They tried other services to protect their website with not much luck. As soon as he switched his DNS to us, we gained a much deeper appreciation and started to see why.

He was getting thousands of requests like these every second:

75.118.29.205 - - [20/Jan/2014:19:32:06 -0500] "GET /?458739416183768700 HTTP/1.1" 200 440 "http://movies.netflix.com/WiPlayer?movieid=70136122&trkid=7882978&t=Weeds" ""
173.245.56.201 - - [20/Jan/2014:19:32:06 -0500] "GET /?458726993617499500 HTTP/1.1" 200 440 "http://landing.pcwhatsap.com/1/?offer_id=3534&aff=1788&url_id=5618&sub_id=whatsapp_pop" ""
79.19.41.22 - - [20/Jan/2014:19:32:06 -0500] "GET /?458741338856272200 HTTP/1.1" 200 440 "http://www.rumoreweb.it/index.php?option=com_news_portal&view=category&id=21&Itemid=259" ""
190.121.64.3 - - [20/Jan/2014:19:32:06 -0500] "GET /?458722169268652700 HTTP/1.1" 200 440 "http://www.eldivisadero.cl/noticias/?task=show&id=37352" ""
186.54.141.146 - - [20/Jan/2014:19:32:06 -0500] "GET /?458741274224646000 HTTP/1.1" 200 440 "http://badoo.com/01244944965/?r=37.4&p=1" ""

To be more exact, he was getting 5,233 HTTP requests every single second. From different IP addresses around the world.

What is important to note here is how this worked against the client’s platform. The client’s website was built on WordPress. The uniqueness of the requests were bypassing the caching system, forcing the system to render and respond to every request. This was bringing about system failures as the server quickly became overwhelmed by the requests.

For illustration purposes, here is a quick geographic distribution of the IP’s hitting the site. This is for 1 second in the attack. Yes, every second these IP’s were changing.

Stopping the DDoS: Once we identified the type of attack, blocking was easy enough. By default, they were not passing our anomaly check, causing the requests to get blocked at the firewall. One of the many anomalies we look for are valid user agents, and if you look carefully you see that the requests didn’t have one. Hopefully, you’ll also noticed that the referrers were dynamic and the packets were the same size, another very interesting signature. Needless to say, this triggered one of our rules, and within minutes his site was back and the attack blocked.

Lire la suite…

Nmon – A nifty little tool to monitor system resources on Linux

26/11/2015 Comments off

Source: binarytides.com

Nmon

Nmon (Nigel’s performance Monitor for Linux) is another very useful command line utility that can display information about various system resources like cpu, memory, disk, network etc. It was developed at IBM and later released open source.

It is available for most common architectures like x86, ARM and platforms like linux, unix etc. It is interactive and the output is well organised similar to htop.

Using Nmon it is possible to view the performance of different system resources on a single screen.
The man page describes nmon as

nmon is is a systems administrator, tuner, benchmark tool. It can display the CPU, memory, network, disks (mini graphs or numbers), file systems, NFS, top processes, resources (Linux version & processors) and on Power micro-partition information.

Project website
http://nmon.sourceforge.net/

Install Nmon

Debian/Ubuntu type distros have nmon in the default repos, so grab it with apt.

$ sudo apt-get install nmon

Fedora users can get it with yum

$ sudo yum install nmon

CentOS users need to install nmon from rpmforge/repoforge repository. It is not present in Epel.
Either download the correct rpm installer from

http://pkgs.repoforge.org/nmon/

Or setup the rpmforge repository by following the instructions here
http://wiki.centos.org/AdditionalResources/Repositories/RPMForge

And then install using yum

$ sudo yum install nmon

Lire la suite…

Categories: Système Tags: ,

Glances gives a quick overview of system usage on Linux

26/11/2015 Comments off

Monitor your Linux system

glances system linuxAs a Linux sysadmin it feels great power when monitoring system resources like cpu, memory on the commandline. To peek inside the system is a good habit here atleast, because that’s one way of driving your Linux system safe. Plenty of tools like Htop, Nmon, Collectl, top and iotop etc help you accomplish the task. Today lets try another tool called Glances.

Glances

Glances is a tool similar to Nmon that has a very compact display to provide a complete overview of different system resources on just a single screen area. It does not support any complex functionality but just gives a brief overview CPU, Load, Memory, Network rate, Disk IO, file system, process number and details.

As a bonus, glances is actually cross platform, which means you can use it on obsolete OSes like windows :P.

Here’s a quick glimpse of it.

glances-linux

The output is color highlighted. Green indicates optimum levels of usage whereas red indicates that the particular resource is under heavy use.

$ glances -v
Glances version 1.6 with PsUtil 0.6.1

Project homepage https://github.com/nicolargo/glances http://nicolargo.github.io/glances/

Lire la suite…

MySQL Query Profiling

26/11/2015 Comments off

You can profile a query by doing following:

mysql> SET SESSION profiling = 1;
mysql> USE database_name;
mysql> SELECT * FROM table WHERE column = 'value';
mysql> SHOW PROFILES;

First line enables profiling for current mysql interactive session only. Global profiling is not recommended.

Second line selects database on which we need to fire query.

Third line is actual query. (do not use EXPLAIN here).

Fourth line shows list of recorded profiles. It’s output looks like:


mysql> SHOW PROFILES;
+----------+------------+-----------------------------------------------------------------------------------------------------------------------------------+
| Query_ID | Duration   | Query                                                                                                                             |
+----------+------------+-----------------------------------------------------------------------------------------------------------------------------------+
|        1 | 0.00008050 | SELECT DATABASE()                                                                                                                 |
|        2 | 0.00034975 | show databases                                                                                                                    |
|        3 | 0.00073850 | show tables                                                                                                                       |
|        4 | 0.00040525 | SELECT * From wp_terms wt INNER JOIN wp_term_taxonomy wtt ON wt.term_id=wtt.term_id WHERE wtt.taxonomy='post_tag' AND wtt.count=0 |
+----------+------------+-----------------------------------------------------------------------------------------------------------------------------------+
4 rows in set (0.00 sec)

Above is list of queries profiled in current session.

You can get execution time breakdown by running another mysql query:

SELECT * FROM INFORMATION_SCHEMA.PROFILING WHERE QUERY_ID=4;

It will print data like:


mysql> SELECT * FROM INFORMATION_SCHEMA.PROFILING WHERE QUERY_ID=4;
+----------+-----+--------------------------------+----------+----------+------------+-------------------+---------------------+--------------+---------------+---------------+-------------------+-------------------+-------------------+-------+-----------------------+---------------+-------------+
| QUERY_ID | SEQ | STATE                          | DURATION | CPU_USER | CPU_SYSTEM | CONTEXT_VOLUNTARY | CONTEXT_INVOLUNTARY | BLOCK_OPS_IN | BLOCK_OPS_OUT | MESSAGES_SENT | MESSAGES_RECEIVED | PAGE_FAULTS_MAJOR | PAGE_FAULTS_MINOR | SWAPS | SOURCE_FUNCTION       | SOURCE_FILE   | SOURCE_LINE |
+----------+-----+--------------------------------+----------+----------+------------+-------------------+---------------------+--------------+---------------+---------------+-------------------+-------------------+-------------------+-------+-----------------------+---------------+-------------+
|        4 |   2 | starting                       | 0.000014 |     NULL |       NULL |              NULL |                NULL |         NULL |          NULL |          NULL |              NULL |              NULL |              NULL |  NULL | NULL                  | NULL          |        NULL |
|        4 |   3 | Waiting for query cache lock   | 0.000002 |     NULL |       NULL |              NULL |                NULL |         NULL |          NULL |          NULL |              NULL |              NULL |              NULL |  NULL | try_lock              | sql_cache.cc  |         646 |
|        4 |   4 | Waiting on query cache mutex   | 0.000001 |     NULL |       NULL |              NULL |                NULL |         NULL |          NULL |          NULL |              NULL |              NULL |              NULL |  NULL | try_lock              | sql_cache.cc  |         650 |
|        4 |   5 | checking query cache for query | 0.000057 |     NULL |       NULL |              NULL |                NULL |         NULL |          NULL |          NULL |              NULL |              NULL |              NULL |  NULL | send_result_to_client | sql_cache.cc  |        1853 |
|        4 |   6 | checking permissions           | 0.000003 |     NULL |       NULL |              NULL |                NULL |         NULL |          NULL |          NULL |              NULL |              NULL |              NULL |  NULL | check_access          | sql_parse.cc  |        5007 |
|        4 |   7 | checking permissions           | 0.000004 |     NULL |       NULL |              NULL |                NULL |         NULL |          NULL |          NULL |              NULL |              NULL |              NULL |  NULL | check_access          | sql_parse.cc  |        5007 |
|        4 |   8 | Opening tables                 | 0.000046 |     NULL |       NULL |              NULL |                NULL |         NULL |          NULL |          NULL |              NULL |              NULL |              NULL |  NULL | open_tables           | sql_base.cc   |        4944 |
|        4 |   9 | System lock                    | 0.000009 |     NULL |       NULL |              NULL |                NULL |         NULL |          NULL |          NULL |              NULL |              NULL |              NULL |  NULL | mysql_lock_tables     | lock.cc       |         299 |
|        4 |  10 | Waiting for query cache lock   | 0.000001 |     NULL |       NULL |              NULL |                NULL |         NULL |          NULL |          NULL |              NULL |              NULL |              NULL |  NULL | try_lock              | sql_cache.cc  |         646 |
|        4 |  11 | Waiting on query cache mutex   | 0.000021 |     NULL |       NULL |              NULL |                NULL |         NULL |          NULL |          NULL |              NULL |              NULL |              NULL |  NULL | try_lock              | sql_cache.cc  |         650 |
|        4 |  12 | init                           | 0.000032 |     NULL |       NULL |              NULL |                NULL |         NULL |          NULL |          NULL |              NULL |              NULL |              NULL |  NULL | mysql_select          | sql_select.cc |        2622 |
|        4 |  13 | optimizing                     | 0.000015 |     NULL |       NULL |              NULL |                NULL |         NULL |          NULL |          NULL |              NULL |              NULL |              NULL |  NULL | optimize              | sql_select.cc |         889 |
|        4 |  14 | statistics                     | 0.000057 |     NULL |       NULL |              NULL |                NULL |         NULL |          NULL |          NULL |              NULL |              NULL |              NULL |  NULL | optimize              | sql_select.cc |        1099 |
|        4 |  15 | preparing                      | 0.000017 |     NULL |       NULL |              NULL |                NULL |         NULL |          NULL |          NULL |              NULL |              NULL |              NULL |  NULL | optimize              | sql_select.cc |        1121 |
|        4 |  16 | executing                      | 0.000002 |     NULL |       NULL |              NULL |                NULL |         NULL |          NULL |          NULL |              NULL |              NULL |              NULL |  NULL | exec                  | sql_select.cc |        1879 |
|        4 |  17 | Sending data                   | 0.000099 |     NULL |       NULL |              NULL |                NULL |         NULL |          NULL |          NULL |              NULL |              NULL |              NULL |  NULL | exec                  | sql_select.cc |        2423 |
|        4 |  18 | end                            | 0.000003 |     NULL |       NULL |              NULL |                NULL |         NULL |          NULL |          NULL |              NULL |              NULL |              NULL |  NULL | mysql_select          | sql_select.cc |        2658 |
|        4 |  19 | query end                      | 0.000003 |     NULL |       NULL |              NULL |                NULL |         NULL |          NULL |          NULL |              NULL |              NULL |              NULL |  NULL | mysql_execute_command | sql_parse.cc  |        4686 |
|        4 |  20 | closing tables                 | 0.000007 |     NULL |       NULL |              NULL |                NULL |         NULL |          NULL |          NULL |              NULL |              NULL |              NULL |  NULL | mysql_execute_command | sql_parse.cc  |        4738 |
|        4 |  21 | freeing items                  | 0.000011 |     NULL |       NULL |              NULL |                NULL |         NULL |          NULL |          NULL |              NULL |              NULL |              NULL |  NULL | mysql_parse           | sql_parse.cc  |        5931 |
|        4 |  22 | logging slow query             | 0.000001 |     NULL |       NULL |              NULL |                NULL |         NULL |          NULL |          NULL |              NULL |              NULL |              NULL |  NULL | log_slow_statement    | sql_parse.cc  |        1625 |
|        4 |  23 | cleaning up                    | 0.000002 |     NULL |       NULL |              NULL |                NULL |         NULL |          NULL |          NULL |              NULL |              NULL |              NULL |  NULL | dispatch_command      | sql_parse.cc  |        1475 |
+----------+-----+--------------------------------+----------+----------+------------+-------------------+---------------------+--------------+---------------+---------------+-------------------+-------------------+-------------------+-------+-----------------------+---------------+-------------+
22 rows in set (0.00 sec)

Source: rtcamp.com