Archive

Archives pour 09/2017

Linux: 20 Iptables Examples For New SysAdmins

28/09/2017 Comments off

According to the official project site:

netfilter is a set of hooks inside the Linux kernel that allows kernel modules to register callback functions with the network stack. A registered callback function is then called back for every packet that traverses the respective hook within the network stack.

This Linux based firewall is controlled by the program called iptables to handles filtering for IPv4, and ip6tables handles filtering for IPv6. I strongly recommend that you first read our quick tutorial that explains how to configure a host-based firewall called Netfilter (iptables) under CentOS / RHEL / Fedora / Redhat Enterprise Linux. This post lists most simple iptables solutions required by a new Linux user to secure his or her Linux operating system from intruders.

IPTABLES Rules Example

  • Most of the actions listed in this post written with the assumption that they will be executed by the root user running the bash or any other modern shell. Do not type commands on the remote system as it will disconnect your access.
  • For demonstration purpose, I’ve used RHEL 6.x, but the following command should work with any modern Linux distro that use the netfliter.
  • It is NOT a tutorial on how to set iptables. See tutorial here. It is a quick cheat sheet to common iptables commands.

#1: Displaying the Status of Your Firewall

Type the following command as root:
# iptables -L -n -v
Sample outputs:

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Above output indicates that the firewall is not active. The following sample shows an active firewall:
# iptables -L -n -v
Sample outputs:

Chain INPUT (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0           state INVALID
  394 43586 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0           state RELATED,ESTABLISHED
   93 17292 ACCEPT     all  --  br0    *       0.0.0.0/0            0.0.0.0/0
    1   142 ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0

Chain FORWARD (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 ACCEPT     all  --  br0    br0     0.0.0.0/0            0.0.0.0/0
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0           state INVALID
    0     0 TCPMSS     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           tcp flags:0x06/0x02 TCPMSS clamp to PMTU
    0     0 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0           state RELATED,ESTABLISHED
    0     0 wanin      all  --  vlan2  *       0.0.0.0/0            0.0.0.0/0
    0     0 wanout     all  --  *      vlan2   0.0.0.0/0            0.0.0.0/0
    0     0 ACCEPT     all  --  br0    *       0.0.0.0/0            0.0.0.0/0

Chain OUTPUT (policy ACCEPT 425 packets, 113K bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain wanin (1 references)
 pkts bytes target     prot opt in     out     source               destination

Chain wanout (1 references)
 pkts bytes target     prot opt in     out     source               destination

Where,

  • -L : List rules.
  • -v : Display detailed information. This option makes the list command show the interface name, the rule options, and the TOS masks. The packet and byte counters are also listed, with the suffix ‘K’, ‘M’ or ‘G’ for 1000, 1,000,000 and 1,000,000,000 multipliers respectively.
  • -n : Display IP address and port in numeric format. Do not use DNS to resolve names. This will speed up listing.

#1.1: To inspect firewall with line numbers, enter:

# iptables -n -L -v --line-numbers
Sample outputs:

Chain INPUT (policy DROP)
num  target     prot opt source               destination
1    DROP       all  --  0.0.0.0/0            0.0.0.0/0           state INVALID
2    ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           state RELATED,ESTABLISHED
3    ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0
4    ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0

Chain FORWARD (policy DROP)
num  target     prot opt source               destination
1    ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0
2    DROP       all  --  0.0.0.0/0            0.0.0.0/0           state INVALID
3    TCPMSS     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp flags:0x06/0x02 TCPMSS clamp to PMTU
4    ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           state RELATED,ESTABLISHED
5    wanin      all  --  0.0.0.0/0            0.0.0.0/0
6    wanout     all  --  0.0.0.0/0            0.0.0.0/0
7    ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0

Chain OUTPUT (policy ACCEPT)
num  target     prot opt source               destination

Chain wanin (1 references)
num  target     prot opt source               destination

Chain wanout (1 references)
num  target     prot opt source               destination

You can use line numbers to delete or insert new rules into the firewall.

#1.2: To display INPUT or OUTPUT chain rules, enter:

# iptables -L INPUT -n -v
# iptables -L OUTPUT -n -v --line-numbers

#2: Stop / Start / Restart the Firewall

If you are using CentOS / RHEL / Fedora Linux, enter:
# service iptables stop
# service iptables start
# service iptables restart

You can use the iptables command itself to stop the firewall and delete all rules:
# iptables -F
# iptables -X
# iptables -t nat -F
# iptables -t nat -X
# iptables -t mangle -F
# iptables -t mangle -X
# iptables -P INPUT ACCEPT
# iptables -P OUTPUT ACCEPT
# iptables -P FORWARD ACCEPT

Where,

  • -F : Deleting (flushing) all the rules.
  • -X : Delete chain.
  • -t table_name : Select table (called nat or mangle) and delete/flush rules.
  • -P : Set the default policy (such as DROP, REJECT, or ACCEPT).

#3: Delete Firewall Rules

To display line number along with other information for existing rules, enter:
# iptables -L INPUT -n --line-numbers
# iptables -L OUTPUT -n --line-numbers
# iptables -L OUTPUT -n --line-numbers | less
# iptables -L OUTPUT -n --line-numbers | grep 202.54.1.1

You will get the list of IP. Look at the number on the left, then use number to delete it. For example delete line number 4, enter:
# iptables -D INPUT 4
OR find source IP 202.54.1.1 and delete from rule:
# iptables -D INPUT -s 202.54.1.1 -j DROP
Where,

  • -D : Delete one or more rules from the selected chain

#4: Insert Firewall Rules

To insert one or more rules in the selected chain as the given rule number use the following syntax. First find out line numbers, enter:
# iptables -L INPUT -n –line-numbers
Sample outputs:

Chain INPUT (policy DROP)
num  target     prot opt source               destination
1    DROP       all  --  202.54.1.1           0.0.0.0/0
2    ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           state NEW,ESTABLISHED 

To insert rule between 1 and 2, enter:
# iptables -I INPUT 2 -s 202.54.1.2 -j DROP
To view updated rules, enter:
# iptables -L INPUT -n --line-numbers
Sample outputs:

Chain INPUT (policy DROP)
num  target     prot opt source               destination
1    DROP       all  --  202.54.1.1           0.0.0.0/0
2    DROP       all  --  202.54.1.2           0.0.0.0/0
3    ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           state NEW,ESTABLISHED

#5: Save Firewall Rules

To save firewall rules under CentOS / RHEL / Fedora Linux, enter:
# service iptables save
In this example, drop an IP and save firewall rules:
# iptables -A INPUT -s 202.5.4.1 -j DROP
# service iptables save

For all other distros use the iptables-save command:
# iptables-save > /root/my.active.firewall.rules
# cat /root/my.active.firewall.rules

#6: Restore Firewall Rules

To restore firewall rules form a file called /root/my.active.firewall.rules, enter:
# iptables-restore < /root/my.active.firewall.rules
To restore firewall rules under CentOS / RHEL / Fedora Linux, enter:
# service iptables restart

#7: Set the Default Firewall Policies

To drop all traffic:
# iptables -P INPUT DROP
# iptables -P OUTPUT DROP
# iptables -P FORWARD DROP
# iptables -L -v -n
#### you will not able to connect anywhere as all traffic is dropped ###
# ping cyberciti.biz
# wget http://www.kernel.org/pub/linux/kernel/v3.0/testing/linux-3.2-rc5.tar.bz2

#7.1: Only Block Incoming Traffic

To drop all incoming / forwarded packets, but allow outgoing traffic, enter:
# iptables -P INPUT DROP
# iptables -P FORWARD DROP
# iptables -P OUTPUT ACCEPT
# iptables -A INPUT -m state --state NEW,ESTABLISHED -j ACCEPT
# iptables -L -v -n
### *** now ping and wget should work *** ###
# ping cyberciti.biz
# wget http://www.kernel.org/pub/linux/kernel/v3.0/testing/linux-3.2-rc5.tar.bz2

#8:Drop Private Network Address On Public Interface

IP spoofing is nothing but to stop the following IPv4 address ranges for private networks on your public interfaces. Packets with non-routable source addresses should be rejected using the following syntax:
# iptables -A INPUT -i eth1 -s 192.168.0.0/24 -j DROP
# iptables -A INPUT -i eth1 -s 10.0.0.0/8 -j DROP

#8.1: IPv4 Address Ranges For Private Networks (make sure you block them on public interface)

  • 10.0.0.0/8 -j (A)
  • 172.16.0.0/12 (B)
  • 192.168.0.0/16 (C)
  • 224.0.0.0/4 (MULTICAST D)
  • 240.0.0.0/5 (E)
  • 127.0.0.0/8 (LOOPBACK)

#9: Blocking an IP Address (BLOCK IP)

To block an attackers ip address called 1.2.3.4, enter:
# iptables -A INPUT -s 1.2.3.4 -j DROP
# iptables -A INPUT -s 192.168.0.0/24 -j DROP

#10: Block Incoming Port Requests (BLOCK PORT)

To block all service requests on port 80, enter:
# iptables -A INPUT -p tcp --dport 80 -j DROP
# iptables -A INPUT -i eth1 -p tcp --dport 80 -j DROP

To block port 80 only for an ip address 1.2.3.4, enter:
# iptables -A INPUT -p tcp -s 1.2.3.4 --dport 80 -j DROP
# iptables -A INPUT -i eth1 -p tcp -s 192.168.1.0/24 --dport 80 -j DROP

#11: Block Outgoing IP Address

To block outgoing traffic to a particular host or domain such as cyberciti.biz, enter:
# host -t a cyberciti.biz
Sample outputs:

cyberciti.biz has address 75.126.153.206

Note down its ip address and type the following to block all outgoing traffic to 75.126.153.206:
# iptables -A OUTPUT -d 75.126.153.206 -j DROP
You can use a subnet as follows:
# iptables -A OUTPUT -d 192.168.1.0/24 -j DROP
# iptables -A OUTPUT -o eth1 -d 192.168.1.0/24 -j DROP

#11.1: Example – Block Facebook.com Domain

First, find out all ip address of facebook.com, enter:
# host -t a www.facebook.com
Sample outputs:

www.facebook.com has address 69.171.228.40

Find CIDR for 69.171.228.40, enter:
# whois 69.171.228.40 | grep CIDR
Sample outputs:

CIDR:           69.171.224.0/19

To prevent outgoing access to www.facebook.com, enter:
# iptables -A OUTPUT -p tcp -d 69.171.224.0/19 -j DROP
You can also use domain name, enter:
# iptables -A OUTPUT -p tcp -d www.facebook.com -j DROP
# iptables -A OUTPUT -p tcp -d facebook.com -j DROP

From the iptables man page:

… specifying any name to be resolved with a remote query such as DNS (e.g., facebook.com is a really bad idea), a network IP address (with /mask), or a plain IP address …

#12: Log and Drop Packets

Type the following to log and block IP spoofing on public interface called eth1
# iptables -A INPUT -i eth1 -s 10.0.0.0/8 -j LOG --log-prefix "IP_SPOOF A: "
# iptables -A INPUT -i eth1 -s 10.0.0.0/8 -j DROP

By default everything is logged to /var/log/messages file.
# tail -f /var/log/messages
# grep --color 'IP SPOOF' /var/log/messages

#13: Log and Drop Packets with Limited Number of Log Entries

The -m limit module can limit the number of log entries created per time. This is used to prevent flooding your log file. To log and drop spoofing per 5 minutes, in bursts of at most 7 entries .
# iptables -A INPUT -i eth1 -s 10.0.0.0/8 -m limit --limit 5/m --limit-burst 7 -j LOG --log-prefix "IP_SPOOF A: "
# iptables -A INPUT -i eth1 -s 10.0.0.0/8 -j DROP

#14: Drop or Accept Traffic From Mac Address

Use the following syntax:
# iptables -A INPUT -m mac --mac-source 00:0F:EA:91:04:08 -j DROP
## *only accept traffic for TCP port # 8080 from mac 00:0F:EA:91:04:07 * ##
# iptables -A INPUT -p tcp --destination-port 22 -m mac --mac-source 00:0F:EA:91:04:07 -j ACCEPT

#15: Block or Allow ICMP Ping Request

Type the following command to block ICMP ping requests:
# iptables -A INPUT -p icmp --icmp-type echo-request -j DROP
# iptables -A INPUT -i eth1 -p icmp --icmp-type echo-request -j DROP

Ping responses can also be limited to certain networks or hosts:
# iptables -A INPUT -s 192.168.1.0/24 -p icmp --icmp-type echo-request -j ACCEPT
The following only accepts limited type of ICMP requests:
### ** assumed that default INPUT policy set to DROP ** #############
iptables -A INPUT -p icmp --icmp-type echo-reply -j ACCEPT
iptables -A INPUT -p icmp --icmp-type destination-unreachable -j ACCEPT
iptables -A INPUT -p icmp --icmp-type time-exceeded -j ACCEPT
## ** all our server to respond to pings ** ##
iptables -A INPUT -p icmp --icmp-type echo-request -j ACCEPT

#16: Open Range of Ports

Use the following syntax to open a range of ports:
iptables -A INPUT -m state --state NEW -m tcp -p tcp --dport 7000:7010 -j ACCEPT 

#17: Open Range of IP Addresses

Use the following syntax to open a range of IP address:
## only accept connection to tcp port 80 (Apache) if ip is between 192.168.1.100 and 192.168.1.200 ##
iptables -A INPUT -p tcp --destination-port 80 -m iprange --src-range 192.168.1.100-192.168.1.200 -j ACCEPT

## nat example ##
iptables -t nat -A POSTROUTING -j SNAT --to-source 192.168.1.20-192.168.1.25

#18: Established Connections and Restaring The Firewall

When you restart the iptables service it will drop established connections as it unload modules from the system under RHEL / Fedora / CentOS Linux. Edit, /etc/sysconfig/iptables-config and set IPTABLES_MODULES_UNLOAD as follows:

IPTABLES_MODULES_UNLOAD = no

#19: Help Iptables Flooding My Server Screen

Use the crit log level to send messages to a log file instead of console:
iptables -A INPUT -s 1.2.3.4 -p tcp --destination-port 80 -j LOG --log-level crit

#20: Block or Open Common Ports

The following shows syntax for opening and closing common TCP and UDP ports:

Replace ACCEPT with DROP to block port:
## open port ssh tcp port 22 ##
iptables -A INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
iptables -A INPUT -s 192.168.1.0/24 -m state --state NEW -p tcp --dport 22 -j ACCEPT
 
## open cups (printing service) udp/tcp port 631 for LAN users ##
iptables -A INPUT -s 192.168.1.0/24 -p udp -m udp --dport 631 -j ACCEPT
iptables -A INPUT -s 192.168.1.0/24 -p tcp -m tcp --dport 631 -j ACCEPT
 
## allow time sync via NTP for lan users (open udp port 123) ##
iptables -A INPUT -s 192.168.1.0/24 -m state --state NEW -p udp --dport 123 -j ACCEPT
 
## open tcp port 25 (smtp) for all ##
iptables -A INPUT -m state --state NEW -p tcp --dport 25 -j ACCEPT
 
# open dns server ports for all ##
iptables -A INPUT -m state --state NEW -p udp --dport 53 -j ACCEPT
iptables -A INPUT -m state --state NEW -p tcp --dport 53 -j ACCEPT
 
## open http/https (Apache) server port to all ##
iptables -A INPUT -m state --state NEW -p tcp --dport 80 -j ACCEPT
iptables -A INPUT -m state --state NEW -p tcp --dport 443 -j ACCEPT
 
## open tcp port 110 (pop3) for all ##
iptables -A INPUT -m state --state NEW -p tcp --dport 110 -j ACCEPT
 
## open tcp port 143 (imap) for all ##
iptables -A INPUT -m state --state NEW -p tcp --dport 143 -j ACCEPT
 
## open access to Samba file server for lan users only ##
iptables -A INPUT -s 192.168.1.0/24 -m state --state NEW -p tcp --dport 137 -j ACCEPT
iptables -A INPUT -s 192.168.1.0/24 -m state --state NEW -p tcp --dport 138 -j ACCEPT
iptables -A INPUT -s 192.168.1.0/24 -m state --state NEW -p tcp --dport 139 -j ACCEPT
iptables -A INPUT -s 192.168.1.0/24 -m state --state NEW -p tcp --dport 445 -j ACCEPT
 
## open access to proxy server for lan users only ##
iptables -A INPUT -s 192.168.1.0/24 -m state --state NEW -p tcp --dport 3128 -j ACCEPT
 
## open access to mysql server for lan users only ##
iptables -I INPUT -p tcp --dport 3306 -j ACCEPT

#21: Restrict the Number of Parallel Connections To a Server Per Client IP

You can use connlimit module to put such restrictions. To allow 3 ssh connections per client host, enter:
# iptables -A INPUT -p tcp --syn --dport 22 -m connlimit --connlimit-above 3 -j REJECT

Set HTTP requests to 20:
# iptables -p tcp --syn --dport 80 -m connlimit --connlimit-above 20 --connlimit-mask 24 -j DROP
Where,

  1. –connlimit-above 3 : Match if the number of existing connections is above 3.
  2. –connlimit-mask 24 : Group hosts using the prefix length. For IPv4, this must be a number between (including) 0 and 32.

#22: HowTO: Use iptables Like a Pro

For more information about iptables, please see the manual page by typing man iptables from the command line:
$ man iptables
You can see the help using the following syntax too:
# iptables -h
To see help with specific commands and targets, enter:
# iptables -j DROP -h

#22.1: Testing Your Firewall

Find out if ports are open or not, enter:
# netstat -tulpn
Find out if tcp port 80 open or not, enter:
# netstat -tulpn | grep :80
If port 80 is not open, start the Apache, enter:
# service httpd start
Make sure iptables allowing access to the port 80:
# iptables -L INPUT -v -n | grep 80
Otherwise open port 80 using the iptables for all users:
# iptables -A INPUT -m state --state NEW -p tcp --dport 80 -j ACCEPT
# service iptables save

Use the telnet command to see if firewall allows to connect to port 80:
$ telnet www.cyberciti.biz 80
Sample outputs:

Trying 75.126.153.206...
Connected to www.cyberciti.biz.
Escape character is '^]'.
^]

telnet> quit
Connection closed.

You can use nmap to probe your own server using the following syntax:
$ nmap -sS -p 80 www.cyberciti.biz
Sample outputs:

Starting Nmap 5.00 ( http://nmap.org ) at 2011-12-13 13:19 IST
Interesting ports on www.cyberciti.biz (75.126.153.206):
PORT   STATE SERVICE
80/tcp open  http

Nmap done: 1 IP address (1 host up) scanned in 1.00 seconds

I also recommend you install and use sniffer such as tcpdupm and ngrep to test your firewall settings.

CONCLUSION:

This post only list basic rules for new Linux users. You can create and build more complex rules. This requires good understanding of TCP/IP, Linux kernel tuning via sysctl.conf, and good knowledge of your own setup. Stay tuned for next topics:

  • Stateful packet inspection.
  • Using connection tracking helpers.
  • Network address translation.
  • Layer 2 filtering.
  • Firewall testing tools.
  • Dealing with VPNs, DNS, Web, Proxy, and other protocols.
 

30 Handy Bash Shell Aliases For Linux / Unix / Mac OS X

28/09/2017 Comments off

An alias is nothing but the shortcut to commands. The alias command allows the user to launch any command or group of commands (including options and filenames) by entering a single word. Use alias command to display a list of all defined aliases. You can add user-defined aliases to ~/.bashrcfile. You can cut down typing time with these aliases, work smartly, and increase productivity at the command prompt.

More about aliases

The general syntax for the alias command for the bash shell is as follows:

Task: List aliases

Type the following command:

alias

Sample outputs:

alias ..='cd ..'
alias amazonbackup='s3backup'
alias apt-get='sudo apt-get'
...

By default alias command shows a list of aliases that are defined for the current user.

Task: Define / create an alias (bash syntax)

To create the alias use the following syntax:

alias name=value
alias name='command'
alias name='command arg1 arg2'
alias name='/path/to/script'
alias name='/path/to/script.pl arg1'

In this example, create the alias c for the commonly used clear command, which clears the screen, by typing the following command and then pressing the ENTER key:

alias c='clear'

Then, to clear the screen, instead of typing clear, you would only have to type the letter ‘c’ and press the [ENTER] key:

c

Task: Disable an alias temporarily (bash syntax)

An alias can be disabled temporarily using the following syntax:

## path/to/full/command
/usr/bin/clear
## call alias with a backslash ##
\c

Task: Remove an alias (bash syntax)

You need to use the command called unalias to remove aliases. Its syntax is as follows:

unalias aliasname

In this example, remove the alias c which was created in an earlier example:

unalias c

You also need to delete the alias from the ~/.bashrc file using a text editor (see next section).

Task: Make aliases permanent (bash syntax)

The alias c remains in effect only during the current login session. Once you logs out or reboot the system the alias c will be gone. To avoid this problem, add alias to your ~/.bashrc file, enter:

vi ~/.bashrc

The alias c for the current user can be made permanent by entering the following line:

alias c='clear'

Save and close the file. System-wide aliases (i.e. aliases for all users) can be put in the /etc/bashrc file. Please note that the alias command is built into a various shells including ksh, tcsh/csh, ash, bash and others.

A note about privileged access

You can add code as follows in ~/.bashrc:

# if user is not root, pass all commands via sudo #
if [ $UID -ne 0 ]; then
    alias reboot='sudo reboot'
    alias update='sudo apt-get upgrade'
fi

A note about os specific aliases

You can add code as follows in ~/.bashrc using the case statement:

### Get os name via uname ###
_myos="$(uname)"
 
### add alias as per os using $_myos ###
case $_myos in
   Linux) alias foo='/path/to/linux/bin/foo';;
   FreeBSD|OpenBSD) alias foo='/path/to/bsd/bin/foo' ;;
   SunOS) alias foo='/path/to/sunos/bin/foo' ;;
   *) ;;
esac

30 uses for aliases

You can define various types aliases as follows to save time and increase productivity.

#1: Control ls command output

The ls command lists directory contents and you can colorize the output:

## Colorize the ls output ##
alias ls='ls --color=auto'
 
## Use a long listing format ##
alias ll='ls -la'
 
## Show hidden files ##
alias l.='ls -d .* --color=auto'

#2: Control cd command behavior

## get rid of command not found ##
alias cd..='cd ..'
 
## a quick way to get out of current directory ##
alias ..='cd ..'
alias ...='cd ../../../'
alias ....='cd ../../../../'
alias .....='cd ../../../../'
alias .4='cd ../../../../'
alias .5='cd ../../../../..'

#3: Control grep command output

grep command is a command-line utility for searching plain-text files for lines matching a regular expression:

## Colorize the grep command output for ease of use (good for log files)##
alias grep='grep --color=auto'
alias egrep='egrep --color=auto'
alias fgrep='fgrep --color=auto'

#4: Start calculator with math support

alias bc='bc -l'

#4: Generate sha1 digest

alias sha1='openssl sha1'

#5: Create parent directories on demand

mkdir command is used to create a directory:

alias mkdir='mkdir -pv'

#6: Colorize diff output

You can compare files line by line using diff and use a tool called colordiff to colorize diff output:

# install colordiff package 🙂
alias diff='colordiff'

#7: Make mount command output pretty and human readable format

alias mount='mount |column -t'

#8: Command short cuts to save time

# handy short cuts #
alias h='history'
alias j='jobs -l'

#9: Create a new set of commands

alias path='echo -e ${PATH//:/\\n}'
alias now='date +"%T"'
alias nowtime=now
alias nowdate='date +"%d-%m-%Y"'

#10: Set vim as default

alias vi=vim
alias svi='sudo vi'
alias vis='vim "+set si"'
alias edit='vim'

#11: Control output of networking tool called ping

# Stop after sending count ECHO_REQUEST packets #
alias ping='ping -c 5'
# Do not wait interval 1 second, go fast #
alias fastping='ping -c 100 -s.2'

#12: Show open ports

Use netstat command to quickly list all TCP/UDP port on the server:

alias ports='netstat -tulanp'

#13: Wakeup sleeping servers

Wake-on-LAN (WOL) is an Ethernet networking standard that allows a server to be turned on by a network message. You can quickly wakeup nas devices and server using the following aliases:

## replace mac with your actual server mac address #
alias wakeupnas01='/usr/bin/wakeonlan 00:11:32:11:15:FC'
alias wakeupnas02='/usr/bin/wakeonlan 00:11:32:11:15:FD'
alias wakeupnas03='/usr/bin/wakeonlan 00:11:32:11:15:FE'

#14: Control firewall (iptables) output

Netfilter is a host-based firewall for Linux operating systems. It is included as part of the Linux distribution and it is activated by default. This post list most common iptables solutions required by a new Linux user to secure his or her Linux operating system from intruders.

## shortcut for iptables and pass it via sudo#
alias ipt='sudo /sbin/iptables'
 
# display all rules #
alias iptlist='sudo /sbin/iptables -L -n -v --line-numbers'
alias iptlistin='sudo /sbin/iptables -L INPUT -n -v --line-numbers'
alias iptlistout='sudo /sbin/iptables -L OUTPUT -n -v --line-numbers'
alias iptlistfw='sudo /sbin/iptables -L FORWARD -n -v --line-numbers'
alias firewall=iptlist

#15: Debug web server / cdn problems with curl

# get web server headers #
alias header='curl -I'
 
# find out if remote server supports gzip / mod_deflate or not #
alias headerc='curl -I --compress'

#16: Add safety nets

# do not delete / or prompt if deleting more than 3 files at a time #
alias rm='rm -I --preserve-root'
 
# confirmation #
alias mv='mv -i'
alias cp='cp -i'
alias ln='ln -i'
 
# Parenting changing perms on / #
alias chown='chown --preserve-root'
alias chmod='chmod --preserve-root'
alias chgrp='chgrp --preserve-root'

#17: Update Debian Linux server

apt-get command is used for installing packages over the internet (ftp or http). You can also upgrade all packages in a single operations:

# distro specific - Debian / Ubuntu and friends #
# install with apt-get
alias apt-get="sudo apt-get"
alias updatey="sudo apt-get --yes"
 
# update on one command
alias update='sudo apt-get update && sudo apt-get upgrade'

#18: Update RHEL / CentOS / Fedora Linux server

yum command is a package management tool for RHEL / CentOS / Fedora Linux and friends:

## distrp specifc RHEL/CentOS ##
alias update='yum update'
alias updatey='yum -y update'

#19: Tune sudo and su

# become root #
alias root='sudo -i'
alias su='sudo -i'

#20: Pass halt/reboot via sudo

shutdown command bring the Linux / Unix system down:

# reboot / halt / poweroff
alias reboot='sudo /sbin/reboot'
alias poweroff='sudo /sbin/poweroff'
alias halt='sudo /sbin/halt'
alias shutdown='sudo /sbin/shutdown'

#21: Control web servers

# also pass it via sudo so whoever is admin can reload it without calling you #
alias nginxreload='sudo /usr/local/nginx/sbin/nginx -s reload'
alias nginxtest='sudo /usr/local/nginx/sbin/nginx -t'
alias lightyload='sudo /etc/init.d/lighttpd reload'
alias lightytest='sudo /usr/sbin/lighttpd -f /etc/lighttpd/lighttpd.conf -t'
alias httpdreload='sudo /usr/sbin/apachectl -k graceful'
alias httpdtest='sudo /usr/sbin/apachectl -t && /usr/sbin/apachectl -t -D DUMP_VHOSTS'

#22: Alias into our backup stuff

# if cron fails or if you want backup on demand just run these commands #
# again pass it via sudo so whoever is in admin group can start the job #
# Backup scripts #
alias backup='sudo /home/scripts/admin/scripts/backup/wrapper.backup.sh --type local --taget /raid1/backups'
alias nasbackup='sudo /home/scripts/admin/scripts/backup/wrapper.backup.sh --type nas --target nas01'
alias s3backup='sudo /home/scripts/admin/scripts/backup/wrapper.backup.sh --type nas --target nas01 --auth /home/scripts/admin/.authdata/amazon.keys'
alias rsnapshothourly='sudo /home/scripts/admin/scripts/backup/wrapper.rsnapshot.sh --type remote --target nas03 --auth /home/scripts/admin/.authdata/ssh.keys --config /home/scripts/admin/scripts/backup/config/adsl.conf'
alias rsnapshotdaily='sudo /home/scripts/admin/scripts/backup/wrapper.rsnapshot.sh --type remote --target nas03 --auth /home/scripts/admin/.authdata/ssh.keys --config /home/scripts/admin/scripts/backup/config/adsl.conf'
alias rsnapshotweekly='sudo /home/scripts/admin/scripts/backup/wrapper.rsnapshot.sh --type remote --target nas03 --auth /home/scripts/admin/.authdata/ssh.keys --config /home/scripts/admin/scripts/backup/config/adsl.conf'
alias rsnapshotmonthly='sudo /home/scripts/admin/scripts/backup/wrapper.rsnapshot.sh --type remote --target nas03 --auth /home/scripts/admin/.authdata/ssh.keys --config /home/scripts/admin/scripts/backup/config/adsl.conf'
alias amazonbackup=s3backup

#23: Desktop specific – play avi/mp3 files on demand

## play video files in a current directory ##
# cd ~/Download/movie-name
# playavi or vlc
alias playavi='mplayer *.avi'
alias vlc='vlc *.avi'
 
# play all music files from the current directory #
alias playwave='for i in *.wav; do mplayer "$i"; done'
alias playogg='for i in *.ogg; do mplayer "$i"; done'
alias playmp3='for i in *.mp3; do mplayer "$i"; done'
 
# play files from nas devices #
alias nplaywave='for i in /nas/multimedia/wave/*.wav; do mplayer "$i"; done'
alias nplayogg='for i in /nas/multimedia/ogg/*.ogg; do mplayer "$i"; done'
alias nplaymp3='for i in /nas/multimedia/mp3/*.mp3; do mplayer "$i"; done'
 
# shuffle mp3/ogg etc by default #
alias music='mplayer --shuffle *'

#24: Set default interfaces for sys admin related commands

vnstat is console-based network traffic monitor. dnstop is console tool to analyze DNS traffic. tcptrack and iftop commands displays information about TCP/UDP connections it sees on a network interface and display bandwidth usage on an interface by host respectively.

## All of our servers eth1 is connected to the Internets via vlan / router etc ##
alias dnstop='dnstop -l 5 eth1'
alias vnstat='vnstat -i eth1'
alias iftop='iftop -i eth1'
alias tcpdump='tcpdump -i eth1'
alias ethtool='ethtool eth1'
 
# work on wlan0 by default #
# Only useful for laptop as all servers are without wireless interface
alias iwconfig='iwconfig wlan0'

#25: Get system memory, cpu usage, and gpu memory info quickly

## pass options to free ##
alias meminfo='free -m -l -t'
 
## get top process eating memory
alias psmem='ps auxf | sort -nr -k 4'
alias psmem10='ps auxf | sort -nr -k 4 | head -10'
 
## get top process eating cpu ##
alias pscpu='ps auxf | sort -nr -k 3'
alias pscpu10='ps auxf | sort -nr -k 3 | head -10'
 
## Get server cpu info ##
alias cpuinfo='lscpu'
 
## older system use /proc/cpuinfo ##
##alias cpuinfo='less /proc/cpuinfo' ##
 
## get GPU ram on desktop / laptop##
alias gpumeminfo='grep -i --color memory /var/log/Xorg.0.log'

#26: Control Home Router

The curl command can be used to reboot Linksys routers.

# Reboot my home Linksys WAG160N / WAG54 / WAG320 / WAG120N Router / Gateway from *nix.
alias rebootlinksys="curl -u 'admin:my-super-password' 'http://192.168.1.2/setup.cgi?todo=reboot'"
 
# Reboot tomato based Asus NT16 wireless bridge
alias reboottomato="ssh admin@192.168.1.1 /sbin/reboot"

#27 Resume wget by default

The GNU Wget is a free utility for non-interactive download of files from the Web. It supports HTTP, HTTPS, and FTP protocols, and it can resume downloads too:

## this one saved by butt so many times ##
alias wget='wget -c'

#28 Use different browser for testing website

## this one saved by butt so many times ##
alias ff4='/opt/firefox4/firefox'
alias ff13='/opt/firefox13/firefox'
alias chrome='/opt/google/chrome/chrome'
alias opera='/opt/opera/opera'
 
#default ff
alias ff=ff13
 
#my default browser
alias browser=chrome

#29: A note about ssh alias

Do not create ssh alias, instead use ~/.ssh/config OpenSSH SSH client configuration files. It offers more option. An example:

Host server10
  Hostname 1.2.3.4
  IdentityFile ~/backups/.ssh/id_dsa
  user foobar
  Port 30000
  ForwardX11Trusted yes
  TCPKeepAlive yes

You can now connect to peer1 using the following syntax:
$ ssh server10

#30: It’s your turn to share…

## set some other defaults ##
alias df='df -H'
alias du='du -ch'
 
# top is atop, just like vi is vim
alias top='atop'
 
## nfsrestart - must be root ##
## refresh nfs mount / cache etc for Apache ##
alias nfsrestart='sync && sleep 2 && /etc/init.d/httpd stop && umount netapp2:/exports/http && sleep 2 && mount -o rw,sync,rsize=32768,wsize=32768,intr,hard,proto=tcp,fsc natapp2:/exports /http/var/www/html && /etc/init.d/httpd start'
 
## Memcached server status ##
alias mcdstats='/usr/bin/memcached-tool 10.10.27.11:11211 stats'
alias mcdshow='/usr/bin/memcached-tool 10.10.27.11:11211 display'
 
## quickly flush out memcached server ##
alias flushmcd='echo "flush_all" | nc 10.10.27.11 11211'
 
## Remove assets quickly from Akamai / Amazon cdn ##
alias cdndel='/home/scripts/admin/cdn/purge_cdn_cache --profile akamai'
alias amzcdndel='/home/scripts/admin/cdn/purge_cdn_cache --profile amazon'
 
## supply list of urls via file or stdin
alias cdnmdel='/home/scripts/admin/cdn/purge_cdn_cache --profile akamai --stdin'
alias amzcdnmdel='/home/scripts/admin/cdn/purge_cdn_cache --profile amazon --stdin'

Conclusion

This post summarizes several types of uses for *nix bash aliases:

  1. Setting default options for a command (e.g. set eth0 as default option for ethtool command via alias ethtool='ethtool eth0' ).
  2. Correcting typos (cd.. will act as cd .. via alias cd..='cd ..').
  3. Reducing the amount of typing.
  4. Setting the default path of a command that exists in several versions on a system (e.g. GNU/grep is located at /usr/local/bin/grep and Unix grep is located at /bin/grep. To use GNU grep use alias grep='/usr/local/bin/grep' ).
  5. Adding the safety nets to Unix by making commands interactive by setting default options. (e.g. rm, mv, and other commands).
  6. Compatibility by creating commands for older operating systems such as MS-DOS or other Unix like operating systems (e.g. alias del=rm ).

I’ve shared my aliases that I used over the years to reduce the need for repetitive command line typing. If you know and use any other bash/ksh/csh aliases that can reduce typing, share below in the comments.

Categories: Système Tags: , , ,

Top 32 Nmap Command Examples For Sys/Network Admins

28/09/2017 Comments off

The purpose of this post is to introduce a user to the nmap command line tool to scan a host and/or network, so to find out the possible vulnerable points in the hosts. You will also learn how to use Nmap for offensive and defensive purposes.

nmap in actionnmap in action

More about nmap

From the man page:

Nmap (“Network Mapper”) is an open source tool for network exploration and security auditing. It was designed to rapidly scan large networks, although it works fine against single hosts. Nmap uses raw IP packets in novel ways to determine what hosts are available on the network, what services (application name and version) those hosts are offering, what operating systems (and OS versions) they are running, what type of packet filters/firewalls are in use, and dozens of other characteristics. While Nmap is commonly used for security audits, many systems and network administrators find it useful for routine tasks such as network inventory, managing service upgrade schedules, and monitoring host or service uptime.

It was originally written by Gordon Lyon and it can answer the following questions easily:

  1. What computers did you find running on the local network?
  2. What IP addresses did you find running on the local network?
  3. What is the operating system of your target machine?
  4. Find out what ports are open on the machine that you just scanned?
  5. Find out if the system is infected with malware or virus.
  6. Search for unauthorized servers or network service on your network.
  7. Find and remove computers which don’t meet the organization’s minimum level of security.

Sample setup (LAB)

Port scanning may be illegal in some jurisdictions. So setup a lab as follows:

                              +---------+
        +---------+           | Network |         +--------+
        | server1 |-----------+ swtich  +---------|server2 |
        +---------+           | (sw0)   |         +--------+
                              +----+----+
                                   | 
                                   |
                         +---------+----------+
                         | wks01 Linux/OSX    |
                         +--------------------+

Where,

  • wks01 is your computer either running Linux/OS X or Unix like operating system. It is used for scanning your local network. The nmap command must be installed on this computer.
  • server1 can be powered by Linux / Unix / MS-Windows operating systems. This is an unpatched server. Feel free to install a few services such as a web-server, file server and so on.
  • server2 can be powered by Linux / Unix / MS-Windows operating systems. This is a fully patched server with firewall. Again, feel free to install few services such as a web-server, file server and so on.
  • All three systems are connected via switch.

How do I install nmap?

See:

  1. Debian / Ubuntu Linux: Install nmap Software For Scanning Network
  2. CentOS / RHEL: Install nmap Network Security Scanner
  3. OpenBSD: Install nmap Network Security Scanner

#1: Scan a single host or an IP address (IPv4)

### Scan a single ip address ###
nmap 192.168.1.1
 
## Scan a host name ###
nmap server1.cyberciti.biz
 
## Scan a host name with more info###
nmap -v server1.cyberciti.biz

Sample outputs:

Fig.01: nmap outputFig.01: nmap output

 

#2: Scan multiple IP address or subnet (IPv4)

nmap 192.168.1.1 192.168.1.2 192.168.1.3
## works with same subnet i.e. 192.168.1.0/24 
nmap 192.168.1.1,2,3

You can scan a range of IP address too:

nmap 192.168.1.1-20

You can scan a range of IP address using a wildcard:

nmap 192.168.1.*

Finally, you scan an entire subnet:

nmap 192.168.1.0/24

#3: Read list of hosts/networks from a file (IPv4)

The -iL option allows you to read the list of target systems using a text file. This is useful to scan a large number of hosts/networks. Create a text file as follows:
cat > /tmp/test.txt
Sample outputs:

server1.cyberciti.biz
192.168.1.0/24
192.168.1.1/24
10.1.2.3
localhost

The syntax is:

nmap -iL /tmp/test.txt

#4: Excluding hosts/networks (IPv4)

When scanning a large number of hosts/networks you can exclude hosts from a scan:

nmap 192.168.1.0/24 --exclude 192.168.1.5
nmap 192.168.1.0/24 --exclude 192.168.1.5,192.168.1.254

OR exclude list from a file called /tmp/exclude.txt

nmap -iL /tmp/scanlist.txt --excludefile /tmp/exclude.txt

#5: Turn on OS and version detection scanning script (IPv4)

nmap -A 192.168.1.254
nmap -v -A 192.168.1.1
nmap -A -iL /tmp/scanlist.txt 

#6: Find out if a host/network is protected by a firewall

nmap -sA 192.168.1.254
nmap -sA server1.cyberciti.biz

#7: Scan a host when protected by the firewall

nmap -PN 192.168.1.1
nmap -PN server1.cyberciti.biz

#8: Scan an IPv6 host/address

The -6 option enable IPv6 scanning. The syntax is:

nmap -6 IPv6-Address-Here
nmap -6 server1.cyberciti.biz
nmap -6 2607:f0d0:1002:51::4
nmap -v A -6 2607:f0d0:1002:51::4

#9: Scan a network and find out which servers and devices are up and running

This is known as host discovery or ping scan:

nmap -sP 192.168.1.0/24

Sample outputs:

Host 192.168.1.1 is up (0.00035s latency).
MAC Address: BC:AE:C5:C3:16:93 (Unknown)
Host 192.168.1.2 is up (0.0038s latency).
MAC Address: 74:44:01:40:57:FB (Unknown)
Host 192.168.1.5 is up.
Host nas03 (192.168.1.12) is up (0.0091s latency).
MAC Address: 00:11:32:11:15:FC (Synology Incorporated)
Nmap done: 256 IP addresses (4 hosts up) scanned in 2.80 second

#10: How do I perform a fast scan?

nmap -F 192.168.1.1

#11: Display the reason a port is in a particular state

nmap --reason 192.168.1.1
nmap --reason server1.cyberciti.biz

#12: Only show open (or possibly open) ports

nmap --open 192.168.1.1
nmap --open server1.cyberciti.biz

#13: Show all packets sent and received

nmap --packet-trace 192.168.1.1
nmap --packet-trace server1.cyberciti.biz

14#: Show host interfaces and routes

This is useful for debugging (ip command or route command or netstat commandlike output using nmap)

nmap --iflist

Sample outputs:

Starting Nmap 5.00 ( http://nmap.org ) at 2012-11-27 02:01 IST
************************INTERFACES************************
DEV    (SHORT)  IP/MASK          TYPE        UP MAC
lo     (lo)     127.0.0.1/8      loopback    up
eth0   (eth0)   192.168.1.5/24   ethernet    up B8:AC:6F:65:31:E5
vmnet1 (vmnet1) 192.168.121.1/24 ethernet    up 00:50:56:C0:00:01
vmnet8 (vmnet8) 192.168.179.1/24 ethernet    up 00:50:56:C0:00:08
ppp0   (ppp0)   10.1.19.69/32    point2point up
 
**************************ROUTES**************************
DST/MASK         DEV    GATEWAY
10.0.31.178/32   ppp0
209.133.67.35/32 eth0   192.168.1.2
192.168.1.0/0    eth0
192.168.121.0/0  vmnet1
192.168.179.0/0  vmnet8
169.254.0.0/0    eth0
10.0.0.0/0       ppp0
0.0.0.0/0        eth0   192.168.1.2

#15: How do I scan specific ports?

nmap -p [port] hostName
## Scan port 80
nmap -p 80 192.168.1.1
 
## Scan TCP port 80
nmap -p T:80 192.168.1.1
 
## Scan UDP port 53
nmap -p U:53 192.168.1.1
 
## Scan two ports ##
nmap -p 80,443 192.168.1.1
 
## Scan port ranges ##
nmap -p 80-200 192.168.1.1
 
## Combine all options ##
nmap -p U:53,111,137,T:21-25,80,139,8080 192.168.1.1
nmap -p U:53,111,137,T:21-25,80,139,8080 server1.cyberciti.biz
nmap -v -sU -sT -p U:53,111,137,T:21-25,80,139,8080 192.168.1.254
 
## Scan all ports with * wildcard ##
nmap -p "*" 192.168.1.1
 
## Scan top ports i.e. scan $number most common ports ##
nmap --top-ports 5 192.168.1.1
nmap --top-ports 10 192.168.1.1

Sample outputs:

Starting Nmap 5.00 ( http://nmap.org ) at 2012-11-27 01:23 IST
Interesting ports on 192.168.1.1:
PORT     STATE  SERVICE
21/tcp   closed ftp
22/tcp   open   ssh
23/tcp   closed telnet
25/tcp   closed smtp
80/tcp   open   http
110/tcp  closed pop3
139/tcp  closed netbios-ssn
443/tcp  closed https
445/tcp  closed microsoft-ds
3389/tcp closed ms-term-serv
MAC Address: BC:AE:C5:C3:16:93 (Unknown)
 
Nmap done: 1 IP address (1 host up) scanned in 0.51 seconds

#16: The fastest way to scan all your devices/computers for open ports ever

nmap -T5 192.168.1.0/24

#17: How do I detect remote operating system?

You can identify a remote host apps and OS using the -O option:

nmap -O 192.168.1.1
nmap -O  --osscan-guess 192.168.1.1
nmap -v -O --osscan-guess 192.168.1.1

Sample outputs:

Starting Nmap 5.00 ( http://nmap.org ) at 2012-11-27 01:29 IST
NSE: Loaded 0 scripts for scanning.
Initiating ARP Ping Scan at 01:29
Scanning 192.168.1.1 [1 port]
Completed ARP Ping Scan at 01:29, 0.01s elapsed (1 total hosts)
Initiating Parallel DNS resolution of 1 host. at 01:29
Completed Parallel DNS resolution of 1 host. at 01:29, 0.22s elapsed
Initiating SYN Stealth Scan at 01:29
Scanning 192.168.1.1 [1000 ports]
Discovered open port 80/tcp on 192.168.1.1
Discovered open port 22/tcp on 192.168.1.1
Completed SYN Stealth Scan at 01:29, 0.16s elapsed (1000 total ports)
Initiating OS detection (try #1) against 192.168.1.1
Retrying OS detection (try #2) against 192.168.1.1
Retrying OS detection (try #3) against 192.168.1.1
Retrying OS detection (try #4) against 192.168.1.1
Retrying OS detection (try #5) against 192.168.1.1
Host 192.168.1.1 is up (0.00049s latency).
Interesting ports on 192.168.1.1:
Not shown: 998 closed ports
PORT   STATE SERVICE
22/tcp open  ssh
80/tcp open  http
MAC Address: BC:AE:C5:C3:16:93 (Unknown)
Device type: WAP|general purpose|router|printer|broadband router
Running (JUST GUESSING) : Linksys Linux 2.4.X (95%), Linux 2.4.X|2.6.X (94%), MikroTik RouterOS 3.X (92%), Lexmark embedded (90%), Enterasys embedded (89%), D-Link Linux 2.4.X (89%), Netgear Linux 2.4.X (89%) Aggressive OS guesses: OpenWrt White Russian 0.9 (Linux 2.4.30) (95%), OpenWrt 0.9 - 7.09 (Linux 2.4.30 - 2.4.34) (94%), OpenWrt Kamikaze 7.09 (Linux 2.6.22) (94%), Linux 2.4.21 - 2.4.31 (likely embedded) (92%), Linux 2.6.15 - 2.6.23 (embedded) (92%), Linux 2.6.15 - 2.6.24 (92%), MikroTik RouterOS 3.0beta5 (92%), MikroTik RouterOS 3.17 (92%), Linux 2.6.24 (91%), Linux 2.6.22 (90%) No exact OS matches for host (If you know what OS is running on it, see http://nmap.org/submit/ ).
TCP/IP fingerprint:
OS:SCAN(V=5.00%D=11/27%OT=22%CT=1%CU=30609%PV=Y%DS=1%G=Y%M=BCAEC5%TM=50B3CA
OS:4B%P=x86_64-unknown-linux-gnu)SEQ(SP=C8%GCD=1%ISR=CB%TI=Z%CI=Z%II=I%TS=7
OS:)OPS(O1=M2300ST11NW2%O2=M2300ST11NW2%O3=M2300NNT11NW2%O4=M2300ST11NW2%O5
OS:=M2300ST11NW2%O6=M2300ST11)WIN(W1=45E8%W2=45E8%W3=45E8%W4=45E8%W5=45E8%W
OS:6=45E8)ECN(R=Y%DF=Y%T=40%W=4600%O=M2300NNSNW2%CC=N%Q=)T1(R=Y%DF=Y%T=40%S
OS:=O%A=S+%F=AS%RD=0%Q=)T2(R=N)T3(R=N)T4(R=Y%DF=Y%T=40%W=0%S=A%A=Z%F=R%O=%R
OS:D=0%Q=)T5(R=Y%DF=Y%T=40%W=0%S=Z%A=S+%F=AR%O=%RD=0%Q=)T6(R=Y%DF=Y%T=40%W=
OS:0%S=A%A=Z%F=R%O=%RD=0%Q=)T7(R=N)U1(R=Y%DF=N%T=40%IPL=164%UN=0%RIPL=G%RID
OS:=G%RIPCK=G%RUCK=G%RUD=G)IE(R=Y%DFI=N%T=40%CD=S)
Uptime guess: 12.990 days (since Wed Nov 14 01:44:40 2012)
Network Distance: 1 hop
TCP Sequence Prediction: Difficulty=200 (Good luck!)
IP ID Sequence Generation: All zeros
Read data files from: /usr/share/nmap
OS detection performed. Please report any incorrect results at http://nmap.org/submit/ .
Nmap done: 1 IP address (1 host up) scanned in 12.38 seconds
           Raw packets sent: 1126 (53.832KB) | Rcvd: 1066 (46.100KB)

See also: Fingerprinting a web-server and a dns server command line tools for more information.

#18: How do I detect remote services (server / daemon) version numbers?

nmap -sV 192.168.1.1

Sample outputs:

Starting Nmap 5.00 ( http://nmap.org ) at 2012-11-27 01:34 IST
Interesting ports on 192.168.1.1:
Not shown: 998 closed ports
PORT   STATE SERVICE VERSION
22/tcp open  ssh     Dropbear sshd 0.52 (protocol 2.0)
80/tcp open  http?
1 service unrecognized despite returning data.

#19: Scan a host using TCP ACK (PA) and TCP Syn (PS) ping

If firewall is blocking standard ICMP pings, try the following host discovery methods:

nmap -PS 192.168.1.1
nmap -PS 80,21,443 192.168.1.1
nmap -PA 192.168.1.1
nmap -PA 80,21,200-512 192.168.1.1

#20: Scan a host using IP protocol ping

nmap -PO 192.168.1.1

#21: Scan a host using UDP ping

This scan bypasses firewalls and filters that only screen TCP:

nmap -PU 192.168.1.1
nmap -PU 2000.2001 192.168.1.1

#22: Find out the most commonly used TCP ports using TCP SYN Scan

### Stealthy scan ###
nmap -sS 192.168.1.1
 
### Find out the most commonly used TCP ports using TCP connect scan (warning: no stealth scan)
### OS Fingerprinting ###
nmap -sT 192.168.1.1
 
### Find out the most commonly used TCP ports using TCP ACK scan
nmap -sA 192.168.1.1
 
### Find out the most commonly used TCP ports using TCP Window scan
nmap -sW 192.168.1.1
 
### Find out the most commonly used TCP ports using TCP Maimon scan
nmap -sM 192.168.1.1

#23: Scan a host for UDP services (UDP scan)

Most popular services on the Internet run over the TCP protocol. DNS, SNMP, and DHCP are three of the most common UDP services. Use the following syntax to find out UDP services:

nmap -sU nas03
nmap -sU 192.168.1.1

Sample outputs:

Starting Nmap 5.00 ( http://nmap.org ) at 2012-11-27 00:52 IST
Stats: 0:05:29 elapsed; 0 hosts completed (1 up), 1 undergoing UDP Scan
UDP Scan Timing: About 32.49% done; ETC: 01:09 (0:11:26 remaining)
Interesting ports on nas03 (192.168.1.12):
Not shown: 995 closed ports
PORT     STATE         SERVICE
111/udp  open|filtered rpcbind
123/udp  open|filtered ntp
161/udp  open|filtered snmp
2049/udp open|filtered nfs
5353/udp open|filtered zeroconf
MAC Address: 00:11:32:11:15:FC (Synology Incorporated)
 
Nmap done: 1 IP address (1 host up) scanned in 1099.55 seconds

#24: Scan for IP protocol

This type of scan allows you to determine which IP protocols (TCP, ICMP, IGMP, etc.) are supported by target machines:

nmap -sO 192.168.1.1

#25: Scan a firewall for security weakness

The following scan types exploit a subtle loophole in the TCP and good for testing security of common attacks:

## TCP Null Scan to fool a firewall to generate a response ##
## Does not set any bits (TCP flag header is 0) ##
nmap -sN 192.168.1.254
 
## TCP Fin scan to check firewall ##
## Sets just the TCP FIN bit ##
nmap -sF 192.168.1.254
 
## TCP Xmas scan to check firewall ##
## Sets the FIN, PSH, and URG flags, lighting the packet up like a Christmas tree ##
nmap -sX 192.168.1.254

See how to block Xmas packkets, syn-floods and other conman attacks with iptables.

#26: Scan a firewall for packets fragments

The -f option causes the requested scan (including ping scans) to use tiny fragmented IP packets. The idea is to split up the TCP header over
several packets to make it harder for packet filters, intrusion detection systems, and other annoyances to detect what you are doing.

nmap -f 192.168.1.1
nmap -f fw2.nixcraft.net.in
nmap -f 15 fw2.nixcraft.net.in

## Set your own offset size with the --mtu option ##
nmap --mtu 32 192.168.1.1

#27: Cloak a scan with decoys

The -D option it appear to the remote host that the host(s) you specify as decoys are scanning the target network too. Thus their IDS might report 5-10 port scans from unique IP addresses, but they won’t know which IP was scanning them and which were innocent decoys:

nmap -n -Ddecoy-ip1,decoy-ip2,your-own-ip,decoy-ip3,decoy-ip4 remote-host-ip
nmap -n -D192.168.1.5,10.5.1.2,172.1.2.4,3.4.2.1 192.168.1.5

#28: Scan a firewall for MAC address spoofing

### Spoof your MAC address ##
nmap --spoof-mac MAC-ADDRESS-HERE 192.168.1.1
 
### Add other options ###
nmap -v -sT -PN --spoof-mac MAC-ADDRESS-HERE 192.168.1.1
 
 
### Use a random MAC address ###
### The number 0, means nmap chooses a completely random MAC address ###
nmap -v -sT -PN --spoof-mac 0 192.168.1.1

#29: How do I save output to a text file?

The syntax is:

nmap 192.168.1.1 > output.txt
nmap -oN /path/to/filename 192.168.1.1
nmap -oN output.txt 192.168.1.1

#30 Scans for web servers and pipes into Nikto for scanning

nmap -p80 192.168.1.2/24 -oG - | /path/to/nikto.pl -h -
nmap -p80,443 192.168.1.2/24 -oG - | /path/to/nikto.pl -h -

#31 Speed up nmap

Pass the -T option:
nmap -v -sS -A -T4 192.168.2.5
Sample outputs:

Starting Nmap 7.40 ( https://nmap.org ) at 2017-05-15 01:52 IST
NSE: Loaded 143 scripts for scanning.
NSE: Script Pre-scanning.
Initiating NSE at 01:52
Completed NSE at 01:52, 0.00s elapsed
Initiating NSE at 01:52
Completed NSE at 01:52, 0.00s elapsed
Initiating ARP Ping Scan at 01:52
Scanning 192.168.2.15 [1 port]
Completed ARP Ping Scan at 01:52, 0.01s elapsed (1 total hosts)
Initiating SYN Stealth Scan at 01:52
Scanning dellm6700 (192.168.2.15) [1000 ports]
Discovered open port 5900/tcp on 192.168.2.15
Discovered open port 80/tcp on 192.168.2.15
Discovered open port 22/tcp on 192.168.2.15
Completed SYN Stealth Scan at 01:53, 4.62s elapsed (1000 total ports)
Initiating Service scan at 01:53
Scanning 3 services on dellm6700 (192.168.2.15)
Completed Service scan at 01:53, 6.01s elapsed (3 services on 1 host)
Initiating OS detection (try #1) against dellm6700 (192.168.2.15)
Retrying OS detection (try #2) against dellm6700 (192.168.2.15)
NSE: Script scanning 192.168.2.15.
Initiating NSE at 01:53
Completed NSE at 01:53, 30.02s elapsed
Initiating NSE at 01:53
Completed NSE at 01:53, 0.00s elapsed
Nmap scan report for dellm6700 (192.168.2.15)
Host is up (0.00044s latency).
Not shown: 996 filtered ports
PORT     STATE  SERVICE VERSION
22/tcp   open   ssh     (protocol 2.0)
| fingerprint-strings: 
|   NULL: 
|_    SSH-2.0-OpenSSH_7.4p1 Ubuntu-10
| ssh-hostkey: 
|   2048 1d:14:84:f0:c7:21:10:0e:30:d9:f9:59:6b:c3:95:97 (RSA)
|_  256 dc:59:c6:6e:33:33:f2:d2:5d:9b:fd:b4:9c:52:c1:0a (ECDSA)
80/tcp   open   http    nginx 1.10.0 (Ubuntu)
| http-methods: 
|_  Supported Methods: GET HEAD
|_http-server-header: nginx/1.10.0 (Ubuntu)
|_http-title: Apache2 Ubuntu Default Page: It works
443/tcp  closed https
5900/tcp open   vnc     VNC (protocol 3.7)
1 service unrecognized despite returning data. If you know the service/version, please submit the following fingerprint at https://nmap.org/cgi-bin/submit.cgi?new-service :
SF-Port22-TCP:V=7.40%I=7%D=5/15%Time=5918BCAA%P=x86_64-apple-darwin16.3.0%
SF:r(NULL,20,"SSH-2\.0-OpenSSH_7\.4p1\x20Ubuntu-10\n");
MAC Address: F0:1F:AF:1F:2C:60 (Dell)
Device type: general purpose
Running (JUST GUESSING): Linux 3.X|4.X|2.6.X (95%), OpenBSD 4.X (85%)
OS CPE: cpe:/o:linux:linux_kernel:3 cpe:/o:linux:linux_kernel:4 cpe:/o:linux:linux_kernel:2.6.32 cpe:/o:openbsd:openbsd:4.0
Aggressive OS guesses: Linux 3.11 - 4.1 (95%), Linux 4.4 (95%), Linux 3.13 (92%), Linux 4.0 (90%), Linux 2.6.32 (89%), Linux 2.6.32 or 3.10 (89%), Linux 3.2 - 3.8 (89%), Linux 3.10 - 3.12 (88%), Linux 2.6.32 - 2.6.33 (87%), Linux 2.6.32 - 2.6.35 (87%)
No exact OS matches for host (test conditions non-ideal).
Uptime guess: 0.000 days (since Mon May 15 01:53:08 2017)
Network Distance: 1 hop
TCP Sequence Prediction: Difficulty=252 (Good luck!)
IP ID Sequence Generation: All zeros
Service Info: OS: Linux; CPE: cpe:/o:linux:linux_kernel

TRACEROUTE
HOP RTT     ADDRESS
1   0.44 ms dellm6700 (192.168.2.15)

NSE: Script Post-scanning.
Initiating NSE at 01:53
Completed NSE at 01:53, 0.00s elapsed
Initiating NSE at 01:53
Completed NSE at 01:53, 0.00s elapsed
Read data files from: /usr/local/bin/../share/nmap
OS and Service detection performed. Please report any incorrect results at https://nmap.org/submit/ .
Nmap done: 1 IP address (1 host up) scanned in 46.02 seconds
           Raw packets sent: 2075 (95.016KB) | Rcvd: 50 (3.084KB)

#32: Not a fan of command line tools?

Try zenmap the official network mapper front end:

Zenmap is the official Nmap Security Scanner GUI. It is a multi-platform (Linux, Windows, Mac OS X, BSD, etc.) free and open source application which aims to make Nmap easy for beginners to use while providing advanced features for experienced Nmap users. Frequently used scans can be saved as profiles to make them easy to run repeatedly. A command creator allows interactive creation of Nmap command lines. Scan results can be saved and viewed later. Saved scan results can be compared with one another to see how they differ. The results of recent scans are stored in a searchable database.

You can install zenmap using the following apt-get command:
$ sudo apt-get install zenmap
Sample outputs:

[sudo] password for vivek: 
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following NEW packages will be installed:
  zenmap
0 upgraded, 1 newly installed, 0 to remove and 11 not upgraded.
Need to get 616 kB of archives.
After this operation, 1,827 kB of additional disk space will be used.
Get:1 http://debian.osuosl.org/debian/ squeeze/main zenmap amd64 5.00-3 [616 kB]
Fetched 616 kB in 3s (199 kB/s)                       
Selecting previously deselected package zenmap.
(Reading database ... 281105 files and directories currently installed.)
Unpacking zenmap (from .../zenmap_5.00-3_amd64.deb) ...
Processing triggers for desktop-file-utils ...
Processing triggers for gnome-menus ...
Processing triggers for man-db ...
Setting up zenmap (5.00-3) ...
Processing triggers for python-central ...

Type the following command to start zenmap:
$ sudo zenmap
Sample outputs

Fig.02: zenmap in actionFig.02: zenmap in action

How to check the file size in Linux/Unix bash shell scripting

28/09/2017 Comments off

 

How to check file size in unix using wc command

The wc command shows the number of lines, words, and bytes contained in file. The syntax is as follows to get the file size:
wc -c /path/to/file
wc -c /etc/passwd

Sample outputs:

5253 /etc/passwd

You can easily extract the first field either using the cut or awk command:
wc -c /etc/passwd | awk '{print $1}'
Sample outputs:

5253

OR assign this size to a bash variable:

myfilesize=$(wc -c "/etc/passwd" | awk '{print $1}')
printf "%d\n" $myfilesize
echo "$myfilesize"

How to get the size of a file in a bash script using stat command

The stat command shows information about the file. The syntax is as follows to get the file size on GNU/Linux stat:
stat -c %s "/etc/passwd"
OR
stat --format=%s "/etc/passwd"
To assign this size to a bash variable:

myfilesize=$(stat --format=%s "/etc/passwd")
echo "$myfilesize"
## or ##
myFileSizeCheck=$(stat -c %s "/etc/resolv.conf")
printf "My file size = %d\n" $myFileSizeCheck

The syntax is as follows to get the file size on BSD/MacOS stat:
stat -f %z "/etc/passwd"
Please note that if the file is symlink you will get size of that link only with the stat command.

du command example

The syntax is

du --apparent-size --block-size=1  "/etc/passwd"
fileName="/etc/hosts"
mfs=$(du --apparent-size --block-size=1  "$fileName" | awk '{ print $1}')
echo "$fileName size = ${mfs}"

Sample outputs from above commands:

Fig.01: How to check size of a file using a bash/ksh/zsh/sh/tcsh shell?Fig.01: How to check size of a file using a bash/ksh/zsh/sh/tcsh shell?

 

Find command example

The syntax is:

find "/etc/passwd" -printf "%s"
find "/etc/passwd" -printf "%s\n"
fileName="/etc/hosts"
mysize=$(find "$fileName" -printf "%s")
printf "File %s size = %d\n" $fileName $mysize
echo "${fileName} size is ${mysize} bytes."
Categories: Système Tags: , , , , ,

How to get domain name from URL in bash shell script

28/09/2017 Comments off
How can I extract or fetch a domain name from a URL string (e.g. https://www.cyberciti.biz/index.php) using bash shell scripting under Linux or Unix-like operating system?

You can use standard Unix commands such as sed, awk, grep, Perl, Python and more to get domain name from URL. No need to write regex. It is pretty simple.

Let use see various commands and option to grab the domain part from given variable under Linux or Unix-like system.

Get domain name from full URL

Say your url name is stored in a bash shell variable such as $x:
x='https://www.dbsysnet.com/'
You can use the awk as follows:
echo "$x" | awk -F/ '{print $3}'
### OR ###
awk -F/ '{print $3}' <<<$x

Sample outputs:

www.dbsysnet.com

Extract domain name from URL using sed

Here is a sample sed command:
url="https://www.dbsysnet.com"
echo "$url" | sed -e 's|^[^/]*//||' -e 's|/.*$||'

Extract domain name from URL using bash shell parameter substitution

Another option is to use bash shell parameter substitution:

# My shell variable 
f="https://www.cyberciti.biz/faq/copy-command/"
 
## Remove protocol part of url ##
f="${f#http://}"
f="${f#https://}"
f="${f#ftp://}"
f="${f#scp://}"
f="${f#scp://}"
f="${f#sftp://}"
 
## Remove username and/or username:password part of URL ##
f="${f#*:*@}"
f="${f#*@}"
 
## Remove rest of urls ##
f=${f%%/*}
 
## Show domain name only ##
echo "$f"

Shell script example

A shell script to purge urls from Cloudflare by matching domain name part:

#!/bin/bash
zone_id=""
api_key=""
 
urls="$@"
bon=$(tput bold)
boff=$(tput sgr0)
c=1
[ "$urls" == "" ] && { echo "Usage: $0 url"; exit 1; }
 
clear
echo "Purging..."
echo
for u in $urls
do
     echo -n "${bon}${c}${boff}.${u}: "
     ## Get domain name ##
     d="$(echo $u | awk -F/ '{ print $3}')"
     ## Set API_KEY, Email_ID, and ZONE_ID as per domain ##
     case $d in
	     www.cyberciti.biz) zone_id="ID_1"; api_key="MY_KEY_1"; email_id="webmaster@cyberciti.biz";;
	     theos.in) zone_id="ID_2"; api_key="MY_KEY_2"; email_id="webmaster@theos.in";;
	     *) echo "Domain not configured."; continue;;
     esac
     ## Do it ##
     curl -X DELETE "https://api.cloudflare.com/client/v4/zones/${zone_id}/purge_cache" \
     -H "X-Auth-Email: ${email_id}" \
     -H "X-Auth-Key: ${api_key}" \
     -H "Content-Type: application/json" \
     --data "{\"files\":[\"${u}\"]}"
     echo
     (( c++ ))
done
echo
Categories: Système Tags: , , ,

How to count total number of word occurrences using grep on Linux or Unix

28/09/2017 Comments off

I want to find out how many times a word (say foo or an IP address) occurs in a text file using the grep command on Linux or Unix-like system?

You can use the grep command to search strings, words, text, and numbers for a given patterns. You can pass the -coption to grep command. It only shows the number of times that the pattern has been matched for each file.

 

 

 

Show the total number of times that the word foo appears in a file named bar.txt

The syntax is:
grep -c string filename
grep -c foo bar.txt

Sample outputs:

3

To count total number of occurrences of word in a file named /etc/passwd root using grep, run:
grep -c root /etc/passwd
To verify that run:
grep --color root /etc/passwd
Pass the -w option to grep to select only an entire word or phrase that matches the specified pattern:
grep -w root /etc/passwd
OR
grep -c -w root /etc/passwd
In this example only match a word being with root:
grep --color -w '^root' /etc/passwd
grep -c -w '^root' /etc/passwd

To show only the matching part of the lines.
grep -o 'root' /etc/passwd
grep -c -o 'root' /etc/passwd

Sample session:

Fig.01: Counting occurrence of words/strings using grep commandFig.01: Counting occurrence of words/strings using grep command

How to find and delete directory recursively on Linux or Unix-like system

28/09/2017 Comments off

I type ‘find . -type d -iname foo -delete‘ command to find the foo directories and delete them. However, I am getting an error message that read as find: cannot delete './hourly.4/data/foo': Directory not empty on Linux server. How do delete directories based on find command output on Linux or Unix-like system?



The -delete option remove the DIRECTORY(ies), if they are empty. You need to use the -execoption to delete all directories and its contents. The syntax is as follows.

Find command syntax to delete dirs

Try:
find /dir/to/search/ -type d -name "dirName" -exec rm -rf {} +
OR
find /dir/to/search/ -type d -name "dirName" -exec rm -rf \;

Warning: Be careful with the rm command when using with find. You may end up deleting unwanted data.

Find will execute given command when it finds files or dirs. For example:
find . -type d -name "foo" -exec rm -rf {} +
OR
find . -type d -name "bar" -exec rm -rf "{}" \;
Sample outputs:

removed './daily.0/bar/.cache/motd.legal-displayed'
removed directory './daily.0/bar/root/.cache'
removed './daily.0/bar/.lesshst'
removed './daily.0/bar/.viminfo'
removed './daily.0/bar/.vim/.netrwhist'
removed directory './daily.0/bar/root/.vim'
removed './daily.0/bar/root/.bashrc'
removed './daily.0/bar/.ssh/authorized_keys'
removed directory './daily.0/bar/root'
removed directory './daily.0/bar/var/spool/cron/crontabs'

You can find directories that are at least four levels deep in the working directory /backups/:
find /backups/ -type d -name "bar" -depth +4 -print0 -exec rm -rf {} +

Find and xargs

The syntax is as follows to find and delete directories on Linux/Unix system:
## delete all empty dirs ##
find /path/to/dir/ -type d -empty -print0 | xargs -0 -I {} /bin/rm -rf "{}"
## delete all foo dirs including subdirs in /backups/
find /backups/ -type d -name "foo*" -print0 | xargs -0 -I {} /bin/rm -rf "{}"

The second command is secure and fast version as it deals with weird dir names such as:

  • “foo bar”
  • “Foo _ *bar”
 
Categories: Système Tags: , , ,

How to display countdown timer in bash shell script running on Linux/Unix

28/09/2017 Comments off
I want to display a countdown before purging cache from CDN network. Is there an existing command to show a conuntdown from 30..1 as 30,29,28,…1 on Linux or Unix bash shell script?

There are various ways to show a countdown in your shell scripts. 

First define your message:
msg="Purging cache please wait..."
Now clear the screen and display the message at row 10 and column 5 using tput:
clear
tput cup 10 5

Next you need to display the message:
echo -n "$msg"

Find out the length of string:
l=${#msg}
Calculate the next column:
l=$(( l+5 ))
Finally use a bash for loop to show countdown:
for i in {30..01}
do
tput cup 10 $l
echo -n "$i"
sleep 1
done
echo

Here is a complete shell script:

#!/bin/bash
# Purpose: Purge urls from Cloudflare Cache
# Author: Vivek Gite {www.cyberciti.biz} under GPL v2.x+
# --------------------------------------------------------
# Set me first #
zone_id="My-ID"
api_key="My_API_KEY"
email_id="My_EMAIL_ID"
row=2
col=2
urls="$@"
countdown() {
        msg="Purging ${1}..."
        clear
        tput cup $row $col
        echo -n "$msg"
        l=${#msg}
        l=$(( l+$col ))
        for i in {30..1}
        do
                tput cup $row $l
                echo -n "$i"
                sleep 1
        done
}
# Do it
for u in $urls
do
     amp_url="${u}amp/"
     curl -X DELETE "https://api.cloudflare.com/client/v4/zones/${zone_id}/purge_cache" \
     -H "X-Auth-Email: ${email_id}" \
     -H "X-Auth-Key: ${api_key}" \
     -H "Content-Type: application/json" \
     --data "{\"files\":[\"${u}\",\"${amp_url}\"]}" &>/dev/null &&  countdown "$u"
 
done
echo

You can run it as follows:
./script.sh url1 url2

POSIX shell version

From this post:

countdown()
(
  IFS=:
  set -- $*
  secs=$(( ${1#0} * 3600 + ${2#0} * 60 + ${3#0} ))
  while [ $secs -gt 0 ]
  do
    sleep 1 &
    printf "\r%02d:%02d:%02d" $((secs/3600)) $(( (secs/60)%60)) $((secs%60))
    secs=$(( $secs - 1 ))
    wait
  done
  echo
)

It can be run as follows:
countdown "00:00:10" # 10 sec
countdown "00:00:30" # 30 sec
countdown "00:01:42" # 1 min 42 sec

 

Categories: Système Tags: , , ,

Bash Shell: Replace a String With Another String In All Files Using sed and Perl -pie Options

28/09/2017 Comments off

String search and replace

How do I replace a string with another string in all files? For example, ~/foo directory has 100s of text file and I’d like to find out xyz string and replace with abc. I’d like to use sed or any other tool to replace all occurrence of the word.

The sed command is designed for this kind of work i.e. find and replace strings or words from a text file under Apple OX, *BSD, Linux, and UNIX like operating systems. The perl can be also used as described below.

sed replace word / string syntax

The syntax is as follows:
sed -i 's/old-word/new-word/g' *.txt

GNU sed command can edit files in place (makes backup if extension supplied) using the -i option. If you are using an old UNIX sed command version try the following syntax:

sed 's/old/new/g' input.txt > output.txt

You can use old sed syntax along with bash for loop:

#!/bin/bash
OLD="xyz"
NEW="abc"
DPATH="/home/you/foo/*.txt"
BPATH="/home/you/bakup/foo"
TFILE="/tmp/out.tmp.$$"
[ ! -d $BPATH ] && mkdir -p $BPATH || :
for f in $DPATH
do
  if [ -f $f -a -r $f ]; then
    /bin/cp -f $f $BPATH
   sed "s/$OLD/$NEW/g" "$f" > $TFILE && mv $TFILE "$f"
  else
   echo "Error: Cannot read $f"
  fi
done
/bin/rm $TFILE

A Note About Bash Escape Character

A non-quoted backslash \ is the Bash escape character. It preserves the literal value of the next character that follows, with the exception of newline. If a \newline pair appears, and the backslash itself is not quoted, the \newline is treated as a line continuation (that is, it is removed from the input stream and effectively ignored). This is useful when you would like to deal with UNIX paths. In this example, the sed command is used to replace UNIX path “/nfs/apache/logs/rawlogs/access.log” with “__DOMAIN_LOG_FILE__”:

#!/bin/bash
## Our path
_r1="/nfs/apache/logs/rawlogs/access.log"
 
## Escape path for sed using bash find and replace 
_r1="${_r1//\//\\/}"
 
# replace __DOMAIN_LOG_FILE__ in our sample.awstats.conf
sed -e "s/__DOMAIN_LOG_FILE__/${_r1}/" /nfs/conf/awstats/sample.awstats.conf  > /nfs/apache/logs/awstats/awstats.conf 
 
# call awstats
/usr/bin/awstats -c /nfs/apache/logs/awstats/awstats.conf

The $_r1 is escaped using bash find and replace parameter substitution syntax to replace each occurrence of / with \/.

perl -pie Syntax For Find and Replace

The syntax is as follows:
perl -pie 's/old-word/new-word/g' input.file > new.output.file

Categories: Système Tags: , , , ,

Bash scripting Tutorial

28/09/2017 Comments off

Hello World Bash Shell Script

linuxconfig.org:~$ which bash > hello-world.sh                          linuxconfig.org:~$ vi hello-world.sh                                    linuxconfig.org:~$ chmod +x hello-world.sh                              linuxconfig.org:~$ ./hello-world.sh                                     Hello World                                                             linuxconfig.org:~$                                                                                                                                                                                                      
00:00
 

First you need to find out where is your bash interpreter located. Enter the following into your command line:

$ which bash

bash interpreter location: /bin/bash

Open up you favorite text editor and create file called hello_world.sh. Insert the following lines to a file:

NOTE:Every bash shell script in this tutorial starts with shebang: »#! » which is not read as a comment. First line is also a place where you put your interpreter which is in this case: /bin/bash.

Here is our first bash shell script example:

#!/bin/bash
# declare STRING variable
STRING="Hello World"
#print variable on a screen
echo $STRING

Navigate to a directory where your hello_world.sh is located and make the file executable:

$ chmod +x hello_world.sh 

Make bash shell script executable

Now you are ready to execute your first bash script:

./hello_world.sh 

Example of simple bash shell script

Simple Backup bash shell script

#!/bin/bash
tar -czf myhome_directory.tar.gz /home/linuxconfig

Simple Backup bash script

Variables

In this example we declare simple bash variable and print it on the screen ( stdout ) with echo command.

#!/bin/bash
 STRING="HELLO WORLD!!!"
 echo $STRING 

Bash string Variables in bash script

Your backup script and variables:

#!/bin/bash
 OF=myhome_directory_$(date +%Y%m%d).tar.gz
 tar -czf $OF /home/linuxconfig 

Bash backup Script with bash Variables

Global vs. Local variables

#!/bin/bash
#Define bash global variable
#This variable is global and can be used anywhere in this bash script
VAR="global variable"
function bash {
#Define bash local variable
#This variable is local to bash function only
local VAR="local variable"
echo $VAR
}
echo $VAR
bash
# Note the bash global variable did not change
# "local" is bash reserved word
echo $VAR

Global vs. Local Bash variables in bash script

Passing arguments to the bash script

#!/bin/bash
# use predefined variables to access passed arguments
#echo arguments to the shell
echo $1 $2 $3 ' -> echo $1 $2 $3'

# We can also store arguments from bash command line in special array
args=("$@")
#echo arguments to the shell
echo ${args[0]} ${args[1]} ${args[2]} ' -> args=("$@"); echo ${args[0]} ${args[1]} ${args[2]}'

#use $@ to print out all arguments at once
echo $@ ' -> echo $@'

# use $# variable to print out
# number of arguments passed to the bash script
echo Number of arguments passed: $# ' -> echo Number of arguments passed: $#' 
/arguments.sh Bash Scripting Tutorial 

Passing arguments to the bash script

Executing shell commands with bash

#!/bin/bash
# use backticks " ` ` " to execute shell command
echo `uname -o`
# executing bash command without backticks
echo uname -o 

Executing shell commands with bash

Reading User Input

#!/bin/bash
 
echo -e "Hi, please type the word: \c "
read  word
echo "The word you entered is: $word"
echo -e "Can you please enter two words? "
read word1 word2
echo "Here is your input: \"$word1\" \"$word2\""
echo -e "How do you feel about bash scripting? "
# read command now stores a reply into the default build-in variable $REPLY
read
echo "You said $REPLY, I'm glad to hear that! "
echo -e "What are your favorite colours ? "
# -a makes read command to read into an array
read -a colours
echo "My favorite colours are also ${colours[0]}, ${colours[1]} and ${colours[2]}:-)" 

Reading User Input with bash

Bash Trap Command

#!/bin/bash
# bash trap command
trap bashtrap INT
# bash clear screen command
clear;
# bash trap function is executed when CTRL-C is pressed:
# bash prints message => Executing bash trap subrutine !
bashtrap()
{
    echo "CTRL+C Detected !...executing bash trap !"
}
# for loop from 1/10 to 10/10
for a in `seq 1 10`; do
    echo "$a/10 to Exit." 
    sleep 1;
done
echo "Exit Bash Trap Example!!!" 

Arrays

Declare simple bash array

#!/bin/bash
#Declare array with 4 elements
ARRAY=( 'Debian Linux' 'Redhat Linux' Ubuntu Linux )
# get number of elements in the array
ELEMENTS=${#ARRAY[@]}

# echo each element in array 
# for loop
for (( i=0;i<$ELEMENTS;i++)); do
    echo ${ARRAY[${i}]}
done 

Declare simple bash array

Read file into bash array

#!/bin/bash
# Declare array
declare -a ARRAY
# Link filedescriptor 10 with stdin
exec 10<&0
# stdin replaced with a file supplied as a first argument
exec < $1
let count=0

while read LINE; do

    ARRAY[$count]=$LINE
    ((count++))
done

echo Number of elements: ${#ARRAY[@]}
# echo array's content
echo ${ARRAY[@]}
# restore stdin from filedescriptor 10
# and close filedescriptor 10
exec 0<&10 10<&-

Bash script execution with an output:

linuxconfig.org $ cat bash.txt 
Bash
Scripting
Tutorial
Guide
linuxconfig.org $ ./bash-script.sh bash.txt 
Number of elements: 4
Bash Scripting Tutorial Guide
linuxconfig.org $ 

Bash if / else / fi statements

Simple Bash if/else statement

Please note the spacing inside the [ and ] brackets! Without the spaces, it won’t work!

#!/bin/bash
directory="./BashScripting"

# bash check if directory exists
if [ -d $directory ]; then
	echo "Directory exists"
else 
	echo "Directory does not exists"
fi 

Bash if else fi statement

Nested if/else

#!/bin/bash
 
# Declare variable choice and assign value 4
choice=4
# Print to stdout
 echo "1. Bash"
 echo "2. Scripting"
 echo "3. Tutorial"
 echo -n "Please choose a word [1,2 or 3]? "
# Loop while the variable choice is equal 4
# bash while loop
while [ $choice -eq 4 ]; do
 
# read user input
read choice
# bash nested if/else
if [ $choice -eq 1 ] ; then
 
        echo "You have chosen word: Bash"

else                   

        if [ $choice -eq 2 ] ; then
                 echo "You have chosen word: Scripting"
        else
         
                if [ $choice -eq 3 ] ; then
                        echo "You have chosen word: Tutorial"
                else
                        echo "Please make a choice between 1-3 !"
                        echo "1. Bash"
                        echo "2. Scripting"
                        echo "3. Tutorial"
                        echo -n "Please choose a word [1,2 or 3]? "
                        choice=4
                fi   
        fi
fi
done 

Nested Bash if else statement

Bash Comparisons

Arithmetic Comparisons

-lt <
-gt >
-le <=
-ge >=
-eq ==
-ne !=
#!/bin/bash
# declare integers
NUM1=2
NUM2=2
if [ $NUM1 -eq $NUM2 ]; then
	echo "Both Values are equal"
else 
	echo "Values are NOT equal"
fi 

Bash Arithmetic Comparisons

#!/bin/bash
# declare integers
NUM1=2
NUM2=1
if [ $NUM1 -eq $NUM2 ]; then
	echo "Both Values are equal"
else 
	echo "Values are NOT equal"
fi 

Bash Arithmetic Comparisons - values are NOT equal

#!/bin/bash
# declare integers
NUM1=2
NUM2=1
if   [ $NUM1 -eq $NUM2 ]; then
	echo "Both Values are equal"
elif [ $NUM1 -gt $NUM2 ]; then
	echo "NUM1 is greater then NUM2"
else 
	echo "NUM2 is greater then NUM1"
fi 

Bash Arithmetic Comparisons - greater then

String Comparisons

= equal
!= not equal
< less then
> greater then
-n s1 string s1 is not empty
-z s1 string s1 is empty
#!/bin/bash
#Declare string S1
S1="Bash"
#Declare string S2
S2="Scripting"
if [ $S1 = $S2 ]; then
	echo "Both Strings are equal"
else 
	echo "Strings are NOT equal"
fi 

Bash String Comparisons - values are NOT equal

#!/bin/bash
#Declare string S1
S1="Bash"
#Declare string S2
S2="Bash"
if [ $S1 = $S2 ]; then
	echo "Both Strings are equal"
else 
	echo "Strings are NOT equal"
fi 

bash interpreter location: /bin/bash

Bash File Testing

-b filename Block special file
-c filename Special character file
-d directoryname Check for directory existence
-e filename Check for file existence
-f filename Check for regular file existence not a directory
-G filename Check if file exists and is owned by effective group ID.
-g filename true if file exists and is set-group-id.
-k filename Sticky bit
-L filename Symbolic link
-O filename True if file exists and is owned by the effective user id.
-r filename Check if file is a readable
-S filename Check if file is socket
-s filename Check if file is nonzero size
-u filename Check if file set-ser-id bit is set
-w filename Check if file is writable
-x filename Check if file is executable
#!/bin/bash
file="./file"
if [ -e $file ]; then
	echo "File exists"
else 
	echo "File does not exists"
fi 

Bash File Testing - File does not exist Bash File Testing - File exists

Similarly for example we can use while loop to check if file does not exists. This script will sleep until file does exists. Note bash negator « ! » which negates the -e option.

#!/bin/bash
 
while [ ! -e myfile ]; do
# Sleep until file does exists/is created
sleep 1
done 

Loops

Bash for loop

#!/bin/bash

# bash for loop
for f in $( ls /var/ ); do
	echo $f
done 

Running for loop from bash shell command line:

$ for f in $( ls /var/ ); do echo $f; done 

Bash for loop

Bash while loop

#!/bin/bash
COUNT=6
# bash while loop
while [ $COUNT -gt 0 ]; do
	echo Value of count is: $COUNT
	let COUNT=COUNT-1
done 

Bash while loop

Bash until loop

#!/bin/bash
COUNT=0
# bash until loop
until [ $COUNT -gt 5 ]; do
        echo Value of count is: $COUNT
        let COUNT=COUNT+1
done 

Bash until loop

Control bash loop with

Here is a example of while loop controlled by standard input. Until the redirection chain from STDOUT to STDIN to the read command exists the while loop continues.

#!/bin/bash
# This bash script will locate and replace spaces
# in the filenames
DIR="."
# Controlling a loop with bash read command by redirecting STDOUT as
# a STDIN to while loop
# find will not truncate filenames containing spaces
find $DIR -type f | while read file; do
# using POSIX class [:space:] to find space in the filename
if [[ "$file" = *[[:space:]]* ]]; then
# substitute space with "_" character and consequently rename the file
mv "$file" `echo $file | tr ' ' '_'`
fi;
# end of while loop
done 

Bash script to replace spaces in the filenames with _

Bash Functions

!/bin/bash
# BASH FUNCTIONS CAN BE DECLARED IN ANY ORDER
function function_B {
        echo Function B.
}
function function_A {
        echo $1
}
function function_D {
        echo Function D.
}
function function_C {
        echo $1
}
# FUNCTION CALLS
# Pass parameter to function A
function_A "Function A."
function_B
# Pass parameter to function C
function_C "Function C."
function_D 

Bash Functions

Bash Select

#!/bin/bash
 
PS3='Choose one word: ' 

# bash select
select word in "linux" "bash" "scripting" "tutorial" 
do
  echo "The word you have selected is: $word"
# Break, otherwise endless loop
  break  
done

exit 0 

Bash Select

Case statement conditional

#!/bin/bash
echo "What is your preferred programming / scripting language"
echo "1) bash"
echo "2) perl"
echo "3) phyton"
echo "4) c++"
echo "5) I do not know !"
read case;
#simple case bash structure
# note in this case $case is variable and does not have to
# be named case this is just an example
case $case in
    1) echo "You selected bash";;
    2) echo "You selected perl";;
    3) echo "You selected phyton";;
    4) echo "You selected c++";;
    5) exit
esac 

bash case statement conditiona

Bash quotes and quotations

Quotations and quotes are important part of bash and bash scripting. Here are some bash quotes and quotations basics.

Escaping Meta characters

Before we start with quotes and quotations we should know something about escaping meta characters. Escaping will suppress a special meaning of meta characters and therefore meta characters will be read by bash literally. To do this we need to use backslash « \ » character. Example:

#!/bin/bash
 
#Declare bash string variable
BASH_VAR="Bash Script"

# echo variable BASH_VAR
echo $BASH_VAR

#when meta character such us "$" is escaped with "\" it will be read literally
echo $BASH_VAR 

# backslash has also special meaning and it can be suppressed with yet another "\"
echo "\" 

escaping meta characters in bash

Single quotes

Single quotes in bash will suppress special meaning of every meta characters. Therefore meta characters will be read literally. It is not possible to use another single quote within two single quotes not even if the single quote is escaped by backslash.

#!/bin/bash
 
 #Declare bash string variable
 BASH_VAR="Bash Script"
 
 # echo variable BASH_VAR
 echo $BASH_VAR
 
 # meta characters special meaning in bash is suppressed when  using single quotes 
 echo '$BASH_VAR  "$BASH_VAR"' 

Using single quotes in bash

Double Quotes

Double quotes in bash will suppress special meaning of every meta characters except « $ », « \ » and « ` ». Any other meta characters will be read literally. It is also possible to use single quote within double quotes. If we need to use double quotes within double quotes bash can read them literally when escaping them with « \ ». Example:

#!/bin/bash
 
#Declare bash string variable
BASH_VAR="Bash Script"

# echo variable BASH_VAR
echo $BASH_VAR

# meta characters and its special meaning in bash is 
# suppressed when using double quotes except "$", "\" and "`"

echo "It's $BASH_VAR  and \"$BASH_VAR\" using backticks: `date`" 

Using double quotes in bash

Bash quoting with ANSI-C style

There is also another type of quoting and that is ANSI-C. In this type of quoting characters escaped with « \ » will gain special meaning according to the ANSI-C standard.

\a alert (bell) \b backspace
\e an escape character \f form feed
\n newline \r carriage return
\t horizontal tab \v vertical tab
\\ backslash \` single quote
\nnn octal value of characters ( see [http://www.asciitable.com/ ASCII table] ) \xnn hexadecimal value of characters ( see [http://www.asciitable.com/ ASCII table] )

The syntax fo ansi-c bash quoting is: $ » . Here is an example:

#!/bin/bash
 
# as a example we have used \n as a new line, \x40 is hex value for @
# and  is octal value for .
echo $'web: www.linuxconfig.org\nemail: web\x40linuxconfigorg' 

quoting in bash with ansi-c stype

Arithmetic Operations

Bash Addition Calculator Example

#!/bin/bash
 
let RESULT1=$1+$2
echo $1+$2=$RESULT1 ' -> # let RESULT1=$1+$2'
declare -i RESULT2
RESULT2=$1+$2
echo $1+$2=$RESULT2 ' -> # declare -i RESULT2; RESULT2=$1+$2'
echo $1+$2=$(($1 + $2)) ' -> # $(($1 + $2))' 

Bash Addition Calculator

Bash Arithmetics

#!/bin/bash
 
echo '### let ###'
# bash addition
let ADDITION=3+5
echo "3 + 5 =" $ADDITION

# bash subtraction
let SUBTRACTION=7-8
echo "7 - 8 =" $SUBTRACTION 

# bash multiplication
let MULTIPLICATION=5*8
echo "5 * 8 =" $MULTIPLICATION

# bash division
let DIVISION=4/2
echo "4 / 2 =" $DIVISION

# bash modulus
let MODULUS=9%4
echo "9 % 4 =" $MODULUS

# bash power of two
let POWEROFTWO=2**2
echo "2 ^ 2 =" $POWEROFTWO


echo '### Bash Arithmetic Expansion ###'
# There are two formats for arithmetic expansion: $[ expression ] 
# and $(( expression #)) its your choice which you use

echo 4 + 5 = $((4 + 5))
echo 7 - 7 = $[ 7 - 7 ]
echo 4 x 6 = $((3 * 2))
echo 6 / 3 = $((6 / 3))
echo 8 % 7 = $((8 % 7))
echo 2 ^ 8 = $[ 2 ** 8 ]


echo '### Declare ###'

echo -e "Please enter two numbers \c"
# read user input
read num1 num2
declare -i result
result=$num1+$num2
echo "Result is:$result "

# bash convert binary number 10001
result=2#10001
echo $result

# bash convert octal number 16
result=8#16
echo $result

# bash convert hex number 0xE6A
result=16#E6A
echo $result 

Bash Arithmetic Operations

Round floating point number

#!/bin/bash
# get floating point number
floating_point_number=3.3446
echo $floating_point_number
# round floating point number with bash
for bash_rounded_number in $(printf %.0f $floating_point_number); do
echo "Rounded number with bash:" $bash_rounded_number
done 

Round floating point number with bash

Bash floating point calculations

#!/bin/bash
# Simple linux bash calculator 
echo "Enter input:" 
read userinput
echo "Result with 2 digits after decimal point:"
echo "scale=2; ${userinput}" | bc 
echo "Result with 10 digits after decimal point:"
echo "scale=10; ${userinput}" | bc 
echo "Result as rounded integer:"
echo $userinput | bc 

Bash floating point calculations

Redirections

STDOUT from bash script to STDERR

#!/bin/bash
 
 echo "Redirect this STDOUT to STDERR" 1>&2 

To prove that STDOUT is redirected to STDERR we can redirect script’s output to file:
STDOUT from bash script to STDERR

STDERR from bash script to STDOUT

#!/bin/bash
 
 cat $1 2>&1 

To prove that STDERR is redirected to STDOUT we can redirect script’s output to file:
STDERR from bash script to STDOUT

stdout to screen

The simple way to redirect a standard output ( stdout ) is to simply use any command, because by default stdout is automatically redirected to screen. First create a file « file1 »:

$ touch file1
$ ls file1 
file1

As you can see from the example above execution of ls command produces STDOUT which by default is redirected to screen.

stdout to file

The override the default behavior of STDOUT we can use « > » to redirect this output to file:

$ ls file1 > STDOUT
$ cat STDOUT 
file1

stderr to file

By default STDERR is displayed on the screen:

$ ls
file1  STDOUT
$ ls file2
ls: cannot access file2: No such file or directory

In the following example we will redirect the standard error ( stderr ) to a file and stdout to a screen as default. Please note that STDOUT is displayed on the screen, however STDERR is redirected to a file called STDERR:

$ ls
file1  STDOUT
$ ls file1 file2 2> STDERR
file1
$ cat STDERR 
ls: cannot access file2: No such file or directory

stdout to stderr

It is also possible to redirect STDOUT and STDERR to the same file. In the next example we will redirect STDOUT to the same descriptor as STDERR. Both STDOUT and STDERR will be redirected to file « STDERR_STDOUT ».

$ ls
file1  STDERR  STDOUT
$ ls file1 file2 2> STDERR_STDOUT 1>&2
$ cat STDERR_STDOUT
ls: cannot access file2: No such file or directory
file1

File STDERR_STDOUT now contains STDOUT and STDERR.

stderr to stdout

The above example can be reversed by redirecting STDERR to the same descriptor as SDTOUT:

$ ls
file1  STDERR  STDOUT
$ ls file1 file2 > STDERR_STDOUT 2>&1
$ cat STDERR_STDOUT 
ls: cannot access file2: No such file or directory
file1

stderr and stdout to file

Previous two examples redirected both STDOUT and STDERR to a file. Another way to achieve the same effect is illustrated below:

$ ls
file1  STDERR  STDOUT
$ ls file1 file2 &> STDERR_STDOUT
$ cat STDERR_STDOUT 
ls: cannot access file2: No such file or directory
file1

or

ls file1 file2 >& STDERR_STDOUT
$ cat STDERR_STDOUT 
ls: cannot access file2: No such file or directory
file1
 
Categories: Système Tags: , , ,