Databases, Systems & Networks

Click on the item in the Finder’s sidebar with the house icon and verify that you are able to write to this folder in the Ownership & Permissions section of the Get Info window, and that it isn’t locked; if it is already set this way and you get that error, open the Terminal in the /Applications/Utilities/ folder and run the following:

mkdir ~/.Trash

If you get a message stating that the folder exists, run the following:

sudo chown $UID ~/.Trash
chmod u+rwx ~/.Trash

The first command in the second set will prompt you for your administrator password; nothing will appear in the Terminal window while it is being typed. In either case, click on the Finder icon in the Dock with the Control and Option keys pressed, and relaunch it.

At the command line, environmental variables are defined for the current shell and become inherited by any running command or process. They can determine anything from the default shell, the PATH, the users home directory, to the terminal emulation type, current working directory, where a history file is located, language and localization settings, and going further to include shell variables, which include everything from customizations to the bash prompt, colorized ls output, and changes to terminal appearance, to aliases, and much more.

Let’s walk through how to list environment and shell variables, and then how to set and add new environment variables at the command line of Mac OS X.

Displaying Current Environment & Shell Variables in Mac OS X

To quickly get a list of environmental variables, you can use the following command:

printenv

If you want to see a complete list of shell variables, the ‘set’ command can be issued as well:

set

The output of these commands can be lengthy so you may wish to pipe the output through the less or more commands.
Lire la suite…

Want to share your terminal over the web for demo, learning or collaboration purpose? Try these two applications to share your terminal as a web application.

Please note that accepting input from remote clients is dangerous for most commands. When you need interaction with the TTY for some reasons, consider starting following tools with tmux or GNU Screen and run your command on it. Use following tools with trusted parties or inside VM. Let us see how to install and use gotty and ttyd on a Unix-like system.

1. gotty

GoTTY is a simple command line tool that turns your CLI tools into web applications. It is written in go programming language.

Installation

You can install gotty on macOS using the brew command:
$ brew install yudai/gotty/gotty

Sample outputs:

Updating Homebrew...
==> Tapping yudai/gotty Cloning into '/usr/local/Homebrew/Library/Taps/yudai/homebrew-gotty'...
remote: Counting objects: 5, done.
remote: Compressing objects: 100% (5/5), done.
remote: Total 5 (delta 1), reused 2 (delta 0), pack-reused 0
Unpacking objects: 100% (5/5), done.
Tapped 1 formula (30 files, 22.7KB)
==> Installing gotty from yudai/gotty
==> Downloading https://github.com/yudai/gotty/releases/download/v1.0.1/gotty_darwin_amd64.tar.gz
==> Downloading from https://github-production-release-asset-2e65be.s3.amazonaws.com/40808571/c401bd34-7bd9-11e7-8
######################################################################## 100.0%
==> Caveats GoTTY! ==> Summary
🍺 /usr/local/Cellar/gotty/v1.0.1: 3 files, 8.2MB, built in 1 minute

Another option for Linux or Unix like system is to type the following command if you have a go language dev setup installed:

$ go get github.com/yudai/gotty

Usage

The syntax is:

gotty command
$ gotty htop

Sample outputs:

2017/09/23 22:31:19 Server is starting with command: htop
2017/09/23 22:31:19 URL: http://127.0.0.1:8080/
2017/09/23 22:31:19 URL: http://[::1]:8080/
2017/09/23 22:31:19 URL: http://[fe80::1]:8080/
2017/09/23 22:31:19 URL: http://[fe80::1c3a:3312:311b:cca4]:8080/
2017/09/23 22:31:19 URL: http://192.168.225.106:8080/
2017/09/23 22:31:19 URL: http://[fe80::6c1b:58ff:fe8a:4e6e]:8080/
2017/09/23 22:31:19 URL: http://[fe80::526a:bad6:960f:369f]:8080/
2017/09/23 22:31:19 URL: http://10.8.0.2:8080/

Fire a browser and type the url:
http://127.0.0.1:8080/
OR from another computer in your LAN/VLAN:
http://192.168.225.106:8080/

Sample outputs:

Gif 01: gotty in action

For more info and documentation see gotty home page.

2. ttyd

ttyd is a simple command-line tool for sharing terminal over the web, inspired by GoTTY. It is built on top of Libwebsockets with C for speed. Works with macOS, Linux, FreeBSD, OpenWrt/LEDE, and MS-Windows oses.

Installation

If you are using macOS, run the following brew command:
$ brew install ttyd
Sample outputs:

Viveks-MacBook-Pro:~ veryv$ brew install ttyd
Updating Homebrew...
==> Auto-updated Homebrew!
Updated 1 tap (homebrew/core).
No changes to formulae.
 
==> Installing dependencies for ttyd: json-c, libwebsockets
==> Installing ttyd dependency: json-c
==> Downloading https://homebrew.bintray.com/bottles/json-c-0.12.1.sierra.bottle.tar.gz
######################################################################## 100.0%
==> Pouring json-c-0.12.1.sierra.bottle.tar.gz
🍺  /usr/local/Cellar/json-c/0.12.1: 27 files, 156.2KB
==> Installing ttyd dependency: libwebsockets
==> Downloading https://homebrew.bintray.com/bottles/libwebsockets-2.2.1.sierra.bottle.tar.gz
######################################################################## 100.0%
==> Pouring libwebsockets-2.2.1.sierra.bottle.tar.gz
🍺  /usr/local/Cellar/libwebsockets/2.2.1: 29 files, 4.2MB
==> Installing ttyd
==> Downloading https://homebrew.bintray.com/bottles/ttyd-1.3.3.sierra.bottle.tar.gz
######################################################################## 100.0%
==> Pouring ttyd-1.3.3.sierra.bottle.tar.gz
🍺  /usr/local/Cellar/ttyd/1.3.3: 6 files, 282.6KB

If you are using a Debian/Ubuntu Linux, run:
$ sudo apt-get install -y software-properties-common
$ sudo add-apt-repository ppa:tsl0922/ttyd-dev
$ sudo apt-get update
$ sudo apt-get install ttyd

Usage

The syntax is:
$ ttyd command
$ ttyd bash
Sample outputs:

[2017/09/23 22:57:32:9322] NOTICE: ttyd 1.3.3 (libwebsockets 2.2.0)
[2017/09/23 22:57:32:9323] NOTICE: tty configuration:
[2017/09/23 22:57:32:9323] NOTICE:   start command: bash
[2017/09/23 22:57:32:9323] NOTICE:   reconnect timeout: 10s
[2017/09/23 22:57:32:9323] NOTICE:   close signal: SIGHUP (1)
[2017/09/23 22:57:32:9323] NOTICE: Initial logging level 7
[2017/09/23 22:57:32:9323] NOTICE: Libwebsockets version: 2.2.0 brew@Sierra.local-
[2017/09/23 22:57:32:9323] NOTICE: IPV6 not compiled in
[2017/09/23 22:57:32:9323] NOTICE: libev support not compiled in
[2017/09/23 22:57:32:9323] NOTICE: libuv support not compiled in
[2017/09/23 22:57:32:9324] NOTICE:  Threads: 1 each 256 fds
[2017/09/23 22:57:32:9325] NOTICE:  mem: platform fd map:  2048 bytes
[2017/09/23 22:57:32:9325] NOTICE:  Compiled with OpenSSL support
[2017/09/23 22:57:32:9325] NOTICE:  SSL disabled: no LWS_SERVER_OPTION_DO_SSL_GLOBAL_INIT
[2017/09/23 22:57:32:9326] NOTICE: Creating Vhost 'default' port 7681, 2 protocols, IPv6 off
[2017/09/23 22:57:32:9337] NOTICE:  Listening on port 7681
[2017/09/23 22:57:32:9338] NOTICE:  mem: per-conn:          568 bytes + protocol rx buf
[2017/09/23 22:57:32:9339] NOTICE:  canonical_hostname = Viveks-MacBook-Pro.local

Fire a web browser and type url:
127.0.0.1:7681
Sample session:

Gif. 02: bash-ttyd demo

For more info and docs see project home page.

An alias is nothing but the shortcut to commands. The alias command allows the user to launch any command or group of commands (including options and filenames) by entering a single word. Use alias command to display a list of all defined aliases. You can add user-defined aliases to ~/.bashrcfile. You can cut down typing time with these aliases, work smartly, and increase productivity at the command prompt.

More about aliases

The general syntax for the alias command for the bash shell is as follows:

Task: List aliases

Type the following command:

alias

Sample outputs:

alias ..='cd ..'
alias amazonbackup='s3backup'
alias apt-get='sudo apt-get'
...

By default alias command shows a list of aliases that are defined for the current user.

Task: Define / create an alias (bash syntax)

To create the alias use the following syntax:

alias name=value
alias name='command'
alias name='command arg1 arg2'
alias name='/path/to/script'
alias name='/path/to/script.pl arg1'

In this example, create the alias c for the commonly used clear command, which clears the screen, by typing the following command and then pressing the ENTER key:

alias c='clear'

Then, to clear the screen, instead of typing clear, you would only have to type the letter ‘c’ and press the [ENTER] key:

c

Task: Disable an alias temporarily (bash syntax)

An alias can be disabled temporarily using the following syntax:

## path/to/full/command
/usr/bin/clear
## call alias with a backslash ##
\c

Task: Remove an alias (bash syntax)

You need to use the command called unalias to remove aliases. Its syntax is as follows:

unalias aliasname

In this example, remove the alias c which was created in an earlier example:

unalias c

You also need to delete the alias from the ~/.bashrc file using a text editor (see next section).

Task: Make aliases permanent (bash syntax)

The alias c remains in effect only during the current login session. Once you logs out or reboot the system the alias c will be gone. To avoid this problem, add alias to your ~/.bashrc file, enter:

vi ~/.bashrc

The alias c for the current user can be made permanent by entering the following line:

alias c='clear'

Save and close the file. System-wide aliases (i.e. aliases for all users) can be put in the /etc/bashrc file. Please note that the alias command is built into a various shells including ksh, tcsh/csh, ash, bash and others.

A note about privileged access

You can add code as follows in ~/.bashrc:

# if user is not root, pass all commands via sudo #
if [ $UID -ne 0 ]; then
    alias reboot='sudo reboot'
    alias update='sudo apt-get upgrade'
fi

A note about os specific aliases

You can add code as follows in ~/.bashrc using the case statement:

### Get os name via uname ###
_myos="$(uname)"
 
### add alias as per os using $_myos ###
case $_myos in
   Linux) alias foo='/path/to/linux/bin/foo';;
   FreeBSD|OpenBSD) alias foo='/path/to/bsd/bin/foo' ;;
   SunOS) alias foo='/path/to/sunos/bin/foo' ;;
   *) ;;
esac

30 uses for aliases

You can define various types aliases as follows to save time and increase productivity.

#1: Control ls command output

The ls command lists directory contents and you can colorize the output:

## Colorize the ls output ##
alias ls='ls --color=auto'
 
## Use a long listing format ##
alias ll='ls -la'
 
## Show hidden files ##
alias l.='ls -d .* --color=auto'

#2: Control cd command behavior

## get rid of command not found ##
alias cd..='cd ..'
 
## a quick way to get out of current directory ##
alias ..='cd ..'
alias ...='cd ../../../'
alias ....='cd ../../../../'
alias .....='cd ../../../../'
alias .4='cd ../../../../'
alias .5='cd ../../../../..'

#3: Control grep command output

grep command is a command-line utility for searching plain-text files for lines matching a regular expression:

## Colorize the grep command output for ease of use (good for log files)##
alias grep='grep --color=auto'
alias egrep='egrep --color=auto'
alias fgrep='fgrep --color=auto'

#4: Start calculator with math support

alias bc='bc -l'

#4: Generate sha1 digest

alias sha1='openssl sha1'

#5: Create parent directories on demand

mkdir command is used to create a directory:

alias mkdir='mkdir -pv'

#6: Colorize diff output

You can compare files line by line using diff and use a tool called colordiff to colorize diff output:

# install colordiff package 🙂
alias diff='colordiff'

#7: Make mount command output pretty and human readable format

alias mount='mount |column -t'

#8: Command short cuts to save time

# handy short cuts #
alias h='history'
alias j='jobs -l'

#9: Create a new set of commands

alias path='echo -e ${PATH//:/\\n}'
alias now='date +"%T"'
alias nowtime=now
alias nowdate='date +"%d-%m-%Y"'

#10: Set vim as default

alias vi=vim
alias svi='sudo vi'
alias vis='vim "+set si"'
alias edit='vim'

#11: Control output of networking tool called ping

# Stop after sending count ECHO_REQUEST packets #
alias ping='ping -c 5'
# Do not wait interval 1 second, go fast #
alias fastping='ping -c 100 -s.2'

#12: Show open ports

Use netstat command to quickly list all TCP/UDP port on the server:

alias ports='netstat -tulanp'

#13: Wakeup sleeping servers

Wake-on-LAN (WOL) is an Ethernet networking standard that allows a server to be turned on by a network message. You can quickly wakeup nas devices and server using the following aliases:

## replace mac with your actual server mac address #
alias wakeupnas01='/usr/bin/wakeonlan 00:11:32:11:15:FC'
alias wakeupnas02='/usr/bin/wakeonlan 00:11:32:11:15:FD'
alias wakeupnas03='/usr/bin/wakeonlan 00:11:32:11:15:FE'

#14: Control firewall (iptables) output

Netfilter is a host-based firewall for Linux operating systems. It is included as part of the Linux distribution and it is activated by default. This post list most common iptables solutions required by a new Linux user to secure his or her Linux operating system from intruders.

## shortcut for iptables and pass it via sudo#
alias ipt='sudo /sbin/iptables'
 
# display all rules #
alias iptlist='sudo /sbin/iptables -L -n -v --line-numbers'
alias iptlistin='sudo /sbin/iptables -L INPUT -n -v --line-numbers'
alias iptlistout='sudo /sbin/iptables -L OUTPUT -n -v --line-numbers'
alias iptlistfw='sudo /sbin/iptables -L FORWARD -n -v --line-numbers'
alias firewall=iptlist

#15: Debug web server / cdn problems with curl

# get web server headers #
alias header='curl -I'
 
# find out if remote server supports gzip / mod_deflate or not #
alias headerc='curl -I --compress'

#16: Add safety nets

# do not delete / or prompt if deleting more than 3 files at a time #
alias rm='rm -I --preserve-root'
 
# confirmation #
alias mv='mv -i'
alias cp='cp -i'
alias ln='ln -i'
 
# Parenting changing perms on / #
alias chown='chown --preserve-root'
alias chmod='chmod --preserve-root'
alias chgrp='chgrp --preserve-root'

#17: Update Debian Linux server

apt-get command is used for installing packages over the internet (ftp or http). You can also upgrade all packages in a single operations:

# distro specific - Debian / Ubuntu and friends #
# install with apt-get
alias apt-get="sudo apt-get"
alias updatey="sudo apt-get --yes"
 
# update on one command
alias update='sudo apt-get update && sudo apt-get upgrade'

#18: Update RHEL / CentOS / Fedora Linux server

yum command is a package management tool for RHEL / CentOS / Fedora Linux and friends:

## distrp specifc RHEL/CentOS ##
alias update='yum update'
alias updatey='yum -y update'

#19: Tune sudo and su

# become root #
alias root='sudo -i'
alias su='sudo -i'

#20: Pass halt/reboot via sudo

shutdown command bring the Linux / Unix system down:

# reboot / halt / poweroff
alias reboot='sudo /sbin/reboot'
alias poweroff='sudo /sbin/poweroff'
alias halt='sudo /sbin/halt'
alias shutdown='sudo /sbin/shutdown'

#21: Control web servers

# also pass it via sudo so whoever is admin can reload it without calling you #
alias nginxreload='sudo /usr/local/nginx/sbin/nginx -s reload'
alias nginxtest='sudo /usr/local/nginx/sbin/nginx -t'
alias lightyload='sudo /etc/init.d/lighttpd reload'
alias lightytest='sudo /usr/sbin/lighttpd -f /etc/lighttpd/lighttpd.conf -t'
alias httpdreload='sudo /usr/sbin/apachectl -k graceful'
alias httpdtest='sudo /usr/sbin/apachectl -t && /usr/sbin/apachectl -t -D DUMP_VHOSTS'

#22: Alias into our backup stuff

# if cron fails or if you want backup on demand just run these commands #
# again pass it via sudo so whoever is in admin group can start the job #
# Backup scripts #
alias backup='sudo /home/scripts/admin/scripts/backup/wrapper.backup.sh --type local --taget /raid1/backups'
alias nasbackup='sudo /home/scripts/admin/scripts/backup/wrapper.backup.sh --type nas --target nas01'
alias s3backup='sudo /home/scripts/admin/scripts/backup/wrapper.backup.sh --type nas --target nas01 --auth /home/scripts/admin/.authdata/amazon.keys'
alias rsnapshothourly='sudo /home/scripts/admin/scripts/backup/wrapper.rsnapshot.sh --type remote --target nas03 --auth /home/scripts/admin/.authdata/ssh.keys --config /home/scripts/admin/scripts/backup/config/adsl.conf'
alias rsnapshotdaily='sudo /home/scripts/admin/scripts/backup/wrapper.rsnapshot.sh --type remote --target nas03 --auth /home/scripts/admin/.authdata/ssh.keys --config /home/scripts/admin/scripts/backup/config/adsl.conf'
alias rsnapshotweekly='sudo /home/scripts/admin/scripts/backup/wrapper.rsnapshot.sh --type remote --target nas03 --auth /home/scripts/admin/.authdata/ssh.keys --config /home/scripts/admin/scripts/backup/config/adsl.conf'
alias rsnapshotmonthly='sudo /home/scripts/admin/scripts/backup/wrapper.rsnapshot.sh --type remote --target nas03 --auth /home/scripts/admin/.authdata/ssh.keys --config /home/scripts/admin/scripts/backup/config/adsl.conf'
alias amazonbackup=s3backup

#23: Desktop specific – play avi/mp3 files on demand

## play video files in a current directory ##
# cd ~/Download/movie-name
# playavi or vlc
alias playavi='mplayer *.avi'
alias vlc='vlc *.avi'
 
# play all music files from the current directory #
alias playwave='for i in *.wav; do mplayer "$i"; done'
alias playogg='for i in *.ogg; do mplayer "$i"; done'
alias playmp3='for i in *.mp3; do mplayer "$i"; done'
 
# play files from nas devices #
alias nplaywave='for i in /nas/multimedia/wave/*.wav; do mplayer "$i"; done'
alias nplayogg='for i in /nas/multimedia/ogg/*.ogg; do mplayer "$i"; done'
alias nplaymp3='for i in /nas/multimedia/mp3/*.mp3; do mplayer "$i"; done'
 
# shuffle mp3/ogg etc by default #
alias music='mplayer --shuffle *'

#24: Set default interfaces for sys admin related commands

vnstat is console-based network traffic monitor. dnstop is console tool to analyze DNS traffic. tcptrack and iftop commands displays information about TCP/UDP connections it sees on a network interface and display bandwidth usage on an interface by host respectively.

## All of our servers eth1 is connected to the Internets via vlan / router etc ##
alias dnstop='dnstop -l 5 eth1'
alias vnstat='vnstat -i eth1'
alias iftop='iftop -i eth1'
alias tcpdump='tcpdump -i eth1'
alias ethtool='ethtool eth1'
 
# work on wlan0 by default #
# Only useful for laptop as all servers are without wireless interface
alias iwconfig='iwconfig wlan0'

#25: Get system memory, cpu usage, and gpu memory info quickly

## pass options to free ##
alias meminfo='free -m -l -t'
 
## get top process eating memory
alias psmem='ps auxf | sort -nr -k 4'
alias psmem10='ps auxf | sort -nr -k 4 | head -10'
 
## get top process eating cpu ##
alias pscpu='ps auxf | sort -nr -k 3'
alias pscpu10='ps auxf | sort -nr -k 3 | head -10'
 
## Get server cpu info ##
alias cpuinfo='lscpu'
 
## older system use /proc/cpuinfo ##
##alias cpuinfo='less /proc/cpuinfo' ##
 
## get GPU ram on desktop / laptop##
alias gpumeminfo='grep -i --color memory /var/log/Xorg.0.log'

#26: Control Home Router

The curl command can be used to reboot Linksys routers.

# Reboot my home Linksys WAG160N / WAG54 / WAG320 / WAG120N Router / Gateway from *nix.
alias rebootlinksys="curl -u 'admin:my-super-password' 'http://192.168.1.2/setup.cgi?todo=reboot'"
 
# Reboot tomato based Asus NT16 wireless bridge
alias reboottomato="ssh admin@192.168.1.1 /sbin/reboot"

#27 Resume wget by default

The GNU Wget is a free utility for non-interactive download of files from the Web. It supports HTTP, HTTPS, and FTP protocols, and it can resume downloads too:

## this one saved by butt so many times ##
alias wget='wget -c'

#28 Use different browser for testing website

## this one saved by butt so many times ##
alias ff4='/opt/firefox4/firefox'
alias ff13='/opt/firefox13/firefox'
alias chrome='/opt/google/chrome/chrome'
alias opera='/opt/opera/opera'
 
#default ff
alias ff=ff13
 
#my default browser
alias browser=chrome

#29: A note about ssh alias

Do not create ssh alias, instead use ~/.ssh/config OpenSSH SSH client configuration files. It offers more option. An example:

Host server10
  Hostname 1.2.3.4
  IdentityFile ~/backups/.ssh/id_dsa
  user foobar
  Port 30000
  ForwardX11Trusted yes
  TCPKeepAlive yes

You can now connect to peer1 using the following syntax:
$ ssh server10

#30: It’s your turn to share…

## set some other defaults ##
alias df='df -H'
alias du='du -ch'
 
# top is atop, just like vi is vim
alias top='atop'
 
## nfsrestart - must be root ##
## refresh nfs mount / cache etc for Apache ##
alias nfsrestart='sync && sleep 2 && /etc/init.d/httpd stop && umount netapp2:/exports/http && sleep 2 && mount -o rw,sync,rsize=32768,wsize=32768,intr,hard,proto=tcp,fsc natapp2:/exports /http/var/www/html && /etc/init.d/httpd start'
 
## Memcached server status ##
alias mcdstats='/usr/bin/memcached-tool 10.10.27.11:11211 stats'
alias mcdshow='/usr/bin/memcached-tool 10.10.27.11:11211 display'
 
## quickly flush out memcached server ##
alias flushmcd='echo "flush_all" | nc 10.10.27.11 11211'
 
## Remove assets quickly from Akamai / Amazon cdn ##
alias cdndel='/home/scripts/admin/cdn/purge_cdn_cache --profile akamai'
alias amzcdndel='/home/scripts/admin/cdn/purge_cdn_cache --profile amazon'
 
## supply list of urls via file or stdin
alias cdnmdel='/home/scripts/admin/cdn/purge_cdn_cache --profile akamai --stdin'
alias amzcdnmdel='/home/scripts/admin/cdn/purge_cdn_cache --profile amazon --stdin'

Conclusion

This post summarizes several types of uses for *nix bash aliases:

1. Setting default options for a command (e.g. set eth0 as default option for ethtool command via alias ethtool='ethtool eth0' ).
2. Correcting typos (cd.. will act as cd .. via alias cd..='cd ..').
3. Reducing the amount of typing.
4. Setting the default path of a command that exists in several versions on a system (e.g. GNU/grep is located at /usr/local/bin/grep and Unix grep is located at /bin/grep. To use GNU grep use alias grep='/usr/local/bin/grep' ).
5. Adding the safety nets to Unix by making commands interactive by setting default options. (e.g. rm, mv, and other commands).
6. Compatibility by creating commands for older operating systems such as MS-DOS or other Unix like operating systems (e.g. alias del=rm ).

I’ve shared my aliases that I used over the years to reduce the need for repetitive command line typing. If you know and use any other bash/ksh/csh aliases that can reduce typing, share below in the comments.

How to check file size in unix using wc command

The wc command shows the number of lines, words, and bytes contained in file. The syntax is as follows to get the file size:
wc -c /path/to/file
wc -c /etc/passwd
Sample outputs:

5253 /etc/passwd

You can easily extract the first field either using the cut or awk command:
wc -c /etc/passwd | awk '{print $1}'
Sample outputs:

5253

OR assign this size to a bash variable:

myfilesize=$(wc -c "/etc/passwd" | awk '{print $1}')
printf "%d\n" $myfilesize
echo "$myfilesize"

How to get the size of a file in a bash script using stat command

The stat command shows information about the file. The syntax is as follows to get the file size on GNU/Linux stat:
stat -c %s "/etc/passwd"
OR
stat --format=%s "/etc/passwd"
To assign this size to a bash variable:

myfilesize=$(stat --format=%s "/etc/passwd")
echo "$myfilesize"
## or ##
myFileSizeCheck=$(stat -c %s "/etc/resolv.conf")
printf "My file size = %d\n" $myFileSizeCheck

The syntax is as follows to get the file size on BSD/MacOS stat:
stat -f %z "/etc/passwd"
Please note that if the file is symlink you will get size of that link only with the stat command.

du command example

The syntax is

du --apparent-size --block-size=1  "/etc/passwd"
fileName="/etc/hosts"
mfs=$(du --apparent-size --block-size=1  "$fileName" | awk '{ print $1}')
echo "$fileName size = ${mfs}"

Sample outputs from above commands:

Fig.01: How to check size of a file using a bash/ksh/zsh/sh/tcsh shell?

Find command example

The syntax is:

find "/etc/passwd" -printf "%s"
find "/etc/passwd" -printf "%s\n"
fileName="/etc/hosts"
mysize=$(find "$fileName" -printf "%s")
printf "File %s size = %d\n" $fileName $mysize
echo "${fileName} size is ${mysize} bytes."

You can use standard Unix commands such as sed, awk, grep, Perl, Python and more to get domain name from URL. No need to write regex. It is pretty simple.

Let use see various commands and option to grab the domain part from given variable under Linux or Unix-like system.

Get domain name from full URL

Say your url name is stored in a bash shell variable such as $x:
x='https://www.dbsysnet.com/'
You can use the awk as follows:
echo "$x" | awk -F/ '{print $3}'
### OR ###
awk -F/ '{print $3}' <<<$x
Sample outputs:

www.dbsysnet.com

Extract domain name from URL using sed

Here is a sample sed command:
url="https://www.dbsysnet.com"
echo "$url" | sed -e 's|^[^/]*//||' -e 's|/.*$||'

Extract domain name from URL using bash shell parameter substitution

Another option is to use bash shell parameter substitution:

# My shell variable 
f="https://www.cyberciti.biz/faq/copy-command/"
 
## Remove protocol part of url ##
f="${f#http://}"
f="${f#https://}"
f="${f#ftp://}"
f="${f#scp://}"
f="${f#scp://}"
f="${f#sftp://}"
 
## Remove username and/or username:password part of URL ##
f="${f#*:*@}"
f="${f#*@}"
 
## Remove rest of urls ##
f=${f%%/*}
 
## Show domain name only ##
echo "$f"

Shell script example

A shell script to purge urls from Cloudflare by matching domain name part:

#!/bin/bash
zone_id=""
api_key=""
 
urls="$@"
bon=$(tput bold)
boff=$(tput sgr0)
c=1
[ "$urls" == "" ] && { echo "Usage: $0 url"; exit 1; }
 
clear
echo "Purging..."
echo
for u in $urls
do
     echo -n "${bon}${c}${boff}.${u}: "
     ## Get domain name ##
     d="$(echo $u | awk -F/ '{ print $3}')"
     ## Set API_KEY, Email_ID, and ZONE_ID as per domain ##
     case $d in
	     www.cyberciti.biz) zone_id="ID_1"; api_key="MY_KEY_1"; email_id="webmaster@cyberciti.biz";;
	     theos.in) zone_id="ID_2"; api_key="MY_KEY_2"; email_id="webmaster@theos.in";;
	     *) echo "Domain not configured."; continue;;
     esac
     ## Do it ##
     curl -X DELETE "https://api.cloudflare.com/client/v4/zones/${zone_id}/purge_cache" \
     -H "X-Auth-Email: ${email_id}" \
     -H "X-Auth-Key: ${api_key}" \
     -H "Content-Type: application/json" \
     --data "{\"files\":[\"${u}\"]}"
     echo
     (( c++ ))
done
echo

I want to find out how many times a word (say foo or an IP address) occurs in a text file using the grep command on Linux or Unix-like system?

You can use the grep command to search strings, words, text, and numbers for a given patterns. You can pass the -coption to grep command. It only shows the number of times that the pattern has been matched for each file.

Show the total number of times that the word foo appears in a file named bar.txt

The syntax is:
grep -c string filename
grep -c foo bar.txt
Sample outputs:

3

To count total number of occurrences of word in a file named /etc/passwd root using grep, run:
grep -c root /etc/passwd
To verify that run:
grep --color root /etc/passwd
Pass the -w option to grep to select only an entire word or phrase that matches the specified pattern:
grep -w root /etc/passwd
OR
grep -c -w root /etc/passwd
In this example only match a word being with root:
grep --color -w '^root' /etc/passwd
grep -c -w '^root' /etc/passwd
To show only the matching part of the lines.
grep -o 'root' /etc/passwd
grep -c -o 'root' /etc/passwd
Sample session:

Fig.01: Counting occurrence of words/strings using grep command

I type ‘find . -type d -iname foo -delete‘ command to find the foo directories and delete them. However, I am getting an error message that read as find: cannot delete './hourly.4/data/foo': Directory not empty on Linux server. How do delete directories based on find command output on Linux or Unix-like system?

The -delete option remove the DIRECTORY(ies), if they are empty. You need to use the -execoption to delete all directories and its contents. The syntax is as follows.

Find command syntax to delete dirs

Try:
find /dir/to/search/ -type d -name "dirName" -exec rm -rf {} +
OR
find /dir/to/search/ -type d -name "dirName" -exec rm -rf \;

Warning: Be careful with the rm command when using with find. You may end up deleting unwanted data.

Find will execute given command when it finds files or dirs. For example:
find . -type d -name "foo" -exec rm -rf {} +
OR
find . -type d -name "bar" -exec rm -rf "{}" \;
Sample outputs:

removed './daily.0/bar/.cache/motd.legal-displayed'
removed directory './daily.0/bar/root/.cache'
removed './daily.0/bar/.lesshst'
removed './daily.0/bar/.viminfo'
removed './daily.0/bar/.vim/.netrwhist'
removed directory './daily.0/bar/root/.vim'
removed './daily.0/bar/root/.bashrc'
removed './daily.0/bar/.ssh/authorized_keys'
removed directory './daily.0/bar/root'
removed directory './daily.0/bar/var/spool/cron/crontabs'

You can find directories that are at least four levels deep in the working directory /backups/:
find /backups/ -type d -name "bar" -depth +4 -print0 -exec rm -rf {} +

Find and xargs

The syntax is as follows to find and delete directories on Linux/Unix system:
## delete all empty dirs ##
find /path/to/dir/ -type d -empty -print0 | xargs -0 -I {} /bin/rm -rf "{}"
## delete all foo dirs including subdirs in /backups/
find /backups/ -type d -name "foo*" -print0 | xargs -0 -I {} /bin/rm -rf "{}"
The second command is secure and fast version as it deals with weird dir names such as:

• “foo bar”
• “Foo _ *bar”

I want to display a countdown before purging cache from CDN network. Is there an existing command to show a conuntdown from 30..1 as 30,29,28,…1 on Linux or Unix bash shell script?

There are various ways to show a countdown in your shell scripts. 

First define your message:
msg="Purging cache please wait..."
Now clear the screen and display the message at row 10 and column 5 using tput:
clear
tput cup 10 5
Next you need to display the message:
echo -n "$msg"
Find out the length of string:
l=${#msg}
Calculate the next column:
l=$(( l+5 ))
Finally use a bash for loop to show countdown:
for i in {30..01}
do
tput cup 10 $l
echo -n "$i"
sleep 1
done
echo
Here is a complete shell script:

#!/bin/bash
# Purpose: Purge urls from Cloudflare Cache
# Author: Vivek Gite {www.cyberciti.biz} under GPL v2.x+
# --------------------------------------------------------
# Set me first #
zone_id="My-ID"
api_key="My_API_KEY"
email_id="My_EMAIL_ID"
row=2
col=2
urls="$@"
countdown() {
        msg="Purging ${1}..."
        clear
        tput cup $row $col
        echo -n "$msg"
        l=${#msg}
        l=$(( l+$col ))
        for i in {30..1}
        do
                tput cup $row $l
                echo -n "$i"
                sleep 1
        done
}
# Do it
for u in $urls
do
     amp_url="${u}amp/"
     curl -X DELETE "https://api.cloudflare.com/client/v4/zones/${zone_id}/purge_cache" \
     -H "X-Auth-Email: ${email_id}" \
     -H "X-Auth-Key: ${api_key}" \
     -H "Content-Type: application/json" \
     --data "{\"files\":[\"${u}\",\"${amp_url}\"]}" &>/dev/null &&  countdown "$u"
 
done
echo

You can run it as follows:
./script.sh url1 url2

POSIX shell version

From this post:

countdown()
(
  IFS=:
  set -- $*
  secs=$(( ${1#0} * 3600 + ${2#0} * 60 + ${3#0} ))
  while [ $secs -gt 0 ]
  do
    sleep 1 &
    printf "\r%02d:%02d:%02d" $((secs/3600)) $(( (secs/60)%60)) $((secs%60))
    secs=$(( $secs - 1 ))
    wait
  done
  echo
)

It can be run as follows:
countdown "00:00:10" # 10 sec
countdown "00:00:30" # 30 sec
countdown "00:01:42" # 1 min 42 sec

How do I replace a string with another string in all files? For example, ~/foo directory has 100s of text file and I’d like to find out xyz string and replace with abc. I’d like to use sed or any other tool to replace all occurrence of the word.

The sed command is designed for this kind of work i.e. find and replace strings or words from a text file under Apple OX, *BSD, Linux, and UNIX like operating systems. The perl can be also used as described below.

sed replace word / string syntax

The syntax is as follows:
sed -i 's/old-word/new-word/g' *.txt

GNU sed command can edit files in place (makes backup if extension supplied) using the -i option. If you are using an old UNIX sed command version try the following syntax:

sed 's/old/new/g' input.txt > output.txt

You can use old sed syntax along with bash for loop:

#!/bin/bash
OLD="xyz"
NEW="abc"
DPATH="/home/you/foo/*.txt"
BPATH="/home/you/bakup/foo"
TFILE="/tmp/out.tmp.$$"
[ ! -d $BPATH ] && mkdir -p $BPATH || :
for f in $DPATH
do
  if [ -f $f -a -r $f ]; then
    /bin/cp -f $f $BPATH
   sed "s/$OLD/$NEW/g" "$f" > $TFILE && mv $TFILE "$f"
  else
   echo "Error: Cannot read $f"
  fi
done
/bin/rm $TFILE

A Note About Bash Escape Character

A non-quoted backslash \ is the Bash escape character. It preserves the literal value of the next character that follows, with the exception of newline. If a \newline pair appears, and the backslash itself is not quoted, the \newline is treated as a line continuation (that is, it is removed from the input stream and effectively ignored). This is useful when you would like to deal with UNIX paths. In this example, the sed command is used to replace UNIX path “/nfs/apache/logs/rawlogs/access.log” with “__DOMAIN_LOG_FILE__”:

#!/bin/bash
## Our path
_r1="/nfs/apache/logs/rawlogs/access.log"
 
## Escape path for sed using bash find and replace 
_r1="${_r1//\//\\/}"
 
# replace __DOMAIN_LOG_FILE__ in our sample.awstats.conf
sed -e "s/__DOMAIN_LOG_FILE__/${_r1}/" /nfs/conf/awstats/sample.awstats.conf  > /nfs/apache/logs/awstats/awstats.conf 
 
# call awstats
/usr/bin/awstats -c /nfs/apache/logs/awstats/awstats.conf

The $_r1 is escaped using bash find and replace parameter substitution syntax to replace each occurrence of / with \/.

perl -pie Syntax For Find and Replace

The syntax is as follows:
perl -pie 's/old-word/new-word/g' input.file > new.output.file