Databases, Systems & Networks

I just had a problem to solve: Compare two server config files on two servers to make sure they’re the same. Rather than using scp to copy the file from one machine to another, I used ssh’s ability to run commands remotely to get the contents of the file and piped it into diff. Here’s an example:

ssh user@server1 'cat /path/to/config/file.conf' | diff /path/to/other/config/file.conf -

I’ll leave it as an exercise for the user to write a shell script that will do this automatically, though it should be fairly easy to do.

Source: movetoiceland.com

I ended up writing a script (I’m trying to teach myself the finer points of bash scripting, so that’s why you don’t see something like Perl here). It’s not exactly a simple affair, but it does most of what I need. It uses Kyle’s suggestion for looking up individual users’ crontabs, but also deals with /etc/crontab(including the scripts launched by run-parts in /etc/cron.hourly, /etc/cron.daily, etc.) and the jobs in the /etc/cron.d directory.source: How do I list all cron jobs for all users?

I ended up writing a script (I’m trying to teach myself the finer points of bash scripting, so that’s why you don’t see something like Perl here). It’s not exactly a simple affair, but it does most of what I need. It uses Kyle’s suggestion for looking up individual users’ crontabs, but also deals with /etc/crontab(including the scripts launched by run-parts in /etc/cron.hourly, /etc/cron.daily, etc.) and the jobs in the /etc/cron.d directory.

It takes all of those and merges them into a display something like the following:

#!/bin/bash
# System-wide crontab file and cron job directory. Change these for your system.
 CRONTAB='/etc/crontab'
 CRONDIR='/etc/cron.d'
# Single tab character. Annoyingly necessary.
 tab=$(echo -en "\t")
# Given a stream of crontab lines, exclude non-cron job lines, replace
 # whitespace characters with a single space, and remove any spaces from the
 # beginning of each line.
 function clean_cron_lines() {
 while read line ; do
 echo "${line}" |
 egrep --invert-match '^($|\s*#|\s*[[:alnum:]_]+=)' |
 sed --regexp-extended "s/\s+/ /g" |
 sed --regexp-extended "s/^ //"
 done;
 }
# Given a stream of cleaned crontab lines, echo any that don't include the
 # run-parts command, and for those that do, show each job file in the run-parts
 # directory as if it were scheduled explicitly.
 function lookup_run_parts() {
 while read line ; do
 match=$(echo "${line}" | egrep -o 'run-parts (-{1,2}\S+ )*\S+')
if [[ -z "${match}" ]] ; then
 echo "${line}"
 else
 cron_fields=$(echo "${line}" | cut -f1-6 -d' ')
 cron_job_dir=$(echo "${match}" | awk '{print $NF}')
if [[ -d "${cron_job_dir}" ]] ; then
 for cron_job_file in "${cron_job_dir}"/* ; do # */
 [[ -f "${cron_job_file}" ]] && echo "${cron_fields} ${cron_job_file}"
 done
 fi
 fi
 done;
 }
# Temporary file for crontab lines.
 temp=$(mktemp) || exit 1
# Add all of the jobs from the system-wide crontab file.
 cat "${CRONTAB}" | clean_cron_lines | lookup_run_parts >"${temp}"
# Add all of the jobs from the system-wide cron directory.
 cat "${CRONDIR}"/* | clean_cron_lines >>"${temp}" # */
# Add each user's crontab (if it exists). Insert the user's name between the
 # five time fields and the command.
 while read user ; do
 crontab -l -u "${user}" 2>/dev/null |
 clean_cron_lines |
 sed --regexp-extended "s/^((\S+ +){5})(.+)$/\1${user} \3/" >>"${temp}"
done <
# Output the collected crontab lines. Replace the single spaces between the
 # fields with tab characters, sort the lines by hour and minute, insert the
 # header line, and format the results as a table.
 cat "${temp}" |
 sed --regexp-extended "s/^(\S+) +(\S+) +(\S+) +(\S+) +(\S+) +(\S+) +(.*)$/\1\t\2\t\3\t\4\t\5\t\6\t\7/" |
 sort --numeric-sort --field-separator="${tab}" --key=2,1 |
 sed "1i\mi\th\td\tm\tw\tuser\tcommand" |
 column -s"${tab}" -t
rm --force "${temp}"

source: How do I list all cron jobs for all users?

An alias is nothing but the shortcut to commands. The alias command allows the user to launch any command or group of commands (including options and filenames) by entering a single word. Use alias command to display a list of all defined aliases. You can add user-defined aliases to ~/.bashrcfile. You can cut down typing time with these aliases, work smartly, and increase productivity at the command prompt.

More about aliases

The general syntax for the alias command for the bash shell is as follows:

Task: List aliases

Type the following command:

alias

Sample outputs:

alias ..='cd ..'
alias amazonbackup='s3backup'
alias apt-get='sudo apt-get'
...

By default alias command shows a list of aliases that are defined for the current user.

Lire la suite…

I want to display a countdown before purging cache from CDN network. Is there an existing command to show a conuntdown from 30..1 as 30,29,28,…1 on Linux or Unix bash shell script?

There are various ways to show a countdown in your shell scripts. 

First define your message:
msg="Purging cache please wait..."
Now clear the screen and display the message at row 10 and column 5 using tput:
clear
tput cup 10 5
Next you need to display the message:
echo -n "$msg"
Find out the length of string:
l=${#msg}
Calculate the next column:
l=$(( l+5 ))
Finally use a bash for loop to show countdown:
for i in {30..01}
do
tput cup 10 $l
echo -n "$i"
sleep 1
done
echo
Here is a complete shell script:

#!/bin/bash
# Purpose: Purge urls from Cloudflare Cache
# Author: Vivek Gite {www.cyberciti.biz} under GPL v2.x+
# --------------------------------------------------------
# Set me first #
zone_id="My-ID"
api_key="My_API_KEY"
email_id="My_EMAIL_ID"
row=2
col=2
urls="$@"
countdown() {
        msg="Purging ${1}..."
        clear
        tput cup $row $col
        echo -n "$msg"
        l=${#msg}
        l=$(( l+$col ))
        for i in {30..1}
        do
                tput cup $row $l
                echo -n "$i"
                sleep 1
        done
}
# Do it
for u in $urls
do
     amp_url="${u}amp/"
     curl -X DELETE "https://api.cloudflare.com/client/v4/zones/${zone_id}/purge_cache" \
     -H "X-Auth-Email: ${email_id}" \
     -H "X-Auth-Key: ${api_key}" \
     -H "Content-Type: application/json" \
     --data "{\"files\":[\"${u}\",\"${amp_url}\"]}" &>/dev/null &&  countdown "$u"
 
done
echo

You can run it as follows:
./script.sh url1 url2

POSIX shell version

From this post:

countdown()
(
  IFS=:
  set -- $*
  secs=$(( ${1#0} * 3600 + ${2#0} * 60 + ${3#0} ))
  while [ $secs -gt 0 ]
  do
    sleep 1 &
    printf "\r%02d:%02d:%02d" $((secs/3600)) $(( (secs/60)%60)) $((secs%60))
    secs=$(( $secs - 1 ))
    wait
  done
  echo
)

It can be run as follows:
countdown "00:00:10" # 10 sec
countdown "00:00:30" # 30 sec
countdown "00:01:42" # 1 min 42 sec

How do I replace a string with another string in all files? For example, ~/foo directory has 100s of text file and I’d like to find out xyz string and replace with abc. I’d like to use sed or any other tool to replace all occurrence of the word.

The sed command is designed for this kind of work i.e. find and replace strings or words from a text file under Apple OX, *BSD, Linux, and UNIX like operating systems. The perl can be also used as described below.

sed replace word / string syntax

The syntax is as follows:
sed -i 's/old-word/new-word/g' *.txt

GNU sed command can edit files in place (makes backup if extension supplied) using the -i option. If you are using an old UNIX sed command version try the following syntax:

sed 's/old/new/g' input.txt > output.txt

You can use old sed syntax along with bash for loop:

#!/bin/bash
OLD="xyz"
NEW="abc"
DPATH="/home/you/foo/*.txt"
BPATH="/home/you/bakup/foo"
TFILE="/tmp/out.tmp.$$"
[ ! -d $BPATH ] && mkdir -p $BPATH || :
for f in $DPATH
do
  if [ -f $f -a -r $f ]; then
    /bin/cp -f $f $BPATH
   sed "s/$OLD/$NEW/g" "$f" > $TFILE && mv $TFILE "$f"
  else
   echo "Error: Cannot read $f"
  fi
done
/bin/rm $TFILE

A Note About Bash Escape Character

A non-quoted backslash \ is the Bash escape character. It preserves the literal value of the next character that follows, with the exception of newline. If a \newline pair appears, and the backslash itself is not quoted, the \newline is treated as a line continuation (that is, it is removed from the input stream and effectively ignored). This is useful when you would like to deal with UNIX paths. In this example, the sed command is used to replace UNIX path “/nfs/apache/logs/rawlogs/access.log” with “__DOMAIN_LOG_FILE__”:

#!/bin/bash
## Our path
_r1="/nfs/apache/logs/rawlogs/access.log"
 
## Escape path for sed using bash find and replace 
_r1="${_r1//\//\\/}"
 
# replace __DOMAIN_LOG_FILE__ in our sample.awstats.conf
sed -e "s/__DOMAIN_LOG_FILE__/${_r1}/" /nfs/conf/awstats/sample.awstats.conf  > /nfs/apache/logs/awstats/awstats.conf 
 
# call awstats
/usr/bin/awstats -c /nfs/apache/logs/awstats/awstats.conf

The $_r1 is escaped using bash find and replace parameter substitution syntax to replace each occurrence of / with \/.

perl -pie Syntax For Find and Replace

The syntax is as follows:
perl -pie 's/old-word/new-word/g' input.file > new.output.file