Archive

Articles taggués ‘commands’

How to check the file size in Linux/Unix bash shell scripting

28/09/2017 Comments off

 

How to check file size in unix using wc command

The wc command shows the number of lines, words, and bytes contained in file. The syntax is as follows to get the file size:
wc -c /path/to/file
wc -c /etc/passwd

Sample outputs:

5253 /etc/passwd

You can easily extract the first field either using the cut or awk command:
wc -c /etc/passwd | awk '{print $1}'
Sample outputs:

5253

OR assign this size to a bash variable:

myfilesize=$(wc -c "/etc/passwd" | awk '{print $1}')
printf "%d\n" $myfilesize
echo "$myfilesize"

How to get the size of a file in a bash script using stat command

The stat command shows information about the file. The syntax is as follows to get the file size on GNU/Linux stat:
stat -c %s "/etc/passwd"
OR
stat --format=%s "/etc/passwd"
To assign this size to a bash variable:

myfilesize=$(stat --format=%s "/etc/passwd")
echo "$myfilesize"
## or ##
myFileSizeCheck=$(stat -c %s "/etc/resolv.conf")
printf "My file size = %d\n" $myFileSizeCheck

The syntax is as follows to get the file size on BSD/MacOS stat:
stat -f %z "/etc/passwd"
Please note that if the file is symlink you will get size of that link only with the stat command.

du command example

The syntax is

du --apparent-size --block-size=1  "/etc/passwd"
fileName="/etc/hosts"
mfs=$(du --apparent-size --block-size=1  "$fileName" | awk '{ print $1}')
echo "$fileName size = ${mfs}"

Sample outputs from above commands:

Fig.01: How to check size of a file using a bash/ksh/zsh/sh/tcsh shell?Fig.01: How to check size of a file using a bash/ksh/zsh/sh/tcsh shell?

 

Find command example

The syntax is:

find "/etc/passwd" -printf "%s"
find "/etc/passwd" -printf "%s\n"
fileName="/etc/hosts"
mysize=$(find "$fileName" -printf "%s")
printf "File %s size = %d\n" $fileName $mysize
echo "${fileName} size is ${mysize} bytes."
Categories: Système Tags: , , , , ,

How to get domain name from URL in bash shell script

28/09/2017 Comments off
How can I extract or fetch a domain name from a URL string (e.g. https://www.cyberciti.biz/index.php) using bash shell scripting under Linux or Unix-like operating system?

You can use standard Unix commands such as sed, awk, grep, Perl, Python and more to get domain name from URL. No need to write regex. It is pretty simple.

Let use see various commands and option to grab the domain part from given variable under Linux or Unix-like system.

Get domain name from full URL

Say your url name is stored in a bash shell variable such as $x:
x='https://www.dbsysnet.com/'
You can use the awk as follows:
echo "$x" | awk -F/ '{print $3}'
### OR ###
awk -F/ '{print $3}' <<<$x

Sample outputs:

www.dbsysnet.com

Extract domain name from URL using sed

Here is a sample sed command:
url="https://www.dbsysnet.com"
echo "$url" | sed -e 's|^[^/]*//||' -e 's|/.*$||'

Extract domain name from URL using bash shell parameter substitution

Another option is to use bash shell parameter substitution:

# My shell variable 
f="https://www.cyberciti.biz/faq/copy-command/"
 
## Remove protocol part of url ##
f="${f#http://}"
f="${f#https://}"
f="${f#ftp://}"
f="${f#scp://}"
f="${f#scp://}"
f="${f#sftp://}"
 
## Remove username and/or username:password part of URL ##
f="${f#*:*@}"
f="${f#*@}"
 
## Remove rest of urls ##
f=${f%%/*}
 
## Show domain name only ##
echo "$f"

Shell script example

A shell script to purge urls from Cloudflare by matching domain name part:

#!/bin/bash
zone_id=""
api_key=""
 
urls="$@"
bon=$(tput bold)
boff=$(tput sgr0)
c=1
[ "$urls" == "" ] && { echo "Usage: $0 url"; exit 1; }
 
clear
echo "Purging..."
echo
for u in $urls
do
     echo -n "${bon}${c}${boff}.${u}: "
     ## Get domain name ##
     d="$(echo $u | awk -F/ '{ print $3}')"
     ## Set API_KEY, Email_ID, and ZONE_ID as per domain ##
     case $d in
	     www.cyberciti.biz) zone_id="ID_1"; api_key="MY_KEY_1"; email_id="webmaster@cyberciti.biz";;
	     theos.in) zone_id="ID_2"; api_key="MY_KEY_2"; email_id="webmaster@theos.in";;
	     *) echo "Domain not configured."; continue;;
     esac
     ## Do it ##
     curl -X DELETE "https://api.cloudflare.com/client/v4/zones/${zone_id}/purge_cache" \
     -H "X-Auth-Email: ${email_id}" \
     -H "X-Auth-Key: ${api_key}" \
     -H "Content-Type: application/json" \
     --data "{\"files\":[\"${u}\"]}"
     echo
     (( c++ ))
done
echo
Categories: Système Tags: , , ,

How to count total number of word occurrences using grep on Linux or Unix

28/09/2017 Comments off

I want to find out how many times a word (say foo or an IP address) occurs in a text file using the grep command on Linux or Unix-like system?

You can use the grep command to search strings, words, text, and numbers for a given patterns. You can pass the -coption to grep command. It only shows the number of times that the pattern has been matched for each file.

 

 

 

Show the total number of times that the word foo appears in a file named bar.txt

The syntax is:
grep -c string filename
grep -c foo bar.txt

Sample outputs:

3

To count total number of occurrences of word in a file named /etc/passwd root using grep, run:
grep -c root /etc/passwd
To verify that run:
grep --color root /etc/passwd
Pass the -w option to grep to select only an entire word or phrase that matches the specified pattern:
grep -w root /etc/passwd
OR
grep -c -w root /etc/passwd
In this example only match a word being with root:
grep --color -w '^root' /etc/passwd
grep -c -w '^root' /etc/passwd

To show only the matching part of the lines.
grep -o 'root' /etc/passwd
grep -c -o 'root' /etc/passwd

Sample session:

Fig.01: Counting occurrence of words/strings using grep commandFig.01: Counting occurrence of words/strings using grep command

How to find and delete directory recursively on Linux or Unix-like system

28/09/2017 Comments off

I type ‘find . -type d -iname foo -delete‘ command to find the foo directories and delete them. However, I am getting an error message that read as find: cannot delete './hourly.4/data/foo': Directory not empty on Linux server. How do delete directories based on find command output on Linux or Unix-like system?



The -delete option remove the DIRECTORY(ies), if they are empty. You need to use the -execoption to delete all directories and its contents. The syntax is as follows.

Find command syntax to delete dirs

Try:
find /dir/to/search/ -type d -name "dirName" -exec rm -rf {} +
OR
find /dir/to/search/ -type d -name "dirName" -exec rm -rf \;

Warning: Be careful with the rm command when using with find. You may end up deleting unwanted data.

Find will execute given command when it finds files or dirs. For example:
find . -type d -name "foo" -exec rm -rf {} +
OR
find . -type d -name "bar" -exec rm -rf "{}" \;
Sample outputs:

removed './daily.0/bar/.cache/motd.legal-displayed'
removed directory './daily.0/bar/root/.cache'
removed './daily.0/bar/.lesshst'
removed './daily.0/bar/.viminfo'
removed './daily.0/bar/.vim/.netrwhist'
removed directory './daily.0/bar/root/.vim'
removed './daily.0/bar/root/.bashrc'
removed './daily.0/bar/.ssh/authorized_keys'
removed directory './daily.0/bar/root'
removed directory './daily.0/bar/var/spool/cron/crontabs'

You can find directories that are at least four levels deep in the working directory /backups/:
find /backups/ -type d -name "bar" -depth +4 -print0 -exec rm -rf {} +

Find and xargs

The syntax is as follows to find and delete directories on Linux/Unix system:
## delete all empty dirs ##
find /path/to/dir/ -type d -empty -print0 | xargs -0 -I {} /bin/rm -rf "{}"
## delete all foo dirs including subdirs in /backups/
find /backups/ -type d -name "foo*" -print0 | xargs -0 -I {} /bin/rm -rf "{}"

The second command is secure and fast version as it deals with weird dir names such as:

  • “foo bar”
  • “Foo _ *bar”
 
Categories: Système Tags: , , ,

Bash Shell: Replace a String With Another String In All Files Using sed and Perl -pie Options

28/09/2017 Comments off

String search and replace

How do I replace a string with another string in all files? For example, ~/foo directory has 100s of text file and I’d like to find out xyz string and replace with abc. I’d like to use sed or any other tool to replace all occurrence of the word.

The sed command is designed for this kind of work i.e. find and replace strings or words from a text file under Apple OX, *BSD, Linux, and UNIX like operating systems. The perl can be also used as described below.

sed replace word / string syntax

The syntax is as follows:
sed -i 's/old-word/new-word/g' *.txt

GNU sed command can edit files in place (makes backup if extension supplied) using the -i option. If you are using an old UNIX sed command version try the following syntax:

sed 's/old/new/g' input.txt > output.txt

You can use old sed syntax along with bash for loop:

#!/bin/bash
OLD="xyz"
NEW="abc"
DPATH="/home/you/foo/*.txt"
BPATH="/home/you/bakup/foo"
TFILE="/tmp/out.tmp.$$"
[ ! -d $BPATH ] && mkdir -p $BPATH || :
for f in $DPATH
do
  if [ -f $f -a -r $f ]; then
    /bin/cp -f $f $BPATH
   sed "s/$OLD/$NEW/g" "$f" > $TFILE && mv $TFILE "$f"
  else
   echo "Error: Cannot read $f"
  fi
done
/bin/rm $TFILE

A Note About Bash Escape Character

A non-quoted backslash \ is the Bash escape character. It preserves the literal value of the next character that follows, with the exception of newline. If a \newline pair appears, and the backslash itself is not quoted, the \newline is treated as a line continuation (that is, it is removed from the input stream and effectively ignored). This is useful when you would like to deal with UNIX paths. In this example, the sed command is used to replace UNIX path “/nfs/apache/logs/rawlogs/access.log” with “__DOMAIN_LOG_FILE__”:

#!/bin/bash
## Our path
_r1="/nfs/apache/logs/rawlogs/access.log"
 
## Escape path for sed using bash find and replace 
_r1="${_r1//\//\\/}"
 
# replace __DOMAIN_LOG_FILE__ in our sample.awstats.conf
sed -e "s/__DOMAIN_LOG_FILE__/${_r1}/" /nfs/conf/awstats/sample.awstats.conf  > /nfs/apache/logs/awstats/awstats.conf 
 
# call awstats
/usr/bin/awstats -c /nfs/apache/logs/awstats/awstats.conf

The $_r1 is escaped using bash find and replace parameter substitution syntax to replace each occurrence of / with \/.

perl -pie Syntax For Find and Replace

The syntax is as follows:
perl -pie 's/old-word/new-word/g' input.file > new.output.file

Categories: Système Tags: , , , ,

BASH: Vérifier qu’une commande existe

28/09/2017 Comments off

Comment vérifier qu’une commande existe en Bash ?
Comment vérifier la disponibilité des prérequis d’un script ?

Une méthode permettant de vérifier qu’une commande existe consiste à tester le retour de « command -v <commande_a_tester> » :

fhh@mafalda ~ $ command -v ls
alias ls='ls --color=auto'
fhh@mafalda ~ $ echo $?
0
fhh@mafalda ~ $ command -v ls > /dev/null && echo "ls existe" ls existe
 

Si la commande existe et est accessible via le PATH ou au chemin spécifié, « command -v » retourne « 0 », sinon, le retour est supérieur à « 0 » :

fhh@mafalda ~ $ command -v youhou
# youhou n'existe pas dans le PATH ou comme commande interne :
fhh@mafalda ~ $ echo $?
1
fhh@mafalda ~ $ command -v ../../usr/bin/date
# date est accessible au chemin spécifié :
fhh@mafalda ~ $ echo $?
0

Lire la suite…

Categories: Système Tags: , , ,

BASH : Suppression des accents, cédilles, etc

28/09/2017 Comments off

Comment supprimer les accents, cédilles, etc, dans une chaine de caractères ?

Méthode classique : la substitution

La suppression des caractères accentués et autres cédilles peut être effectuée, en Bash, en utilisant « sed » ou « tr » :

fhh@aaricia ~ $ _str="Une chaine avec des é, des Ù, des À, des ç et des œ"
fhh@aaricia ~ $ echo $_str | sed 'y/áàâäçéèêëîïìôöóùúüñÂÀÄÇÉÈÊËÎÏÔÖÙÜÑ/aaaaceeeeiiiooouuunAAACEEEEIIOOUUN/' Une chaine avec des e, des U, des A, des c et des œ

La méthode est fonctionnelle mais sous entend que tous les caractères à substituer aient été définis. Dans l’exemple, le « œ » n’a pas été remplacé car aucun caractère de remplacement ne lui est alloué.

Autre problème de cette méthode, remplacer une lettre par deux autres tel que « œ » par « oe » ou le « ß » allemand par « ss » nécessite la définition de règles particulières à chaque cas.

Ce sont ces raisons qui nous poussent à éviter cette méthode au profit de la conversion de chaines de caractères.

Méthode recommandée : la conversion

Plus complète, la méthode de conversion présente en sus l’avantage d’être plus concise.

« iconv » est utilisé pour « convertir » la chaine de caractères du format de base, UTF-8 dans l’exemple (option « -f » pour « from »), vers le format ASCII (option « -t » pour « to »).

Avec l’option « TRANSLIT », si un caractère ne peut être transcrit dans le format de destination, il est converti en une chaine de caractère équivalente.

fhh@aaricia ~ $ _str="Une chaine avec des é, des Ù, des À, des çÇ et des œ"
fhh@aaricia ~ $ echo $_str | iconv -f utf8 -t ascii//TRANSLIT Une chaine avec des e, des U, des A, des cC et des oe

La méthode fonctionne sur un large panel de caractères :

fhh@aaricia ~ $ echo "\"ß\"" | iconv -f utf8 -t ascii//TRANSLIT "ss"
fhh@aaricia ~ $ echo "āáǎàēéěèīíǐìōóǒòūúǔùǖǘǚǜĀÁǍÀĒÉĚÈĪÍǏÌŌÓǑÒŪÚǓÙǕǗǙǛ" | iconv -f utf8 -t ascii//TRANSLIT aaaaeeeeiiiioooouuuuuuuuAAAAEEEEIIIIOOOOUUUUUUUU

et peut être utilisée sur des fichiers :

fhh@aaricia ~ $ cat myfile.txt Une chaine avec des é, des Ù, des À, des ç et des œ   et même des "ß"
fhh@aaricia ~ $ iconv -f utf8 -t ascii//TRANSLIT < myfile.txt > noaccents.txt
fhh@aaricia ~ $ cat noaccents.txt Une chaine avec des e, des U, des A, des c et des oe   et meme des "ss"
 
Categories: Système Tags: , , , ,

Bash: How do I get the command history in a screen session?

05/07/2017 Comments off

If I start a screen session with screen -dmS name, how would I access the command history of that screen session with a script?

Using the , the last executed command appears, even in screen.

ANSWER:

I use the default bash shell on my system and so might not work with other shells.

this is what I have in my ~/.screenrc file so that each new screen window gets its own command history:

Default Screen Windows With Own Command History

To open a set of default screen windows, each with their own command history file, you could add the following to the ~/.screenrc file:

screen -t "window 0" 0 bash -ic 'HISTFILE=~/.bash_history.${WINDOW} bash'
screen -t "window 1" 1 bash -ic 'HISTFILE=~/.bash_history.${WINDOW} bash'
screen -t "window 2" bash -ic 'HISTFILE=~/.bash_history.${WINDOW} bash'

Ensure New Windows Get Their Own Command History

The default screen settings mean that you create a new window using Ctrl+a c or Ctrl+a Ctrl+c. However, with just the above in your ~/.screenrc file, these will use the default ~/.bash_history file. To fix this, we will overwrite the key bindings for creating new windows. Add this to your ~/.screenrc file:

bind c screen bash -ic 'HISTFILE=~/.bash_history.${WINDOW} bash'
bind ^C screen bash -ic 'HISTFILE=~/.bash_history.${WINDOW} bash'

Now whenever you create a new screen window, it’s actually launching a bash shell, setting the HISTFILE environmental variable to something that includes the current screen window’s number ($WINDOW).

Command history files will be shared between screen sessions with the same window numbers.

Write Commands to $HISTFILE on Execution

As is normal bash behavior, the history is only written to the $HISTFILE file by upon exiting the shell/screen window. However, if you want commands to be written to the history files after the command is executed, and thus available immediately to other screen sessions with the same window number, you could add something like this to your ~/.bashrc file:

export PROMPT_COMMAND="history -a; history -c; history -r; ${PROMPT_COMMAND}"
Categories: Système Tags: , , ,

Ubuntu Check RAM Memory Chip Speed and Specification From Within a Linux System

22/06/2016 Comments off

I want to add more RAM to my server running Ubuntu Linux. How do I find out my current RAM chip information such as its speed, type and manufacturer name within a Linux system without opening the case?

You need to use the dmidecode command which is a tool for dumping a computer’s DMI (some say SMBIOS) table contents in a human-readable format. This table contains a description of the system’s hardware components (such as RAM), as well as other useful pieces of information such as serial numbers and BIOS revision. Thanks to this table, you can retrieve hardware information without having to probe for the actual hardware. Open a command-line terminal (select Applications > Accessories > Terminal), and then type:

$ sudo dmidecode --type memory

OR

# dmidecode --type memory | less

OR

$ sudo dmidecode --type 17

Sample outputs:

# dmidecode 2.10
SMBIOS version fixup (2.51 -> 2.6).
SMBIOS 2.6 present.
Handle 0x0011, DMI type 16, 15 bytes
Physical Memory Array
	Location: System Board Or Motherboard
	Use: System Memory
	Error Correction Type: None
	Maximum Capacity: 4 GB
	Error Information Handle: Not Provided
	Number Of Devices: 4
Handle 0x0012, DMI type 17, 27 bytes
Memory Device
	Array Handle: 0x0011
	Error Information Handle: No Error
	Total Width: 72 bits
	Data Width: 64 bits
	Size: 2048 MB
	Form Factor: DIMM
	Set: 1
	Locator: DIMM#1A
	Bank Locator: Bank 1
	Type: DDR2
	Type Detail: Synchronous
	Speed: 667 MHz
	Manufacturer: Not Specified
	Serial Number: Not Specified
	Asset Tag: Not Specified
	Part Number: Not Specified
Handle 0x0013, DMI type 17, 27 bytes
Memory Device
	Array Handle: 0x0011
	Error Information Handle: No Error
	Total Width: 72 bits
	Data Width: 64 bits
	Size: 2048 MB
	Form Factor: DIMM
	Set: 1
	Locator: DIMM#2A
	Bank Locator: Bank 2
	Type: DDR2
	Type Detail: Synchronous
	Speed: 667 MHz
	Manufacturer: Not Specified
	Serial Number: Not Specified
	Asset Tag: Not Specified
	Part Number: Not Specified
Handle 0x0014, DMI type 17, 27 bytes
Memory Device
	Array Handle: 0x0011
	Error Information Handle: No Error
	Total Width: 72 bits
	Data Width: 64 bits
	Size: 2048 MB
	Form Factor: DIMM
	Set: 1
	Locator: DIMM#1B
	Bank Locator: Bank 1
	Type: DDR2
	Type Detail: Synchronous
	Speed: 667 MHz
	Manufacturer: Not Specified
	Serial Number: Not Specified
	Asset Tag: Not Specified
	Part Number: Not Specified
Handle 0x0015, DMI type 17, 27 bytes
Memory Device
	Array Handle: 0x0011
	Error Information Handle: No Error
	Total Width: 72 bits
	Data Width: 64 bits
	Size: 2048 MB
	Form Factor: DIMM
	Set: 1
	Locator: DIMM#2B
	Bank Locator: Bank 2
	Type: DDR2
	Type Detail: Synchronous
	Speed: 667 MHz
	Manufacturer: Not Specified
	Serial Number: Not Specified
	Asset Tag: Not Specified
	Part Number: Not Specified

10 Amazing and Mysterious Uses of (!) Symbol or Operator in Linux Commands

16/06/2016 Comments off

Linux-logo-without-version-number-banner-sized-348x196The '!' symbol or operator in Linux can be used as Logical Negation operator as well as to fetch commands from history with tweaks or to run previously run command with modification. All the commands below have been checked explicitly in bash Shell. Though I have not checked but a major of these won’t run in other shell. Here we go into the amazing and mysterious uses of '!' symbol or operator in Linux commands.

1. Run a command from history by command number.

You might not be aware of the fact that you can run a command from your history command (already/earlier executed commands). To get started first find the command number by running ‘history‘ command.

$ history

History shell command

Now run a command from history just by the number at which it appears, in the output of history. Say run a command that appears at number 1551 in the output of ‘history‘ command.

$ !1551

History by number

And, it runs the command (top command in the above case), that was listed at number 1551. This way to retrieving already executed command is very helpful specially in case of those commands which are long. You just need to call it using ![Number at which it appears in the output of history command].

Lire la suite…