Articles taggués ‘commands’

Find Duplicate Files (based on size first, then MD5 hash)

14/05/2021 Aucun commentaire
Terminal – Find Duplicate Files (based on size first, then MD5 hash)
find -not -empty -type f -printf "%s\n" | sort -rn | uniq -d | xargs -I{} -n1 find -type f -size {}c -print0 | xargs -0 md5sum | sort | uniq -w32 --all-repeated=separate
Find Duplicate Files (based on size first, then MD5 hash)

This dup finder saves time by comparing size first, then md5sum, it doesn’t delete anything, just lists them.

Terminal – Alternatives
find -type f -exec md5sum '{}' ';' | sort | uniq --all-repeated=separate -w 33 | cut -c 35-

Calculates md5 sum of files. sort (required for uniq to work). uniq based on only the hash. use cut ro remove the hash from the result.

find -type d -name ".svn" -prune -o -not -empty -type f -printf "%s\n" | sort -rn | uniq -d | xargs -I{} -n1 find -type d -name ".svn" -prune -o -type f -size {}c -print0 | xargs -0 md5sum | sort | uniq -w32 --all-repeated=separate

Improvement of the command “Find Duplicate Files (based on size first, then MD5 hash)” when searching for duplicate files in a directory containing a subversion working copy. This way the (multiple dupicates) in the meta-information directories are ignored.

Can easily be adopted for other VCS as well. For CVS i.e. change “.svn” into “.csv”:

find -type d -name ".csv" -prune -o -not -empty -type f -printf "%s\n" | sort -rn | uniq -d | xargs -I{} -n1 find -type d -name ".csv" -prune -o -type f -size {}c -print0 | xargs -0 md5sum | sort | uniq -w32 --all-repeated=separate

find -not -empty -type f -printf "%-30s'\t\"%h/%f\"\n" | sort -rn -t$'\t' | uniq -w30 -D | cut -f 2 -d $'\t' | xargs md5sum | sort | uniq -w32 --all-repeated=separate

Finds duplicates based on MD5 sum. Compares only files with the same size. Performance improvements on:

find -not -empty -type f -printf "%s\n" | sort -rn | uniq -d | xargs -I{} -n1 find -type f -size {}c -print0 | xargs -0 md5sum | sort | uniq -w32 --all-repeated=separateThe new version takes around 3 seconds where the old version took around 17 minutes. The bottle neck in the old command was the second find. It searches for the files with the specified file size. The new version keeps the file path and size from the beginning.

find . -type f -exec md5 '{}' ';' | sort | uniq -f 3 -d | sed -e "s/.*(\(.*\)).*/\1/"

This works on Mac OS X using the `md5` command instead of `md5sum`, which works similarly, but has a different output format. Note that this only prints the name of the duplicates, not the original file. This is handy because you can add `| xargs rm` to the end of the command to delete all the duplicates while leaving the original.

Categories: Système Tags: ,

Typical iptables

13/05/2021 Aucun commentaire
# Modify this file accordingly for your specific requirement.
# 1. Delete all existing rules
iptables -F

# 2. Set default chain policies
iptables -P INPUT DROP
iptables -P FORWARD DROP
iptables -P OUTPUT DROP

# 3. Block a specific ip-address
#iptables -A INPUT -s "$BLOCK_THIS_IP" -j DROP
 Lire la suite...

(D)DoS Deflate

13/05/2021 Aucun commentaire


(D)DoS Deflate is a lightweight bash shell script designed to assist in the process of blocking a denial of service attack. It utilizes the command below to create a list of IP addresses connected to the server, along with their total number of connections. It is one of the simplest and easiest to install solutions at the software level.

netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n

IP addresses with over a pre-configured number of connections are automatically blocked in the server’s firewall, which can be direct iptables or Advanced Policy Firewall (APF). (We highly recommend that you use APF on your server in general, but deflate will work without it.)

Notable Features

  • It is possible to whitelist IP addresses, via /usr/local/ddos/ignore.ip.list.
  • Simple configuration file: /usr/local/ddos/ddos.conf
  • IP addresses are automatically unblocked after a preconfigured time limit (default: 600 seconds)
  • The script can run at a chosen frequency via the configuration file (default: 1 minute)
  • You can receive email alerts when IP addresses are blocked.


chmod 0700


chmod 0700 uninstall.ddos


Although most things are explained on this page, if you have any further questions, you may contact the original developer of the script, Zaf.

Categories: Réseau, Système Tags: ,

cssh: un outil pour les paresseux…

08/05/2021 Aucun commentaire

Cluster ssh

Cluster ssh

Cluster ssh est un outil merveilleux ! Combien de fois ai-je du retaper les mêmes commandes dans deux fenêtres shell différentes sous prétexte que les serveurs faisaient la même chose… Je pense à mes deux serveurs DNS, par exemple.

Évidemment, il faut faire attention à ce qu’on tape car la moindre erreur est reproduite en double… Mais quel bonheur de faire deux fois le travail et surtout d’avoir des configurations identiques quand on le désire.

Autre avantage: si on ouvre plusieurs sessions ssh dans la même commande shell, cssh redimensionne les fenêtres automatiquement.

Attention: si vous ajouter des fenêtres après coup sur Ubuntu, il les superposera sur les précédentes ce qui n’est pas folichon… Il faut prévoir votre nombre de session au départ. Par contre, sur Mac OS X, l’excellent csshX fait le boulot correctement.

Categories: Système Tags: ,

15 Linux Find Command Examples (Part2)

07/05/2021 Aucun commentaire

Find Files Based on Access / Modification / Change Time

You can find files based on following three file time attribute.

Access time of the file. Access time gets updated when the file accessed.
Modification time of the file. Modification time gets updated when the file content modified.
Change time of the file. Change time gets updated when the inode data changes. Lire la suite…

Categories: Système Tags: ,