Find Duplicate Files (based on size first, then MD5 hash)

14/05/2021 Categories: Système Tags: , Aucun commentaire
Terminal – Find Duplicate Files (based on size first, then MD5 hash)
find -not -empty -type f -printf "%s\n" | sort -rn | uniq -d | xargs -I{} -n1 find -type f -size {}c -print0 | xargs -0 md5sum | sort | uniq -w32 --all-repeated=separate
Find Duplicate Files (based on size first, then MD5 hash)

This dup finder saves time by comparing size first, then md5sum, it doesn’t delete anything, just lists them.

Terminal – Alternatives
find -type f -exec md5sum '{}' ';' | sort | uniq --all-repeated=separate -w 33 | cut -c 35-

Calculates md5 sum of files. sort (required for uniq to work). uniq based on only the hash. use cut ro remove the hash from the result.

find -type d -name ".svn" -prune -o -not -empty -type f -printf "%s\n" | sort -rn | uniq -d | xargs -I{} -n1 find -type d -name ".svn" -prune -o -type f -size {}c -print0 | xargs -0 md5sum | sort | uniq -w32 --all-repeated=separate

Improvement of the command “Find Duplicate Files (based on size first, then MD5 hash)” when searching for duplicate files in a directory containing a subversion working copy. This way the (multiple dupicates) in the meta-information directories are ignored.

Can easily be adopted for other VCS as well. For CVS i.e. change “.svn” into “.csv”:

find -type d -name ".csv" -prune -o -not -empty -type f -printf "%s\n" | sort -rn | uniq -d | xargs -I{} -n1 find -type d -name ".csv" -prune -o -type f -size {}c -print0 | xargs -0 md5sum | sort | uniq -w32 --all-repeated=separate

find -not -empty -type f -printf "%-30s'\t\"%h/%f\"\n" | sort -rn -t$'\t' | uniq -w30 -D | cut -f 2 -d $'\t' | xargs md5sum | sort | uniq -w32 --all-repeated=separate

Finds duplicates based on MD5 sum. Compares only files with the same size. Performance improvements on:

find -not -empty -type f -printf "%s\n" | sort -rn | uniq -d | xargs -I{} -n1 find -type f -size {}c -print0 | xargs -0 md5sum | sort | uniq -w32 --all-repeated=separateThe new version takes around 3 seconds where the old version took around 17 minutes. The bottle neck in the old command was the second find. It searches for the files with the specified file size. The new version keeps the file path and size from the beginning.

find . -type f -exec md5 '{}' ';' | sort | uniq -f 3 -d | sed -e "s/.*(\(.*\)).*/\1/"

This works on Mac OS X using the `md5` command instead of `md5sum`, which works similarly, but has a different output format. Note that this only prints the name of the duplicates, not the original file. This is handy because you can add `| xargs rm` to the end of the command to delete all the duplicates while leaving the original.

Categories: Système Tags: ,

Split OpenVPN configuration files

14/05/2021 Categories: Réseau, Système Tags: , , Aucun commentaire

Source: npm

Splits OpenVPN (.ovpn) files into separate files for private key, user+ca certificates and tls-auth key, for use with network-manager in debian/ubuntu.

openvpn-config-splitter can be installed using npm:

# NPM:
npm install -g openvpn-config-splitter
# Install globally
$ npm install -g openvpn-config-splitter
# Run it, specifying your unsplit OpenVPN configuration file
$ ovpnsplit path/to/some/config.ovpn
# Config is now split into separate files, new configuration
# linking to the split files has been generated
$ ls path/to/some
ca.crt  client.key  client.ovpn  client.split.ovpn  ta.key  user.crt
var fs = require('fs'),
 configPath = '/some/path/to',
 splitter = require('openvpn-config-splitter');
var paths = {
 'caCert': configPath + '/openvpn-ca.crt',
 'userCert': configPath + '/openvpn-user.crt',
 'privateKey': configPath + '/openvpn-private.key',
 'tlsAuth': configPath + '/openvpn-tls.key'
fs.readFile(configPath + '/config.ovpn', function(err, originalConfig) {
 if (err) {
 console.error('Could not read file (' + err.path + ')');
 splitter.split(originalConfig, paths, function(err, parts, missing) {
 if (err) {
 * `parts` now contain the matched parts of the config + new config
 * (caCert, userCert, privateKey, tlsAuth, config)
 * `missing` is an array containing the parts that were NOT found -
 * use this if you want to warn the user or fall back if you require
 * a specific part to be present
 // Want to write the split files? 
 splitter.writeToFiles(parts, paths, function(err) {
 if (err) {
 console.log('Hooray, we split the files and wrote them to disk!');
Categories: Réseau, Système Tags: , ,

Typical iptables

# Modify this file accordingly for your specific requirement.
# 1. Delete all existing rules
iptables -F

# 2. Set default chain policies
iptables -P INPUT DROP
iptables -P FORWARD DROP
iptables -P OUTPUT DROP

# 3. Block a specific ip-address
#iptables -A INPUT -s "$BLOCK_THIS_IP" -j DROP
 Lire la suite...

(D)DoS Deflate

13/05/2021 Categories: Réseau, Système Tags: , Aucun commentaire


(D)DoS Deflate is a lightweight bash shell script designed to assist in the process of blocking a denial of service attack. It utilizes the command below to create a list of IP addresses connected to the server, along with their total number of connections. It is one of the simplest and easiest to install solutions at the software level.

netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n

IP addresses with over a pre-configured number of connections are automatically blocked in the server’s firewall, which can be direct iptables or Advanced Policy Firewall (APF). (We highly recommend that you use APF on your server in general, but deflate will work without it.)

Notable Features

  • It is possible to whitelist IP addresses, via /usr/local/ddos/ignore.ip.list.
  • Simple configuration file: /usr/local/ddos/ddos.conf
  • IP addresses are automatically unblocked after a preconfigured time limit (default: 600 seconds)
  • The script can run at a chosen frequency via the configuration file (default: 1 minute)
  • You can receive email alerts when IP addresses are blocked.


chmod 0700


chmod 0700 uninstall.ddos


Although most things are explained on this page, if you have any further questions, you may contact the original developer of the script, Zaf.

Categories: Réseau, Système Tags: ,

How To Use Port Knocking to Hide your SSH Daemon from Attackers on Ubuntu


Servers, by definition, are implemented as a means of providing services and making applications and resources accessible to users. However, any computer connected to the internet is inevitably targeted by malicious users and scripts hoping to take advantage of security vulnerabilities.

Firewalls exist and should be used to block access on ports not being utilized by a service, but there is still the question of what to do about services that you want access to, but do not want to expose to everybody. You want access when you need it, but want it blocked off otherwise.

Port knocking is one method of obscuring the services that you have running on your machine. It allows your firewall to protect your services until you ask for a port to be opened through a specific sequence of network traffic.

In this guide, we will discuss how to implement port knocking as a method of obscuring your SSH daemon on an Ubuntu 12.04 VPS using the knockd package.

Note: This tutorial covers IPv4 security. In Linux, IPv6 security is maintained separately from IPv4. For example, “iptables” only maintains firewall rules for IPv4 addresses but it has an IPv6 counterpart called “ip6tables”, which can be used to maintain firewall rules for IPv6 network addresses.

If your VPS is configured for IPv6, please remember to secure both your IPv4 and IPv6 network interfaces with the appropriate tools. For more information about IPv6 tools, refer to this guide: How To Configure Tools to Use IPv6 on a Linux VPS

Lire la suite…