Archive

Articles taggués ‘sauvegarde’

How to enable incremental file sync for many users on Linux

02/12/2023 Comments off

Suppose you as a software developer has set up daily builds of your software for testing purposes. Every day you make a new build, users have to re-download the updated build to evaluate it. In this case you may want to enable differential downloads, so that users can download only difference between two builds, thereby saving on the server’s bandwidth. Users will also be happy as they don’t have to wait to re-download the whole thing. Similar situations are encountered when you want to set up a download archive which allows incremental sync for users.

In these cases, how would you distribute incrementally updated files efficiently for multiple users? In fact, there are open-source storage solutions that come with “delta sync” capability built-in, such as ownCloud or Syncthing. These kinds of full-blown solutions with built-in GUI require users to install a dedicated client, and thus may be an overkill for simple file distribution that you are looking for.

Barring full-blown third-party software like these, perhaps rsync may come to mind, which can do bandwidth-efficient file sync. The problem with rsync, however, is that all the heavy duty computations is done at the server side. That is, when a client requests for a file sync, the server needs to perform block-by-block checksum computation and search for blocks not available on the client. Obviously this procedure can place a significant strain on the server’s resources if the server needs to handle many users, and thus is not scalable.

What is Zsync?

This is when a command-line tool called zsync comes in handy. While zsync uses the same delta-encoding based sync algorithm as rsync does, it moves the heavy duty computation away from the server and onto the clients. What do I mean by that?

Well, in zsync, the server maintains a separate .zsync metadata file for a file to distribute, which contains a list of “precomputed” checksums for individual blocks of the file. When zsync client requests for a file sync, the client downloads .zsync metadata file first, and performs block-by-block checksum calculation to find missing blocks on its own. The client then requests for missing blocks using HTTP range requests. As you can see, the server is totally out of the loop from the sync algorithm, and simply serves requested file blocks over HTTP, which makes it ideal when incremental file sync is needed for many users.

Here is a quick rundown on the server-side overhead difference between rsync and zsync. In the plot below, I compare rsync and zsync in terms of the server’s CPU usage when 200 users are downloading a tarball file with 2.5% discrepancy of a previous version. For fair comparison, SSH is not used for rsync.

With zsync, since all checksum computation overhead has shifted from the server to individual clients, the server overhead is reduced dramatically. A small neat idea makes zsync a real winner!

In the rest of the tutorial, I will show how to distribute a file incrementally using zsync under the Linux environment.

Zsync: Client Side Setup

On the client side, you need to install zsync to initiate file transfer from a remote web server. zsync is extremely lightweight, and is included in the most Linux distros. Here is how to install zsync on various distros.

Debian, Ubuntu or Linux Mint:

$ sudo apt-get install zsync

Fedora:

$ sudo yum install zsync

CentOS or RHEL:

First, enable Repoforge repository, and then run:

$ sudo yum install zsync

Arch Linux:

$ sudo pacman -S zsync

Lire la suite…

Categories: Logiciel Tags: , ,

Five reliable Windows Server backup solutions

30/11/2023 Comments off

When disaster strikes (and it will), you can put the pieces back together if you’ve got a solid backup solution in place.

If you have a Windows server and you’re not backing it up, you are tempting fate. It’s only a matter of time before something tragic will happen: a hardware failure, a hacker, or a corrupted Windows update. Something will bring that server down and do everything it can to take your data with it.

Luckily, several reliable backup solutions are available for the Windows Server environment. They aren’t perfect. But the five solutions discussed here have their strengths and can be depended upon to keep your data safe. Now, before I reveal the big five, I want to make sure you understand how critical it is to be backing up that data to an external drive and to rotate that drive offsite. If you’re backing up drives and leaving everything onsite, should a fire (or some other catastrophe) occur, that data will be lost.

Note: This list is also available as a photo gallery.

1: Acronis Backup and Restore (with Universal Restore)

Acronis Backup and Restore (Figure A) is the backup tool I use the most. It’s costly but well worth it. The newer version of Acronis does a great job of ensuring that there is room on your external drive for your latest backup. You can overwrite the old backup or go by the traditional cleanup rules established in Acronis Echo. But if you go the Acronis route, you should also purchase the Universal Restore feature. With this tool, you can take a full image of a backup and restore it to any hardware — so long as it meets the requirements of your platform. That way, you don’t need an exact duplicate of your server hardware sitting around waiting to have an image restored onto it. Acronis also has add-ons for Exchange and SQL Servers.

Figure A

 

Acronis backup
Acronis Backup and Restore

2: Symantec Backup Exec

Symantec Backup Exec (Figure B) is a powerful backup solution that does a great job of “just working.” And with its at-a-glance calendar, it’s easy to know when your backup has succeeded and when it has worked. If you’re using Symantec Backup Exec across a network (to back up clients/hosts), you can feel safe because it uses SSL for all network backup connections. Like Acronis, Symantec offers solutions for Exchange and SQL. Also, the ability to manage Vaults (backup destinations) is superb with Symantec. Just make sure that if a Vault is physically (and permanently) removed, you remove it from within the backup software. Otherwise, it will cause problems.

Figure B

Symantec Backup Exec

Symantec Backup Exec

3: Carbonite

Carbonite (Figure C) is primarily an online backup (although the Business and BusinessPremier plans allow you to back up to an external drive). It’s meant to back up data only. Carbonite is probably one of the most reliable online backup tools and can back up email, documents, POS files, financial data, pictures and other multimedia, and more. But to use Carbonite for Windows Server, you must use the BusinessPremier Plan ($229.00 per year), which will allow you to include as many servers running Window Server 2003/2008/2011 as you like. Unlike Acronis, you won’t be backing up an image of your server with Carbonite — it’s a data-only backup tool. But it’s an excellent addition to any onsite backup solution.

Figure C

Carbonite

Carbonite

4: Windows NT Backup

Windows NT Backup (Figure D) is the free backup solution that comes with any Windows Server. Why do I include it here? Although it just does data backup and doesn’t offer many of the bells and whistles of the other tools, it is one of the only truly reliable backup solutions that will clear the Exchange logs for you. I always set up an NT Backup to run once a week (or month, depending upon the needs), just to have it clear the Exchange logs. Otherwise, you’ll be including a lot of unnecessary data in your backups, causing their size to get out of hand.

Figure D

Windows NT Backup

Windows NT Backup

5: ShadowProtect 4 (Server)

ShadowProtect 4 (Figure E) is probably the least-known product in this list. But it falls into place next to Acronis as a feature-rich tool that can do everything — including recover to dissimilar hardware. ShadowProtect 4 also includes an easy-to-use tool that allows you to consolidate backup image files and convert the backups to either .VHD or .VMDK virtual machines. Another outstanding feature is its granular recovery, which allows for the quick recovery of individual files from within a backup. Unique to ShadowProtect is the Virtual Boot technology, which provides for automatic failover to a virtual server.

Figure E

ShadowProtect

ShadowProtect 4

Which backup?

Every company needs a backup plan. How far you go with your backup depends upon the size of the company, your needs, and your budget. But any of the above solutions will serve as a great place to start — or stop — for your Windows Server backup needs. Do you have a different backup solution to add to the list? Share your recommendations with fellow TechRepublic members.

 
Source: Jack Wallen – TechRepublic
Categories: Logiciel Tags: , ,

Simple versioned TimeMachine-like backup using rsync

18/10/2023 Comments off

time machine rsyncUsing TimeMachine with rsync: Over many years, I have dealt with scripts that do backup versioning, i.e., maintain multiple backups. Due to their flexibility, they have been complex to understand and configure. Here is a simple rsync-based tool with a different focus: The experienced systems administrator who wants to keep his system’s complexity down.

Backup in action: TimeMachine and rsync

It consists of a simple script, which you can call rsync-backup.sh and store wherever you like, e.g., in /usr/local/sbin. I will use these names and paths in the examples.

#!/bin/sh
# Usage: rsync-backup.sh <src> <dst> <label>
if [ "$#" -ne 3 ]; then
    echo "$0: Expected 3 arguments, received $#: $@" >&2
    exit 1
fi
if [ -d "$2/__prev/" ]; then
    rsync -a --delete --link-dest="$2/__prev/" "$1" "$2/$3"
else
    rsync -a                                   "$1" "$2/$3"
fi
rm -f "$2/__prev"
ln -s "$3" "$2/__prev"

During normal operation, it boils down to three simple statements:

  1. rsync with --link-dest: Copying the contents of <src> to <dst>/<label>, reusing the files from the previous backup with hard links ((The non---link-dest rsync does not use --delete to reduce the risk of accidentally deleting files when called with wrong parameters))
  2. rm and ln: Remember this backup location for the next incremental backup.

Voilà – it doesn’t get much easier than that!

Of course, there is something missing: The actual backup policy. It is separated into cron, which I consider an advantage. Using this separation of duties, many policies can be implemented very easily and composed in a modular way:

Create daily backups  for every weekday

You might know this from automysqlbackup or autopostgresqlbackup: A backup is created every day and overwritten after 7 days. This is achieved by adding the following file to /etc/cron.daily/:

#!/bin/sh
/usr/local/bin/rsync-backup.sh /home /data/backup `date +%A`

All your user’s files are copied daily to /data/backup, named after the current day, overwritten weekly.

Daily backups for a month

Sure, this is easy as well, by putting this with a descriptive name into/etc/cron.daily/:

#!/bin/sh
/usr/local/bin/rsync-backup.sh /home /data/backup `date +Day-%d`

Lire la suite…

Categories: Système Tags: , , , ,

HowTo: Make Ubuntu a Perfect Mac File Server and Time Machine Volume

30/09/2023 Comments off

ubuntu time machineFor quite some time I use my Ubuntu machine as a file and backup server for all Macs in my network which is perfectly accessible from the Finder in Mac OS X. There are some instructions available in the web for this task but all failed in my case so I wrote my own tutorial with all the steps needed for it to work properly.

So here’s my little Tutorial for connecting Mac OS X Leopard with Ubuntu and using your Ubuntu machine as a backup volume for Time Machine but all steps can be reproduced on every Linux box and they work with Mac OS X 10.4 Tiger too. At the end of this tutorial you will have a server which shows up in the Finder sidebar and behaves just like a Mac server when accessing it from your Macs. To be perfectly integrated with Mac OS X we’re going to use Apple’s Filing Protocol (AFP) for network and file sharing.

Although this Tutorial involves using the Terminal in Ubuntu and looks a bit geeky it’s very easy even for beginners. I have tried to explain all steps and Terminal commands so you may learn a bit about the Terminal too. At the end of the article you can download my Server Displays icon pack quickly made by me with custom icons for a Mac, Ubuntu and Windows server.

Personally I use a fresh installation of Ubuntu 8.04 Hardy Heron Desktop version (32bit on one machine, 64bit on the other) and Mac OS X Leopard (10.5.3 and later) to connect to them. On my Ubuntu boxes there’s no other file sharing protocol like samba (Windows sharing) or NFS activated.

Rumors are Apple will add some undocumented AFP commands with the Mac OS X 10.5.6 update which therefor won’t be supported by the current Netatalk package (and maybe never will). So be sure to check the latest comments on this article when the 10.5.6 update is out to see if this rumor is true and if there are problems caused by that.

Here are the steps involved in setting up your Ubuntu box as a Mac file server:

  1. Modify and install Netatalk (Open Source AFP implementation)
  2. Configure Netatalk
  3. Configure shared volumes (and Time Machine volume)
  4. Install Avahi (Open Source Bonjour implementation)
  5. Configure Avahi and advertise services
  6. Configure TimeMachine
  7. Conclusion, Problems and more informations
  8. Downloading and using the Server Display Icons
  9. Translations Of This Article

Lire la suite…

Categories: Système Tags: , , ,

Websync, web interface to manage your rsync tasks

14/08/2023 Comments off

Source: freedif.org

tasks_tabRsync is a great tool to replicate, sync some data on your computer. And I’m heavily relying on it to backup my server and to mirror some opensource projects and GNU/Linux Distributions.

But I’ve recently found a Web interface to manage all my rsync tasks called websync.

Websync is a web based rsync task manager where you can add, edit, clone, remove, scheduled,…. your rsync tasks while being able to have a remote host as source or destination of the task (With SSH RSA key too)

Under the free license MIT, Websync has been developped by Sander Struijk and is still actively being maintained, as you can see on github forum. But it is still an early project, so if you face any issue, make sure to report them on the issue tracker.

Interested to give it a shot, here is how to install Websync!

Lire la suite…

Categories: Logiciel, Système Tags: , , ,