rss logo

How to use Rsync with examples

rsync logo

Rsync is a powerfull command line tool to copy/syncronize files and folders to local or remote hosts.

It's perfectly suited for copying files and backing up a file server for example. However, it is not suitable for copying files that are in use such as databases or virtual machines because in this case you will end up with corrupted (unusable) file copies.

Install rsync

Generally rsync is not preinstalled so the first thing to do is to install it.

  • Debian/Ubuntu :
user@host:~$ sudo apt install rsync
  • Arch Linux :
[root@host ~]# pacman -S rsync

Examples

Copy a file to a different folder

Let's start simple, we will use rsync to copy a file from our /home/std/ folder to /tmp/.

a linux files tree where we copy /home/std/myfile_source file to /tmp/myfile_dest with rsync
  • Command line :
user@host:~$ rsync -av /home/std/myfile_source /tmp/myfile_dest
  • Options :
    • -a or --archive which include :
      • -r : recurse into directories
      • -l : copy symlinks as symlinks
      • -p : preserve permissions
      • -t : preserve modification times
      • -g : preserve group
      • -o : preserve owner (super-user only)
      • -D (--devices)
    • -v : This option increases the amount of information you are given during the transfer.

Copy complete folder

We can also use rsync to recursively copy a folder. Here we will copy our entire /home/std folder to /tmp/.

a linux files tree where we copy /home/std/ folder to /tmp/folder_dest with rsync
  • Command line :
user@host:~$ rsync -av /home/std/ /tmp/folder_dest
  • Options :
    • -a or --archive which include :
      • -r : recurse into directories
      • -l : copy symlinks as symlinks
      • -p : preserve permissions
      • -t : preserve modification times
      • -g : preserve group
      • -o : preserve owner (super-user only)
      • -D (--devices)
    • -v : This option increases the amount of information you are given during the transfer.

Copy over the network

More interestingly, we can use rsync to copy files or folders over the network. For this to work, rsync must be installed on both parties and an ssh server must be installed on the remote host.

A Linux directory structure of two hosts with an rsync copy between them
  • Command line :
user@host:~$ rsync -av -e 'ssh -p 22' /home/std user@192.168.1.100:/data/
  • Options :
    • -a or --archive which include :
      • -r : recurse into directories
      • -l : copy symlinks as symlinks
      • -p : preserve permissions
      • -t : preserve modification times
      • -g : preserve group
      • -o : preserve owner (super-user only)
      • -D (--devices)
    • -v : This option increases the amount of information you are given during the transfer.
    • -e : Specify the remote shell to use, allows here to specify the ssh port (useful if our remote ssh service is listening on a different port)

Misc

  • Copy files from remote to local host :
user@host:~$ rsync -av -e 'ssh -p 22' user@192.168.1.100:/data/std /home/
  • Rsync with sudo if we need root rights on the destination machine :
user@host:~$ rsync -av -e 'ssh -p 22' --rsync-path='sudo rsync' /home/std user@192.168.1.100:/data/

Backups

As explained earlier rsync is particulary suitable for backing up file servers. We will see how to make backup to usb drive, to a remote host and also how to do incrementals in order to save disk space.

Backup to USB drive

Here we will make a full backup of our GNU/Linux operating system. We will exclude special or useless folders (/dev, /mnt, /proc, /sys and /tmp). The destination folder on the usb disk will be automaticaly named with the current date in back-YYYY-MM-dd format.

A Linux directory structure with an rsync backup to a USB Disk
  • Command line (the source is / and the destination /mnt/usb/) :
root@host:~# rsync -av --delete --delete-excluded --exclude=/tmp --exclude=/proc --exclude=/sys --exclude=/dev --exclude=/mnt / /mnt/usb/back-$(date "+%Y-%m-%d")
  • Options :
    • -a or --archive which include :
      • -r : recurse into directories
      • -l : copy symlinks as symlinks
      • -p : preserve permissions
      • -t : preserve modification times
      • -g : preserve group
      • -o : preserve owner (super-user only)
      • -D (--devices)
    • -v : This option increases the amount of information you are given during the transfer.
    • --exclude=PATTERN : exclude files matching PATTERN.
    • --delete (optional here) : delete extraneous files from dest dirs.
    • --delete-excluded (optional here) : also delete excluded files from dest dirs.

Backup with incrementals

One of the most beautiful things about rsync is that it is able to do incremental backups. To do this, it uses hardlinks. For each file to be backed up, rsync compares to the last backup (the current folder on the USB drive which is a symbolic link to the last backup folder) : if the file is the same, it creates a hardlink and does not consume disk space, otherwise (if the file is new or is different) it creates a new file.

Of course the current symbolic link must be created manually and recreated after each backup in order to make it point to the last backup.

As we can see on the diagram, the first backup will take the most space (10GB here), then the others backups will only take the difference (1GB or 2GB in this example).

One last thing to know, in case we want to recover space on our USB disk we will have to delete the backups from the oldest to the most recent (in order to avoid breaking the hardlinks).

  • The three phases to make incremental rsync backups :
    • PRE-BACKUP : the usb disk is mounted on /mnt/usb/
    • BACKUP : we make the rsync backup, rsync uses current symbolic link which points to the last backup which is here back-2022-08-14.
    • POST-BACKUP : once the rsync backup is finished, we have to update the current link so that it points to the new last backup which is now back-2022-08-15.
Three phases to make rsync incremental backup to a usb drive

Backup to USB drive with incrementals

It is the detailed application of what has been seen just above.

A Linux directory structure with an rsync incremental backup to a USB Disk
  • Options :
    • -a or --archive which include :
      • -r : recurse into directories
      • -l : copy symlinks as symlinks
      • -p : preserve permissions
      • -t : preserve modification times
      • -g : preserve group
      • -o : preserve owner (super-user only)
      • -D (--devices)
    • -v : This option increases the amount of information you are given during the transfer.
    • --exclude=PATTERN : exclude files matching PATTERN.
    • --link-dest=/mnt/usb/current : hardlink to files in current when unchanged.
    • --delete : delete extraneous files from dest dirs.
    • --delete-excluded : also delete excluded files from dest dirs.
  • Command line to backup (the source is / and the destination is /mnt/usb/) :
user@host:~$ rsync -av --delete --delete-excluded --exclude=/tmp --exclude=/proc --exclude=/sys --exclude=/dev --exclude=/mnt --exclude=/media --link-dest=/mnt/USB/current / /mnt/usb/back-$(date "+%Y-%m-%d")
  • Update current symbolic link :
    • Remove symbolic link named current :
user@host:~$ rm -f /mnt/usb/current
  • Recreate current symbolic link :
user@host:~$ ln -s /mnt/usb/back-$(date "+%Y-%m-%d") /mnt/usb/current
  • Check the space used by incremental backups :
user@host:~$ du -sh /mnt/usb/back-* 10G back-2022-08-11 1G back-2022-08-12 2G back-2022-08-13 1G back-2022-08-14

Backup to Network Server with incrementals

Same as above except that this time the destination is a GNU/Linux server so the backup will be done over the network.

A Linux directory structure with an rsync incremental backup to a Network Linux server
  • Options :
    • -a or --archive which include :
      • -r : recurse into directories
      • -l : copy symlinks as symlinks
      • -p : preserve permissions
      • -t : preserve modification times
      • -g : preserve group
      • -o : preserve owner (super-user only)
      • -D (--devices)
    • -v : This option increases the amount of information you are given during the transfer.
    • --exclude=PATTERN : exclude files matching PATTERN.
    • --link-dest=/mnt/usb/current : hardlink to files in current when unchanged.
    • --delete : delete extraneous files from dest dirs.
    • --delete-excluded : also delete excluded files from dest dirs.
  • Command line to backup (the source is / and the destination is /backup/ on the 192.168.1.100 server) :
user@host:~$ rsync -av --delete --delete-excluded --exclude=/tmp --exclude=/proc --exclude=/sys --exclude=/dev --exclude=/mnt --exclude=/media --link-dest=/backup/current / user@192.168.1.100:/backup/back-$(date "+%Y-%m-%d")
  • Update current symbolic link :
    • Remove symbolic link named current :
user@host:~$ ssh -p 22 user@192.168.1.100 "rm /backup/current"
  • Recreate current symbolic link :
user@host:~$ ssh -p 22 user@192.168.1.100 "ln -s /backup/back-$(date "+%Y-%m-%d") /backup/current"

Misc

I put here everything I found useful.

  • Show progress during transfer :
user@host:~$ rsync --progress -av /home/std/folder_source/ remoteuser@192.168.1.100:/tmp/folder_dest
  • Print stats at the end of the transfer :
user@host:~$ rsync --stats -av /home/std/folder_source/ remoteuser@192.168.1.100:/tmp/folder_dest
  • Limit the bandwidth used to a defined threshold :
user@host:~$ rsync --progress -av --bwlimit=100k /home/std/folder_source/ remoteuser@192.168.1.100:/tmp/folder_dest
  • Transfer only files with a maximum modification date of 3 days max to /tmp/ directory :

Note : needs local transfer or ssh passwordless to work

user@host:~$ rsync --progress -av remoteuser@192.168.1.100:/ --no-relative --files-from=<(ssh remoteuser@192.168.1.100 find /data/* -mtime -3 -type f) /tmp/
  • Mirror copy (--delete : delete extraneous files from dest dirs) :
user@host:~$ rsync --progress /home/std/ /tmp/folder_dest
Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

Contact :

contact mail address