How to transfer a huge file through internet thanks to GNU/Linux

Network diagram

For the example I use a private ip address (192.168.1.200) but it obviously works with a public ip.

The operation will consist in dividing the big file into several small ones. So in case of a network failure we will be able to resume the transfer where it has stopped.

Installing prerequisites

• In order to transfer our files we will need, on these two machines, to install rsync and openssh :

root@host:~# apt-get install rsync openssh-server

Split the file

• As said before, we will divide the large file into several small ones thanks to the split tool :

• split options :
  • -d : use numeric suffixes starting at 0
  • -b : files size, here we will divides our big file into 5M bytes files.

user@SOURCE:~$ split MyBigFile.mkv -d -b 5M

user@SOURCE:~$ ls -lh
-rw-r--r-- 1 std std 5,0M  8 déc.  23:47 x00
-rw-r--r-- 1 std std 5,0M  8 déc.  23:47 x01
[...]
-rw-r--r-- 1 std std 5,0M  8 déc.  23:48 x9481
-rw-r--r-- 1 std std 5,0M  8 déc.  23:48 x9482

Files transfer

Now we can transfer the files to our destination host. In case of failure the rsync software will be able to resume the transfer from the last file transferred and thus avoid starting from the beginning.

• rsync options :
  • -a : archive mode
  • --bwlimit=40k : (Optional) We can set maximum bandwidth. Here, 40kB/s.
  • --rsh='ssh -p 22' : Specify the 192.168.1.200 ssh port.
  • --stats : (Optional) In the end, give some file-transfer stats.
  • --progress : (Optional) show progress during transfer.
  • user@192.168.1.200:/home/SOURCE/x* : Source folder, where our split files are. Here in /home/SOURCE/ directory.
  • /home/DESTINATION/ : Destination folder.

user@DESTINATION:~$ rsync --bwlimit=40k -a -v --rsh='ssh -p 22' --stats --progress user@192.168.1.200:/home/SOURCE/x* /home/DESTINATION/

Rejoining split files

• Once transfer complete, it's time to rejoin split files :

user@DESTINATION:~$ cd /home/DESTINATION/

user@DESTINATION:~$ cat x* > glory.41.720p.hdtv.x264-verum.mkv

user@DESTINATION:~$ ls -lh glory.41.720p.hdtv.x264-verum.mkv
-rw-r--r-- 1 toi toi 2,9G  8 déc.  23:45 glory.41.720p.hdtv.x264-verum.mkv