rss logo

How to transfer a large file over the Internet by splitting it into smaller parts with GNU/Linux

Tux logo

From time to time, I need to transfer large files between two GNU/Linux hosts over a relatively slow connection. To do this, I split the large files into several smaller ones, which I reassemble once all the smaller files have been transferred. The advantage is manifold: if the copy fails during transfer, I don't have to transfer the whole thing again, and what's more, I can make the copy in several stages. Let's see how to do this in detail below!

Network diagram

In this example, I'm using a private IP address (192.168.1.200), but it obviously works with a public IP.

As indicated in the introduction, the operation consists in dividing the large file into several smaller ones.

GNU/Linux transfer large file via ssh and rsync diagram

Installing prerequisites

  • To transfer our files, we will need to install rsync and openssh on both machines:
root@host:~# apt-get update && apt-get install rsync openssh-server

Split file

  • The du command shows the total size to be transferred:
user@SOURCE:~$ du -sh MyBigFile.mkv
-rw-r--r-- 1 user std 2,9G  8 déc.  12:12 MyBigFile.mkv

To split the large file into several smaller files, we'll use the cleverly named: split command.

  • The split command options we'll be using:
    • -d: use numeric suffixes starting at 0
    • -b: files size, here we'll divide our large file into several smaller files of 5M bytes.
user@SOURCE:~$ split MyBigFile.mkv -d -b 5M
user@SOURCE:~$ ls -lh
-rw-r--r-- 1 user std 5,0M  8 déc.  23:47 x00
-rw-r--r-- 1 user std 5,0M  8 déc.  23:47 x01
[...]
-rw-r--r-- 1 user std 5,0M  8 déc.  23:48 x9481
-rw-r--r-- 1 user std 5,0M  8 déc.  23:48 x9482

Small file transfer

We can now transfer the files to our destination host. In the event of failure, the rsync software will be able to resume the transfer from the last file transferred, thus avoiding the need to start all over again.

  • rsync options:
    • -a: archive mode
    • --bwlimit=400k: (Optional) We can define the maximum bandwidth. Here, 400kB/s.
    • --rsh='ssh -p 22': Specify the 192.168.1.200 host ssh port.
    • --stats: (Optional) At the end, give some statistics on file transfer.
    • --progress: (Optional) Show progress during transfer.
    • user@192.168.1.200:/home/SOURCE/x*: Source folder, where our split files are located. Here in the /home/SOURCE/ directory.
    • /home/DESTINATION/: Destination folder.
user@DESTINATION:~$ rsync --bwlimit=400k -a -v --rsh='ssh -p 22' --stats --progress user@192.168.1.200:/home/SOURCE/x* /home/DESTINATION/

Joining split files

  • Once the transfer is complete, it's time to reunite the separate files into a single:
user@DESTINATION:~$ cd /home/DESTINATION/
user@DESTINATION:~$ cat x* > glory.41.720p.hdtv.x264-verum.mkv
user@DESTINATION:~$ ls -lh glory.41.720p.hdtv.x264-verum.mkv
-rw-r--r-- 1 user std 2,9G  8 déc.  23:45 glory.41.720p.hdtv.x264-verum.mkv