Data compression benchmark under GNU/Linux

Intro

I was wondering about which was the best files compression and decompression software under GNU/Linux so I made a real-world test to see the differences in performance.

Test Environment

• I created six folders with the following characteristics:
  • AUDIO: 101MB, seven mp3 and seven ogg files
  • IMG: 102MB, 17 jpg files
  • PDF: 102MB, 68 pdf files
  • RANDOM TXT: 95MB, one big txt file, made with base64 /dev/urandom | head -c 100000000 > file.txt command.
  • MAN TXT: 7.2MB, 870 man files
  • VIDEOS: 290MB, two mp4, one mpeg and one webm file

Tools

7zip

• Compression: yes
• Archiving: yes
• Initial release: 1999
• Misc: doesn't save owner/group of files

Compress

• Add file/directory to the archive :

user@SOURCE:~$ 7z a DEST.7z SOURCE

Decompress

• Extract with full path :

user@SOURCE:~$ 7z x DEST.7z

bzip2

• Compression: yes
• Archiving: no
• Initial release: 1996
• Misc:

Compress

• Compress FILE to FILE.bz2 and remove FILE (-k to keep):

user@SOURCE:~$ bzip2 -z FILE

• Same but will keep FILE:

user@SOURCE:~$ bzip2 -c FILE > FILE.bz2

Decompress

• Decompress FILE.bz2 to FILE and remove FILE.bz2:

user@SOURCE:~$ bzip2 -d FILE

LZ4

• Compression: yes
• Archiving: no
• Initial release: 2011
• Misc:

Compress File

• Compress FILE to FILE.lz4:

user@SOURCE:~$ lz4 -z FILE FILE.lz4

Decompress

• Decompress FILE.lz4 to FILE:

user@SOURCE:~$ lz4 -d FILE.lz4

gzip

• Compression: yes
• Archiving: no
• Initial release: 1992
• Misc:

Compress

• Compress FILE to FILE.gz and remove FILE (-k to keep):

user@SOURCE:~$ gzip FILE

• Same but will keep FILE:

user@SOURCE:~$ gzip -c FILE > FILE.gz

Decompress

• Decompress FILE.gz to FILE:

user@SOURCE:~$ gzip -d FILE.gz

xz

• Compression: yes
• Archiving: no
• Initial release:
• Misc:

Compress

• Compress FILE to FILE.xz and remove FILE (-k to keep):

user@SOURCE:~$ xz -z FILE

• Same but will keep FILE:

user@SOURCE:~$ xz -c FILE > FILE.gz

Decompress

• Decompress FILE.xz to FILE:

user@SOURCE:~$ xz -d FILE.xz

zip

• Compression: yes
• Archiving: yes
• Initial release: 1989
• Misc: doesn't save owner/group of files

Compress

• Zip file to the archive:

user@SOURCE:~$ zip FILE.zip FILE

• Zip directory to the archive:

user@SOURCE:~$ zip -r DIR.zip DIR

Decompress

• Decompress FILE.zip:

user@SOURCE:~$ unzip FILE.zip

zstd

• Compression: yes
• Archiving: no
• Initial release: 2015
• Misc:

Compress

• Compress FILE to FILE.zstd:

user@SOURCE:~$ zstd -z FILE

Decompress

• Decompress FILE.zst to FILE:

user@SOURCE:~$ zstd -d FILE.zst

Benchmark Results

Command used

The following command allows to measure the size of the elements before and after compression, and also to know the execution time.

user@SOURCE:~$ echo "BEFORE : $(du -sh "AUDIO")"; time ( tar --gzip -cvf "AUDIO".tar.gz "AUDIO" ); echo "AFTER : $(du -sh "AUDIO".tar.gz)"

BEFORE : 101M	AUDIO
AUDIO/
AUDIO/The_Dolphins_-_Demo_-_Easy_Way.mp3
AUDIO/The_Dolphins_-_Demo_-_Easy_Way.ogg
AUDIO/The_Dolphins_-_Demo_-_Illusions_And_Witnesses.mp3
AUDIO/The_Dolphins_-_Demo_-_Illusions_And_Witnesses.ogg
AUDIO/The_Dolphins_-_Demo_-_Impro.mp3
AUDIO/The_Dolphins_-_Demo_-_Impro.ogg
AUDIO/The_Dolphins_-_Demo_-_Insubstantial_As_Me.mp3
AUDIO/The_Dolphins_-_Demo_-_Something.ogg
AUDIO/The_Dolphins_-_Demo_-_The_Elements_Of_A_State.mp3
AUDIO/The_Dolphins_-_Demo_-_The_Elements_Of_A_State.ogg
AUDIO/The_Dolphins_-_Demo_-_The_Wisp.mp3
AUDIO/The_Dolphins_-_Demo_-_The_Wisp.ogg
AUDIO/The_Dolphins_-_Demo_-_Unnecessary_Evil.mp3
AUDIO/The_Dolphins_-_Demo_-_Unnecessary_Evil.ogg

real	0m3,390s
user	0m3,294s
sys	0m0,167s
AFTER : 101M	AUDIO.tar.gz

Performance Chart