Data compression benchmark under GNU/Linux

Intro

I was wondering about which was the best files compression and decompression software under GNU/Linux so I made a real-world test to see the differences in performance.

Test Environment

• I created six folders with the following characteristics :
  • AUDIO : 101MB, seven mp3 and seven ogg files
  • IMG : 102MB, 17 jpg files
  • PDF : 102MB, 68 pdf files
  • RANDOM TXT : 95MB, one big txt file, made with base64 /dev/urandom | head -c 100000000 > file.txt command.
  • MAN TXT : 7.2MB, 870 man files
  • VIDEOS : 290MB, two mp4, one mpeg and one webm file

Tools

7zip

• Compression : yes
• Archiving : yes
• Initial release : 1999
• Misc : doesn't save owner/group of files

Compress

• Add file/directory to the archive :

user@SOURCE:~$ 7z a DEST.7z SOURCE

Decompress

• Extract with full path :

user@SOURCE:~$ 7z x DEST.7z

bzip2

• Compression : yes
• Archiving : no
• Initial release : 1996
• Misc :

Compress

• Compress FILE to FILE.bz2 and remove FILE (-k to keep) :

user@SOURCE:~$ bzip2 -z FILE

• Same but will keep FILE :

user@SOURCE:~$ bzip2 -c FILE > FILE.bz2

Decompress

• Decompress FILE.bz2 to FILE and remove FILE.bz2 :

user@SOURCE:~$ bzip2 -d FILE

LZ4

• Compression : yes
• Archiving : no
• Initial release : 2011
• Misc :

Compress File

• Compress FILE to FILE.lz4 :

user@SOURCE:~$ lz4 -z FILE FILE.lz4

Decompress

• Decompress FILE.lz4 to FILE :

user@SOURCE:~$ lz4 -d FILE.lz4

gzip

• Compression : yes
• Archiving : no
• Initial release : 1992
• Misc :

Compress

• Compress FILE to FILE.gz and remove FILE (-k to keep) :

user@SOURCE:~$ gzip FILE

• Same but will keep FILE :

user@SOURCE:~$ gzip -c FILE > FILE.gz

Decompress

• Decompress FILE.gz to FILE :

user@SOURCE:~$ gzip -d FILE.gz

xz

• Compression : yes
• Archiving : no
• Initial release :
• Misc :

Compress

• Compress FILE to FILE.xz and remove FILE (-k to keep) :

user@SOURCE:~$ xz -z FILE

• Same but will keep FILE :

user@SOURCE:~$ xz -c FILE > FILE.gz

Decompress

• Decompress FILE.xz to FILE :

user@SOURCE:~$ xz -d FILE.xz

zip

• Compression : yes
• Archiving : yes
• Initial release : 1989
• Misc : doesn't save owner/group of files

Compress

• Zip file to the archive :

user@SOURCE:~$ zip FILE.zip FILE

• Zip directory to the archive :

user@SOURCE:~$ zip -r DIR.zip DIR

Decompress

• Decompress FILE.zip :

user@SOURCE:~$ unzip FILE.zip

zstd

• Compression : yes
• Archiving : no
• Initial release : 2015
• Misc :

Compress

• Compress FILE to FILE.zstd :

user@SOURCE:~$ zstd -z FILE.zst

Decompress

• Decompress FILE.zst to FILE :

user@SOURCE:~$ zstd -d FILE.zst

Benchmark Results

Command used

The following command allows to measure the size of the elements before and after compression, and also to know the execution time.

user@SOURCE:~$ echo "BEFORE : $(du -sh "AUDIO")"; time ( tar --gzip -cvf "AUDIO".tar.gz "AUDIO" ); echo "AFTER : $(du -sh "AUDIO".tar.gz)"

BEFORE : 101M	AUDIO
AUDIO/
AUDIO/The_Dolphins_-_Demo_-_Easy_Way.mp3
AUDIO/The_Dolphins_-_Demo_-_Easy_Way.ogg
AUDIO/The_Dolphins_-_Demo_-_Illusions_And_Witnesses.mp3
AUDIO/The_Dolphins_-_Demo_-_Illusions_And_Witnesses.ogg
AUDIO/The_Dolphins_-_Demo_-_Impro.mp3
AUDIO/The_Dolphins_-_Demo_-_Impro.ogg
AUDIO/The_Dolphins_-_Demo_-_Insubstantial_As_Me.mp3
AUDIO/The_Dolphins_-_Demo_-_Something.ogg
AUDIO/The_Dolphins_-_Demo_-_The_Elements_Of_A_State.mp3
AUDIO/The_Dolphins_-_Demo_-_The_Elements_Of_A_State.ogg
AUDIO/The_Dolphins_-_Demo_-_The_Wisp.mp3
AUDIO/The_Dolphins_-_Demo_-_The_Wisp.ogg
AUDIO/The_Dolphins_-_Demo_-_Unnecessary_Evil.mp3
AUDIO/The_Dolphins_-_Demo_-_Unnecessary_Evil.ogg

real	0m3,390s
user	0m3,294s
sys	0m0,167s
AFTER : 101M	AUDIO.tar.gz

Performance Chart