EDIT: The author of the CoLoRd paper responded with clarification and some parts of the note were corrected with corresponding EDIT tag. His response is copied at the bottom of the post (also in the comments section way below). Thank you!
I recently found out about a new raw sequencing data compression and archiving tool called CoLoRd (reference Kokot M, Gudys A, Li H, Deorowicz S. CoLoRd: Compressing long reads. bioRxiv. 2021 Jan 1. And direct link to the github repository: https://github.com/refresh-bio/CoLoRd). The new tool is developed in collaboration with the inimitable Heng Li (http://www.liheng.org/), the mind behind SAMtools, htslib, BWA short aligner, minimap2, and many other tools that anyone interested in modern sequencing and genomics must have utilized at least once in their career. What a world, for a mere warehouse worker to be able to find out about tools like this in pre-print stages and test them out right away.
From a casual read of the preprint, CoLoRd aims to differentiate itself from other general purpose compression tools used in bioinformatics today by treating the compression object as a specifically biological data in form of fastq raw reads. The tool processes raw fastq data and behaves somewhat like a sloppy mapper, utilizing overlap graphs (I wonder if code from minimap2 went into this part) and then using them as ‘compression anchors’ to arrive at a smaller sized data than one might get with a general purpose compression algorithm. I should note that cluster based compression of raw sequencing data had been in research/use for quite a while now, as described in the pre-print itself, but CoLoRd is the only one of its kind that utilized overlap graphs and is designed exclusively for compression of long-read sequencing data.
Since I made a note on raw sequencing data compression on consumer hardware before (https://naturepoker.wordpress.com/2021/05/17/frugal-bioinformatics-genome-archiving-bench-test/), I thought it would be fun to see how CoLoRd compares against other options on relatively anemic hardware.
The test compression sets (using our 4.7GB Deinococcus radiophilus raw reads) were run across two different machines this time – first is my personal laptop, a Thinkpad X201 (released in 2010) with 8GB of ram and 1st generation core i5 M540 processor with four threads. The second is our lab workstation with 128GB of ram and Xeon CPU E5-2670 with 32 threads. Test script was as follows:
#!/usr/bin/env bash
echo "Starting gzip compression"
time gzip gzip/deino_reads
echo "Starting gzip decompression"
time gzip -dk gzip/deino_reads.gz
echo "Starting zstd dictionary compression"
time ./zstd --rm -D 3_deino_sra_dictionary zstd-d/deino_reads
echo "Starting zstd dictionary decompression"
time ./zstd -D 3_deino_sra_dictionary --decompress zstd-d/deino_reads.zst
echo "Starting zstd compression"
time ./zstd --rm zstd-test/deino_reads
echo "Starting zstd decompression"
time ./zstd -d zstd-test/deino_reads.zst
echo "Starting 7zip compression"
time 7z a deino_reads.7z 7z/deino_reads
echo "Starting 7zip decompression"
time 7z e deino_reads.7z
rm deino_reads
echo "Starting colord ONT compression"
time ./colord compress-ont colord-test/deino_reads colord-test/deino_archive
The output from the Thinkpad X201:
Starting gzip compression
real 7m57.347s
user 7m48.162s
sys 0m8.784s
deino_reads 2.3gb
Starting gzip decompression
real 0m52.766s
user 0m45.284s
sys 0m5.900s
Starting zstd dictionary compression
zstd-d/deino_reads : 48.43% ( 4.62 GiB => 2.24 GiB, zstd-d/deino_reads.zst)
real 1m37.813s
user 1m25.838s
sys 0m8.090s
deino_reads 2.3gb
Starting zstd dictionary decompression
zstd-d/deino_reads.zst: 4960501078 bytes
real 0m49.899s
user 0m14.173s
sys 0m6.789s
Starting zstd compression
zstd-test/deino_reads : 48.43% ( 4.62 GiB => 2.24 GiB, zstd-test/deino_reads.zst)
real 1m52.149s
user 1m24.952s
sys 0m8.227s
deino_reads 2.3gb
Starting zstd decompression
zstd-test/deino_reads.zst: 4960501078 bytes
real 0m50.021s
user 0m14.618s
sys 0m6.644s
Starting 7zip compression
7-Zip [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
p7zip Version 16.02 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,64 bits,4 CPUs Intel(R) Core(TM) i5 CPU M 540 @ 2.53GHz (20652),ASM,AES-NI)
Scanning the drive:
1 file, 4960501078 bytes (4731 MiB)
Creating archive: deino_reads.7z
Items to compress: 1
Files read from disk: 1
Archive size: 2090631247 bytes (1994 MiB)
Everything is Ok
real 60m17.810s
user 172m22.098s
sys 1m11.012s
deino_reads 2.0gb
Starting 7zip decompression
7-Zip [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
p7zip Version 16.02 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,64 bits,4 CPUs Intel(R) Core(TM) i5 CPU M 540 @ 2.53GHz (20652),ASM,AES-NI)
Scanning the drive for archives:
1 file, 2090631247 bytes (1994 MiB)
Extracting archive: deino_reads.7z
--
Path = deino_reads.7z
Type = 7z
Physical Size = 2090631247
Headers Size = 138
Method = LZMA2:24
Solid = -
Blocks = 1
Everything is Ok
Size: 4960501078
Compressed: 2090631247
real 2m59.132s
user 2m44.109s
sys 0m6.881s
Starting colord ONT compression
time ./colord compress-ont colord-test/deino_reads colord-test/deino_archive
Counting k-mers.
Stage 1: 100%
Stage 2: 100%
Filtering k-mers.
100%
Running compression.
100%
DNA size : 407936879
Quality size : 380225121
Header size : 5579892
Meta size : 53
Info size : 127
Total time : 1869.7s
real 31m9.876s
user 101m16.811s
sys 0m34.989s
deino_reads 757mb
And the output from our workstation with Xeon processor:
Starting gzip compression
real 8m47.465s
user 8m42.241s
sys 0m5.000s
deino_reads 2.3gb
Starting gzip decompression
real 0m55.767s
user 0m50.810s
sys 0m4.947s
Starting zstd dictionary compression
zstd-d/deino_reads : 48.43% (4960501078 => 2402284374 bytes, zstd-d/deino_reads.zst)
real 1m14.469s
user 1m14.618s
sys 0m4.788s
deino_reads 2.3gb
Starting zstd dictionary decompression
zstd-d/deino_reads.zst: 4960501078 bytes
real 0m17.928s
user 0m12.891s
sys 0m5.035s
Starting zstd compression
zstd-test/deino_reads : 48.43% (4960501078 => 2402284037 bytes, zstd-test/deino_reads.zst)
real 1m13.985s
user 1m14.303s
sys 0m4.190s
deino_reads 2.3gb
Starting zstd decompression
zstd-test/deino_reads.zst: 4960501078 bytes
real 0m17.841s
user 0m12.848s
sys 0m4.992s
Starting 7zip compression
7-Zip [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
p7zip Version 16.02 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,64 bits,32 CPUs Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz (206D7),ASM,AES-NI)
Scanning the drive:
1 file, 4960501078 bytes (4731 MiB)
Creating archive: deino_reads.7z
Items to compress: 1
Files read from disk: 1
Archive size: 2090631247 bytes (1994 MiB)
Everything is Ok
real 6m42.321s
user 139m25.023s
sys 1m22.728s
deino_reads 2.0gb
Starting 7zip decompression
7-Zip [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
p7zip Version 16.02 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,64 bits,32 CPUs Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz (206D7),ASM,AES-NI)
Scanning the drive for archives:
1 file, 2090631247 bytes (1994 MiB)
Extracting archive: deino_reads.7z
--
Path = deino_reads.7z
Type = 7z
Physical Size = 2090631247
Headers Size = 138
Method = LZMA2:24
Solid = -
Blocks = 1
Everything is Ok
Size: 4960501078
Compressed: 2090631247
real 2m32.988s
user 2m28.492s
sys 0m4.376s
Starting colord ONT compression
time ./colord compress-ont colord-test/deino_reads colord-test/deino_archive
Counting k-mers.
Stage 1: 100%
Stage 2: 100%
Filtering k-mers.
100%
Running compression.
100%
DNA size : 407936880
Quality size : 380225120
Header size : 5579891
Meta size : 53
Info size : 127
Total time : 249.285s
real 4m9.651s
user 88m50.677s
sys 1m4.218s
deino_reads 757mb
Some caveats to note on the results – 7zip by default uses all available cores. Gzip is a single thread process; while there is a multithreaded alternative called pigzip, I decided to stick with what people are mostly likely going to have on their machine (if you have a somewhat higher end laptop, I can see the gzip compression time falling to about 40%~ of what we have on the results above).
I also ran two different zstd processes. Zstd-d is a brief test of their dictionary based compression and decompression function. I created a makeshift dictionary file from three Deinococcus based long-read fastq files (ours plus two from SRA – Deinococcus grandis and Deinococcus sp. D7000), by splitting them into 10MB pieces (there’s a minimum file number requirement for the training set). Alas, the results are nothing much to speak of. Maybe churning through all ONT long-read data and using them as training set will result in a noticeable difference.
As you can see, zstd remains compression time leader – about a minute to wrap everything up, compressing a 4.7GB data into 2.3GB archive. 7zip does a little better in size, with extra 13%~ compression in return for using all available cores on a machine and a significantly longer running time compared to most options. Zstd can achieve a similar size too by running it with -19 argument (highest level of compression without resorting to their ‘ultra’ argument), but the ridiculously increased running time (more so than 7zip) just isn’t a good trade-off in most scenarios.
Now, the most exciting part of the result is final compressed file size from CoLoRd- 4.7GB raw data to 757MB final archive, which is around a mere 16% of the original file size. While it should be noted that the default behavior of CoLoRd is to utilize all available cores on the machine with no option to control the number of threads on the part of the user (EDIT: this part is incorrect – thread control can be turned on with -t argument, as pointed out by the first author of the paper. His additional notes and clarifications are copied at the bottom of this post for future reference), the ridiculous 84% decrease in file size might make it fully worth it for many researchers out there.
I do need to point out one oddity that might matter to some researchers. When using utilities like zstd or gz, the decompressed output from one’s archive (.zst, .gz or otherwise) is expected to be an exact match of the original item. That is not the case for CoLoRd. (EDIT: The author of the paper responded to the post! Please refer to the copied section below for clarification).
When I compare the original deino_reads.fastq file with the decompressed output of a CoLoRd archive (created from the original deino_reads.fastq file, of course) using diff, say, by using below script:
diff deino_reads.fastq color_decompressed_reads.fastq | awk '{print $1}' | grep -c ">"
diff deino_reads.fastq color_decompressed_reads.fastq | awk '{print $1}' | grep -c "<"
The output is 273555- meaning 273555 different lines across both files. If the two .fastq files are perfect copies of one another, the output should be zero/none. From what I can see, CoLoRd does format the input fastq file so that the formatting of the decompressed output will eventually be different from the original, while preserving the information itself. The 273555 number is meaningful in this context since the number of individual fastq reads in a file is derived by:
echo $(cat deino_reads.fastq | wc -l )/4|bc
Simply meaning raw line count divided by 4 – for our fastq file, the number is – you guessed it – 273555 reads. Running the same command on the decompressed CoLoRd archive gives us the same number of reads as well.
I really don’t think this is a big deal – the eventual assembly from a decompressed CoLoRd archive should be the same as the one you’d get with raw, never-compressed fastq file straight from a sequencer. However researchers looking at downstream processing using the raw fastq data itself might want to keep the difference in mind. For an amateur like me, I might just stick with more conventional compression tools for now… (EDIT: CoLoRd in fact does come with a lossless compression option even for the quality score section of the main data, as pointed out by the author of the paper below this post, anyone interested in using the tool should definitely take the time to check it out!)
#Example command for lossless compression of the fastq raw data
./colord compress-ont -q org test.fq o.3rd
On a side note, I also tried to compile the CoLoRd package for the raspberry pi 400 – a small platform I’m working to turn into a mobile bioinformatics terminal with an experimentalist bent. Alas, CoLoRd relies on some x86-64 specific header files for compilation, and replacing them for ARM7/8 platform seems nontrivial. Minimap2 was the same way as well – I wonder how our future bioinformatics workflow will look like as servers and clusters slowly transition to ARM or any number of other alternate platforms?
EDIT 2021 July 30th:
The first author for the CoLoRd compression papers, Marek Kokot kindly dropped by and clarified quite a few things regarding both the questions in the post and the function of the tool. I believe his response is going to be valuable to anyone who read this post & are interested in learning more about CoLoRd. The whole body of his responses are copied below with the author’s permission.
Hello,
great to see the tests of our compressor!
I would like to add some notes and maybe clarify a little.
In fact, the user may specify the number of threads during compression.
An example command line to use 12 threads would be:
./colord compress-ont -t 12 -q org test.fq o.3rd
I would also make a comment about the difference between the original and decompressed file. I think it is a really important matter.
In CoLoRd we split the fastq file into three streams: DNA, quality, header, hence in the output you may see the sizes of each of these streams (the remaining data, i.e. meta and info are always very small and presented for completeness, yet not important for a typical user).
The DNA and header streams are always compressed losslessly.
The header size is relatively small (especially in the case of long reads) so it does not affect the compression ratio very much and a simple approach is sufficient.
The fanciest part of the algorithm is DNA compression which utilizes a very simple (for efficiency) assembly-like approach.
Since we are able to find a lot of similarities using this approach we are able to compress DNA stream very efficiently.
The quality stream is a different story and your note about downstream processing is very important.
The problem with the quality scores is that those are really hard to compress lossless and very often they are not very informative, at least it seems that the number of possible values of quality scores may be reduced without affecting downstream analyses.
In the paper, we show how lossy quality score compression affects some typical downstream analyses (invaluable help from Heng Li in this matter). In short, the conclusion is that it does not affect them very much (detailed results in the paper).
Having this in mind we made lossy quality score compression the default mode, but if one really needs the original quality scores there is a parameter for this, command example:
./colord compress-ont -q org test.fq o.3rd
In fact, CoLoRd allows for a couple of quality scores compression modes, but we have chosen the one that we think (basing on experiment results) is most appropriate for most of the regular users.
There are also three general modes (or priorities): memory, balanced, ratio, which allow controlling the resource requirements and compression ratio.
We are totally aware that the number of internal parameters of CoLoRd is large and non of the regular users will ever try to understand them, and we think that defaults should just work well. We really hope that the defaults we have chosen will do.
I think the above description (longer than I intended) explains the differences you notices. They are just from the default lossy quality compression.
The most upsetting part of your post is the sentence “For an amateur like me, I might just stick with more conventional compression tools for now…”.
Out of curiosity, if CoLoRd would compress quality scores lossless by default, would your conclusion be any different?
I do really like your final note about ARM compatibility. In our team, we are aware of the issue and I do really hope that we will slowly adjust our tools to be ARM-compatible. By using x86-64 specific headers we are often able to offer a much better performance of the code than in the opposite case.
Yet, it is probably better to have a code that may run on ARM, even if it is somehow less performant, than have a code that does not run on ARM at all.
Thanks for checking our compressor and posting your results!
Thank you for a quick response.
Thanks for the suggestion about -h. In fact, there is are two levels of help, the main one, that you have shown and the one deeper, dedicated to each subcommand, so for example you can do:
./colord compress-ont -h
it will show quite a big number of options, one of them is -t.
The reason for introducing two levels is that the options available depends on the command (e.g. compress-, decompress).
Setting the number of threads for decompression is not possible and decompression always uses a couple of threads (4 or 5 as far as I remember).
Maybe we should add in the general help information, that each subcommand has its help…
Well, I am not too upset and I totally understand your point of view. Also, I would like to thank you for the kind words.
This is obviously true that chances for any issues with well-known compressors are much less likely than in the case of new ones. Yet, I also feel like it is my obligation to try to convince people to use our software, especially if it offers much better compression ratios. The more users will use the software the better it should become because some bugs may be found and reported, although I hope that we tested the software well and on a wide range of inputs, so there (hopefully) will be not many bugs.
You may compress lossless, just use “-q org” or “–qual org”, example:
./colord compress-ont -q org test.fq o.3rd
If you are not too busy, I would suggest you extend your experiments to include a lossless variant. The compression ratio will be probably worse, yet after decompression, you will have an identical file like before compression.
Maybe we should expose in help the variants of quality compression modes more clearly…
I am not sure what is the coverage of your dataset, in general, the higher the coverage the better compression ratio CoLoRd should provide.
I don’t mind copy-paste my response. In fact, I would be grateful if you do! (the same for any part of this post if you are interested).
Thanks again!
