REINDEER: efficient indexing of k-mer presence and abundance in sequencing datasets

• Motivation
• Installation
  • Requirements
  • Quick start
• Output
• Beta options
  • log counts/quantized counts
  • input paired-end reads (to bcalm)
• Reproduce the manuscript's results
• Citation and resources
• Advanced FAQ

REINDEER builds a data-structure that indexes k-mers and their abundances in a collection of datasets (raw RNA-seq or metagenomic reads for instance). Then, a sequence (FASTA) can be queried for its presence and abundance in each indexed dataset. While other tools (e.g. SBT, BIGSI) were also designed for large-scale k-mer presence/absence queries, retrieving abundances was so far unsupported (except for single datasets, e.g. using some k-mer counters like KMC, Jellyfish). REINDEER combines fast queries, small index size, and low memory footprint during indexing and queries. We showed it allows to index 2585 RNA-seq datasets (~4 billions k-mers) using less than 60GB of RAM and a final index size lower than 60GB on the disk. Then, a REINDEER index can either be queried on disk (low RAM usage) or be loaded in RAM for faster queries.

Note on presence/absence queries: REINDEER supports this type of queries, although other data structures are more fit for this task. See for instance:

• HowDeSBT
• Vari-Merge
• BIGSI
• BiFrost

to name a few.

• GCC >= 4.8
• CMAKE > 3.10.0

To install, first clone the project:

git clone --recursive https://github.com/kamimrcht/REINDEER.git

Then:

cd REINDEER

sh install.sh

or

cd REINDEER

make

Test can be run:

make test

If REINDEER gives a Error: no such instruction during compilation, try replacing -march=native -mtune=native by -msse4 in the file makefile. If this did not work, please file an issue.

Have a look at the file of file format in test/fof_unitigs.txt. REINDEER assumes unitig files have been created using BCALM. You can provide a file of file of each unitig file (fasta) instead of the read files (-f) and an output directory for the index files (-o). By default, index files will be written in reindeer_index_files_ + date + a tag. Then build the index:

./Reindeer --index -f test/fof_unitigs.txt -o quick_out

and query: simply provide the fasta query file (single line) to Reindeer using -q, along with the directory of index files that were generated during index construction (-l), and a writeable path for the output (-o). Default output will be written in the query_result directory:

./Reindeer --query -q test/query_test.fa -l quick_out -o quick_out/result_test.txt

In this example, results should be in quick_out/result_test.txt.

Help:

./Reindeer --help

When using output with counts, we have 4 kinds of counts:

The last 3 formats are available since version 1.4.

These formats compute the sum of all k-mer in query sequence, so if length(query sequence) = k, you will get sum = average.

Let's say that the query is 51 bases long and we look for 31-mers. There are 21 k-mers in the query, from k-mer 0 to k-mer 20. The output of REINDEER looks like:

Why can we observe different values for k-mers in a single query? I answer this question in the advanced FAQ.

By default, REINDEER records k-mers abundances in each input dataset. In order to have k-mer presence/absence instead of abundance per indexed dataset, use --nocount option.

./Reindeer --index -f test/fof_unitigs.txt --nocount -o index_nocount

./Reindeer --query -l index_nocount -q test/query_test.fa --nocount

./Reindeer --index --log-count -f fof_unitigs.txt

./Reindeer --index --quantization -f test/fof_unitigs.txt

./Reindeer --index --paired-end --bcalm -f fof.txt

We provide a page with scripts to reproduce the results we show in our manuscript, link here.

REINDEER has been published in Bioinformatics. It was presented during ISMB conference in 2020. Main developper: Camille Marchet.

Access to the preprint: REINDEER: efficient indexing of k-mer presence and abundance in sequencing datasets.

Presentation recorded during BiATA 2020.

Citations:

@inproceedings{marchet2020reindeer,
  title ={{REINDEER}: efficient indexing of k-mer presence and abundance in sequencing datasets},
 author={Marchet, Camille and Iqbal, Zamin and Gautheret, Daniel and Salson, Mika{\"e}l and Chikhi, Rayan},
  booktitle =        {28th Intelligent Systems for Molecular Biology (ISMB 2020)},
  year =         {2020},
  doi =         {10.1101/2020.03.29.014159},
}
@article{marchet2020reindeer,
  title={REINDEER: efficient indexing of k-mer presence and abundance in sequencing datasets},
  author={Marchet, Camille and Iqbal, Zamin and Gautheret, Daniel and Salson, Mika{\"e}l and Chikhi, Rayan},
  journal={Bioinformatics},
  volume={36},
  number={Supplement\_1},
  pages={i177--i185},
  year={2020},
  publisher={Oxford University Press}
}

Major

• Default disk mode (index written on disk and disk queries)
• Dependency on C++17
• Output now supports different k-mer countig modes (sum, average, normalized)

Minor

• Change of index file names
• Correction of various bugs
• Switched to object implementation
• Some modifications to pass the input/output files
• Implementation of code for socket mode [beta]
• Allowing lowercase in query files
• Multi-threading inactivated in query mode

Major

• since version 1.4, we change the format of reindeer_matrix_eqc_info file. It is now a text file instead of a binary format. To convert the file, you can use the update_reindeer_info program to update the file for index build with previous version of Reindeer. The normalisation format require the total number of k-mer in each unitig file (bcalm output). For previous index build, 2 possibility:
  • You have access to the unitig files: If you give in argument the fof.txt file (-f option), listing the location of unitig file, the program will count the total k-mer in each unitig file and store them in the file.
  • You don't have the unitig files: convert the reindeer_matrix_eqc_info with the converter without -f option.
    • If you don't insert the total k-mer per sample in this file, it will be not possible to use the format 'normalization'.