2 research outputs found
MetaCache-GPU: Ultra-Fast Metagenomic Classification
The cost of DNA sequencing has dropped exponentially over the past decade,
making genomic data accessible to a growing number of scientists. In
bioinformatics, localization of short DNA sequences (reads) within large
genomic sequences is commonly facilitated by constructing index data structures
which allow for efficient querying of substrings. Recent metagenomic
classification pipelines annotate reads with taxonomic labels by analyzing
their -mer histograms with respect to a reference genome database. CPU-based
index construction is often performed in a preprocessing phase due to the
relatively high cost of building irregular data structures such as hash maps.
However, the rapidly growing amount of available reference genomes establishes
the need for index construction and querying at interactive speeds. In this
paper, we introduce MetaCache-GPU -- an ultra-fast metagenomic short read
classifier specifically tailored to fit the characteristics of CUDA-enabled
accelerators. Our approach employs a novel hash table variant featuring
efficient minhash fingerprinting of reads for locality-sensitive hashing and
their rapid insertion using warp-aggregated operations. Our performance
evaluation shows that MetaCache-GPU is able to build large reference databases
in a matter of seconds, enabling instantaneous operability, while popular
CPU-based tools such as Kraken2 require over an hour for index construction on
the same data. In the context of an ever-growing number of reference genomes,
MetaCache-GPU is the first metagenomic classifier that makes analysis pipelines
with on-demand composition of large-scale reference genome sets practical. The
source code is publicly available at https://github.com/muellan/metacache .Comment: 11 pages. To be published in ICPP 202
HashGraph -- Scalable Hash Tables Using A Sparse Graph Data Structure
Hash tables are ubiquitous and used in a wide range of applications for
efficient probing of large and unsorted data. If designed properly, hash-tables
can enable efficients look ups in a constant number of operations or commonly
referred to as O(1) operations. As data sizes continue to grow and data becomes
less structured (as is common for big-data applications), the need for
efficient and scalable hash table also grows. In this paper we introduce
HashGraph, a new scalable approach for building hash tables that uses concepts
taken from sparse graph representations--hence the name HashGraph. We show two
different variants of HashGraph, a simple algorithm that outlines the method to
create the hash-table and an advanced method that creates the hash table in a
more efficient manner (with an improved memory access pattern). HashGraph shows
a new way to deal with hash-collisions that does not use "open-addressing" or
"chaining", yet has all the benefits of both these approaches. HashGraph
currently works for static inputs, though recent progress with dynamic graph
data structures suggest that HashGraph might be extended to dynamic inputs as
well. We show that HashGraph can deal with a large number of hash-values per
entry without loss of performance as most open-addressing and chaining
approaches have. Further, we show that HashGraph is indifferent to the
load-factor. Lastly, we show a new probing algorithm for the second phase of
value lookups. Given the above, HashGraph is extremely fast and outperforms
several state of the art hash-table implementations. The implementation of
HashGraph in this paper is for NVIDIA GPUs, though HashGraph is not
architecture dependent. Using a NVIDIA GV100 GPU, HashGraph is anywhere from
2X-8X faster than cuDPP, WarpDrive, and cuDF. HashGraph is able to build a
hash-table at a rate of 2.5 billion keys per second and can probe at nearly the
same rate