Characterizing and Accelerating Bioinformatics Workloads on Modern Microarchitectures

Bioinformatics, the use of computer techniques to analyze biological data, has been a particularly active research field in the last two decades. Advances in this field have contributed to the collection of enormous amounts of data, and the sheer amount of available data has started to overtake the processing capability possible with current computer systems. Clearly, computer architects need to have a better understanding of how bioinformatics applications work and what kind of architectural techniques could be used to accelerate these important scientific workloads on future processors. In this dissertation, we develop a bioinformatic benchmark suite and provide a detailed characterization of these applications in common use today from a computer architect's point of view. We analyze a wide range of detailed execution characteristics including instruction mix, IPC measurements, L1 and L2 cache misses on a real architecture; and proceed to analyze the workloads' memory access characteristics. We then concentrate on accelerating a particularly computationally intensive bioinformatics workload on the novel Cell Broadband Engine multiprocessor architecture. The HMMER workload is used for protein profile searching using hidden Markov models, and most of its execution time is spent running the Viterbi algorithm. We parallelize and partition the HMMER application to implement it on the Cell Broadband Engine. In order to run the Viterbi algorithm on the 256KB local stores of the Cell BE synergistic processing units (SPEs), we present a method to develop a fast SIMD implementation of the Viterbi algorithm that reduces the storage requirements significantly. Our HMMER implementation for the Cell BE architecture, Cell-HMMER, exploits the multiple levels of parallelism inherent in this application, and can run protein profile searches up to 27.98 times faster than a modern dual-core x86 microprocessor

Galaxy based BLAST submission to distributed national high throughput computing resources

To assist the bioinformatic community in leveraging the national cyberinfrastructure, the National Center for Genomic Analysis Support (NCGAS) along with Indiana University's High Throughput Computing (HTC) group have engineered a method to use the Galaxy to submit BLAST jobs to the Open Science Grid (OSG). OSG is a collaboration of resource providers that utilize opportunistic cycles at more than 100 universities and research centers in the US. BLAST jobs make a significant portion of the research conducted on NCGAS resources, moving jobs that are conducive to an HTC environment to the national cyberinfrastructure would alleviate load on resources at NCGAS and provide a cost effective solution for getting more cycles to reduce the unmet needs of bioinformatic researchers. To this point researchers have tackled this issue by purchasing additional resources or enlisting collaborators doing the same type of research, while HTC experts have focused on expanding the number of resources available to historically HTC friendly science workflows. In this paper, we bring together expertise from both areas to address how a bioinformatics researcher using their normal interface, Galaxy, can seamlessly access the OSG which routinely supplies researchers with millions of compute hours daily. Efficient use of these results will supply additional compute time to researcher and help provide a yet unmet need for BLAST computing cycles.This material is based upon work supported by the National Science Foundation under Grant No. ABI-1062432, Craig Stewart, PI. William Barnett, Matthew Hahn, and Michael Lynch, co-PIs. This work was supported in part by the Lilly Endowment, Inc. and the Indiana University Pervasive Technology Institute. Any opinions presented here are those of the presenter(s) and do not necessarily represent the opinions of the National Science Foundation or any other funding agencie

Detection of RNA from a Novel West Nile-like Virus and High Prevalence of an Insect-specific Flavivirus in Mosquitoes in the Yucatan Peninsula of Mexico

As part of our ongoing surveillance efforts for West Nile virus (WNV) in the Yucatan Peninsula of Mexico, 96,687 mosquitoes collected from January through December 2007 were assayed by virus isolation in mammalian cells. Three mosquito pools caused cytopathic effect. Two isolates were orthobunyaviruses (Cache Valley virus and Kairi virus) and the identity of the third infectious agent was not determined. A subset of mosquitoes was also tested by reverse transcription-polymerase chain reaction (RT-PCR) using WNV-, flavivirus-, alphavirus-, and orthobunyavirus-specific primers. A total of 7,009 Culex quinquefasciatus in 210 pools were analyzed. Flavivirus RNA was detected in 146 (70%) pools, and all PCR products were sequenced. The nucleotide sequence of one PCR product was most closely related (71-73% identity) with homologous regions of several other flaviviruses, including WNV, St. Louis encephalitis virus, and Ilheus virus. These data suggest that a novel flavivirus (tentatively named T\u27Ho virus) is present in Mexico. The other 145 PCR products correspond to Culex flavivirus, an insect-specific flavivirus first isolated in Japan in 2003. Culex flavivirus was isolated in mosquito cells from approximately one in four homogenates tested. The genomic sequence of one isolate was determined. Surprisingly, heterogeneous sequences were identified at the distal end of the 5\u27 untranslated region

정확한 서열정렬기법과 인메모리 핵심 유전자 데이터베이스 기반의 향상된 메타유전체 분류법

학위논문 (박사) -- 서울대학교 대학원 : 자연과학대학 협동과정 생물정보학전공, 2020. 8. 천종식.샷건 메타지노믹스는 미생물과 숙주 또는 환경사이의 미치는 영향을 이해하는데 매우 중요한 역할을 하고 있다. 기술의 발달과 더불어 메타지노믹스를 통한 올바른 미생물 종의 동정과 각 종들의 분포는 마이크로바이옴 연구의 핵심 구성요소가 되었으며, 지난 10년간 샷건 메타지노믹스 분석을 위한 여러 알고리즘과 데이터베이스들이 개발되어져 왔다. 하지만 서로 다른 기준 데이터 혹은 알고리즘을 사용한 방법들은 서로 다른 분류 정보와 분석 파이프라인으로 인하여 편향된 결과를 나타내기도 하였는데, 이를 보완하고 보다 정확한 분류 동정을 위해 배양이 어려운 표준 균주와 같은 다양한 균주의 유전체 데이터를 포함하는 기준 데이터베이스의 중요성이 대두되고 있다. 샷건 메타지노믹스 분석에서 또 다른 중요한 요소는 분석에 소요되는 시간이라 할 수 있는데 대부분의 생물정보학적 프로그램들은 계산을 수행함에 있어 메모리와 알고리즘 최적화가 되어있지 않아 분석에 상당한 시간이 소요되는 문제점이 있다. 이러한 문제를 해결하기 위해, 본 연구에서는 exact match k-mer classification과 같은 방법을 사용하여 분석 속도를 향상시켰으며 Up-to-date Bacterial Core Gene (UBCG)를 기준 데이터베이스로 사용하여 보다 정확한 샷건 메타지노믹 분석을 수행할 수 있게 하였다. 분석의 효율성을 높이기 위해 두개의 기준 UBCG 데이터베이스가 만들어 졌으며 한 개는 박테리아의 분류체계에서 유효한 종명 (Valid names)만을 가지고 있는 데이터베이스와 다른 하나는 유효한 종명과 함께 EzBioCloud에 있는 genomospecies를 가지고 생성하였다. 검증을 위해 Streptococcus 종을 포함하는 (i) 합성된 메타지놈 샘플과 (ii) 만성 폐쇄성 폐질환(COPD) 환자의 임상 검체 (iii) 혈류 감염 환자의 임상 검체로 이루어진 세개의 데이터 셋을 이용하였으며 기존에 널리 알려진 샷것 파이프라인인 MetaPhlan2과 본 연구의 파이프라인을 비교 분석하였다. 위 검증 분석에서 UBCG를 기준 서열로 사용하기에 충분함을 검증하였으며, 빠르고 정확하게 기준 유전체에서 UBCG 서열을 뽑아 샷건 분석에 용이함을 증명하였다. 또한 genomospecies를 기준 데이터베이스에 추가함으로써, 보다 개선된 분류 정확도를 얻을 수 있음을 제시하였다. 마지막으로 비록 여러 파이프라인과 데이터베이스들이 존재하지만 보다 신뢰할 수 있는 분류결과를 얻기 위해선 기준 데이터베이스의 지속적인 업데이트와 분류 체계의 검증의 중요함을 강조하였다. 이후 본 연구에서 개발된 파이프라인을 이용하여 4,000개의 샷건 메타지놈 샘플에서 사람에 장내에 가장 많이 발견되는 Bacteroides 종에 대한 분석을 수행하였다. 많은 양의 데이터를 분석하여야 하기 때문에 기존에 많이 사용되는 MetaPhlAn2 과 같은 방법은 사용할 수 없었으며 분석 결과 Bacteroides는 도시화된 사람에게 많이 분포하는 반면 아프리카 혹은 남미지역에서 원시적 부족의 삶을 사는 사람에게서는 상대적으로 적게 분포함을 확인할 수 있었다. 또한 각 나라별 인구에서는 우점되는 Bacteroides 종이 다름을 확인할 수 있었는데 이는 각 연구의 샘플링 방법 혹은 위치에 따라 설명되어 질 수 있었다. 실험용 쥐의 결과에서는 가장 다양한 Bacteroides를 관찰할 수 있었으며 이는 많은 수의 기준 유전체가 생쥐에게서 나왔기 때문인 것으로 생각된다. 또한 고양이나 강아지 같은 반려동물의 샘플에서도 높은 상관관계를 발견할 수 있었는데 각 동물들의 생활양식과 먹이에 따른 결과인 것으로 보인다. 본 연구를 통해 보다 많은 메타지놈 데이터 분석의 필요성을 강조하고 있으며, 핵심 유전자들을 기준 데이터로 사용하는 방법의 실효성과 성능을 검증하였다. 이러한 핵심 유전자 기반의 기준 데이터베이스는 보다 정확하고 전체 미생물의 풍부도를 예측하는데 중요한 역할을 하는 것을 확인하였고 k-mer 방법을 통해 기존에 존재하던 다른 파이프라인 보다 더욱 빠른 결과를 도출할 수 있었다. 마지막으로 빠르게 기준 데이터베이스를 만들 수 있기 때문에 항상 최신의 데이터를 가지고 분석을 수행할 수 있으며 이는 궁극적으로 본 연구의 파이프라인을 실질적으로 연구나 진단 목적으로 이용하는 연구자들에게 큰 도움이 될 것이다.Shotgun metagenomics is of great importance to understand the microbial community composition of a sample and the impact it has on its host. The proper identification and quantification of bacterial species is a key component of any microbiome research that is based on metagenomic samples. In the last decade, several algorithms and databases have been developed, however the differences between references and the type of algorithm used for the classification makes the comparisons among themselves unfair and bias. The contents of the reference database, including genome sequences of type strains or reference genomes of uncultured species, have a great impact on the performance of the classification results of metagenomic samples. Another significant factor on shotgun metagenomics is the classification speed as most current bioinformatic tools lack computational and memory optimization. Here, I propose several enhancements to a well-known method, exact match k-mer classification in order to increase the overall speed of a metagenomic classification. This method was further improved by the use of Up-to-date Bacterial Core Gene (UBCG) sequences to provide better method for a faster and accurate shotgun metagenomic profiling classification. In order to prove the efficiency of our method, I built two UBCG-based reference databases: one containing UBCG sequences of valid named species, and the second one containing UBCG sequences of all valid named species and genomospecies in the EzBioCloud database. Three datasets containing Streptococcus species were used to evaluate the improved method against the MetaPhlan2 tool which is the most widely used open-source shotgun metagenomic classifier: (i) synthetic metagenomic samples, (ii) clinical sputum samples from patients with chronic obstructive pulmonary disease (COPD), and (iii) clinical samples of a blood stream infection. In this analysis, I demonstrated that UBCG sequences can be used as references for metagenomic classification, showing that they are easy to extract from genome sequences and accurate when predicting relative abundance. I also showed that the inclusion of genomospecies in the reference databases, significantly improves the classification accuracy of bacterial species within a metagenomic sample. Finally, I showed that while publicly available pipelines and databases are easily accessible, for accurate and reliable taxonomic classification, an updated database with proper taxonomic and genomic curation must be used. The method devised in this work is then applied to profile the Bacteroides species in over 4,000 shotgun metagenomic samples, which is one of most abundant members of the human gut microbiome. This task cannot be accomplished using conventional tools such as MetaPhlAn2 due to the high processing time they require. The results in this study showed that Bacteroides is high abundant in human samples from urban areas while being low abundant in humans from rural areas, particularly African and South American tribes. Countries showed dominance for a specific Bacteroides species, but this could also be explained by the type of study were the samples came from. Mice samples showed the most diversity of Bacteroides, this can be attributed by the number of bacterial references isolated from this organism. House cat and dog samples showed correlation between each other, this may be attributed to the similarities of their lifestyle and diet. This study shows the importance of having a great number of samples for any given metagenomic analysis, and even though, we have profiled thousands of samples, more might be needed in the future. The method proposed in this thesis demonstrates that core genes are reliable reference sequences for shotgun metagenomics. Their implementation as reference sequences in metagenomic databases improves the accuracy of the abundance prediction of any given sample. Additionally, with the use of a k-mer approach, this methods running time outperforms the most popular shotgun metagenomic tools. The work presented in this thesis aims to help microbial research by providing faster and accurate metagenomic taxonomic predictions. Finally, with the ability of updating a metagenomic database with ease, will help researchers to obtain the most up-to-date results to find potential diagnosis or treatments for diseases associated to human microbial communities.Chapter 1. General Introduction 1 1.1. Introduction to metagenomics 2 1.2. 16S rRNA sequencing 3 1.3. Shotgun metagenomic sequencing 5 1.3.1. History 5 1.3.2. Sample extraction 7 1.3.3. Library preparation 8 1.3.4. Sequencing 8 1.4. Shotgun metagenomic classification 9 1.4.1. Homology-based approaches 9 1.4.2. Exact match K-mer approaches 11 Chapter 2. An exact match k-mer algorithm 13 2.1. An exact match k-mer classification approach 14 2.1.1. Definition of the problem 14 2.1.2. Building a k-mer reference database 14 2.1.2.1. K-mer counting 14 2.1.2.2. K-mer mapping 16 2.1.3. Classification of a metagenomic read 16 2.1.3.1. K-mer search 19 2.1.3.2. Scoring a metagenomic read 20 2.1.4. Calculating the metagenome profile 20 2.1.4.1. Normalization for LCA-assigned reads 21 2.1.4.2. Normalization for cell count relative abundance 22 2.2. RAM memory usage 22 2.3. Quality Control 23 2.3.1. Read Trimming 23 2.3.2. Host read removal 24 Chapter 3. Revealing unrecognized species in the genus Streptococcus 28 3.1. A brief history of streptococcus in clinical metagenomics 29 3.2. Results and Discussion 32 3.2.1. Building a core gene reference database 32 3.2.2. Evaluation of Pipelines using Synthetic Metagenomes 36 3.2.3. Chronic obstructive pulmonary disease samples 44 3.2.3. Evaluating the value of genomospecies references in a metagenomic database 56 3.2.4. Identifying accurately a Streptococcal infection using clinical data 63 3.2.5. Effects of different ANI thresholds on the classification of genomospecies 69 3.3. Materials and Methods 76 3.3.1. Selecting the reference genomes 76 3.3.2. Average nucleotide identity and hierarchical clustering 76 3.3.3. Synthetic and Real metagenomic samples 77 3.3.4. Extracting the core genes 77 3.3.5. Taxonomic profiling 83 3.3.6. Biomarker discovery 84 3.4. Conclusions 85 Chapter 4. A large-scale shotgun metagenomic analysis on Bacteroides 86 4.1. Introduction 87 4.2. Bacteroides on the human gut 89 4.2.1. Collecting the samples 89 4.2.2. Methods 89 4.2.2.1. Reference Genomes 89 4.2.2.2. Metagenome profiling 90 4.2.3. Results 103 4.3. Bacteroides on Animal Species 128 4.3.1. Methods 128 4.3.2. Results 128 4.4. Discussion and conclusions 133 General Conclusion 135 References 139 Appendix I. A list of genomes from the genus Streptococcus used on Chapters 3 analysis. 146 국문초록 155Docto

Mechanisms to improve the efficiency of hardware data prefetchers

A well known performance bottleneck in computer architecture is the so-called memory wall. This term refers to the huge disparity between on-chip and off-chip access latencies. Historically speaking, the operating frequency of processors has increased at a steady pace, while most past advances in memory technology have been in density, not speed. Nowadays, the trend for ever increasing processor operating frequencies has been replaced by an increasing number of CPU cores per chip. This will continue to exacerbate the memory wall problem, as several cores now have to compete for off-chip data access. As multi-core systems pack more and more cores, it is expected that the access latency as observed by each core will continue to increase. Although the causes of the memory wall have changed, it is, and will continue to be in the near future, a very significant challenge in terms of computer architecture design. Prefetching has been an important technique to amortize the effect of the memory wall. With prefetching, data or instructions that are expected to be used in the near future are speculatively moved up in the memory hierarchy, were the access latency is smaller. This dissertation focuses on hardware data prefetching at the last cache level before memory (last level cache, LLC). Prefetching at the LLC usually offers the best performance increase, as this is where the disparity between hit and miss latencies is the largest. Hardware prefetchers operate by examining the miss address stream generated by the cache and identifying patterns and correlations between the misses. Most prefetchers divide the global miss stream in several sub-streams, according to some pre-specified criteria. This process is known as localization. The benefits of localization are well established: it increases the accuracy of the predictions and helps filtering out spurious, non-predictable misses. However localization has one important drawback: since the misses are classified into different sub-streams, important chronological information is lost. A consequence of this is that most localizing prefetchers issue prefetches in an untimely manner, fetching data too far in advance. This behavior promotes data pollution in the cache. The first part of this thesis proposes a new class of prefetchers based on the novel concept of Stream Chaining. With Stream Chaining, the prefetcher tries to reconstruct the chronological information lost in the process of localization, while at the same time keeping its benefits. We describe two novel Stream Chaining prefetching algorithms based on two state of the art localizing prefetchers: PC/DC and C/DC. We show how both prefetchers issue prefetches in a more timely manner than their nonchaining counterparts, increasing performance by as much as 55% (10% on average) on a suite of sequential benchmarks, while consuming roughly the same amount of memory bandwidth. In order to hide the effects of the memory wall, hardware prefetchers are usually configured to aggressively prefetch as much data as possible. However, a highly aggressive prefetcher can have negative effects on performance. Factors such as prefetching accuracy, cache pollution and memory bandwidth consumption have to be taken into account. This is specially important in the context of multi-core systems, where typically each core has its own prefetching engine and there is high competition for accessing memory. Several prefetch throttling and filtering mechanisms have been proposed to maximize the effect of prefetching in multi-core systems. The general strategy behind these heuristics is to promote prefetches that are more likely to be used and cause less interference. Traditionally these methods operate at the source level, i.e., directly into the prefetch engine they are assigned to control. In multi-core systems all prefetches are aggregated in a FIFO-like data structure called the Prefetch Request Queue (PRQ), where they wait to be dispatched to memory. The second part of this thesis shows that a traditional FIFO PRQ does not promote a timely prefetching behavior and usually hinders part of the performance benefits achieved by throttling heuristics. We propose a novel approach to prefetch aggressiveness control in multi-cores that performs throttling at the PRQ (i.e., global) level, using global knowledge of the metrics of all prefetchers and information about the global state of the PRQ. To do this, we introduce the Resizable Prefetching Heap (RPH), a data structure modeled after a binary heap that promotes timely dispatch of prefetches as well as fairness in the distribution of prefetching bandwidth. The RPH is designed as a drop-in replacement of traditional FIFO PRQs. We compare our proposal against a state-of-the-art source-level throttling algorithm (HPAC) in a 8-core system. Unlike previous research, we evaluate both multiprogrammed and multithreaded (parallel) workloads, using a modern prefetching algorithm (C/DC). Our experimental results show that RPH-based throttling increases the throttling performance benefits obtained by HPAC by as much as 148% (53.8% average) in multiprogrammed workloads and as much as 237% (22.5% average) in parallel benchmarks, while consuming roughly the same amount of memory bandwidth. When comparing the speedup over fixed degree prefetching, RPH increased the average speedup of HPAC from 7.1% to 10.9% in multiprogrammed workloads, and from 5.1% to 7.9% in parallel benchmarks