14 research outputs found

    Resiliency Mechanisms for In-Memory Column Stores

    Get PDF
    The key objective of database systems is to reliably manage data, while high query throughput and low query latency are core requirements. To date, database research activities mostly concentrated on the second part. However, due to the constant shrinking of transistor feature sizes, integrated circuits become more and more unreliable and transient hardware errors in the form of multi-bit flips become more and more prominent. In a more recent study (2013), in a large high-performance cluster with around 8500 nodes, a failure rate of 40 FIT per DRAM device was measured. For their system, this means that every 10 hours there occurs a single- or multi-bit flip, which is unacceptably high for enterprise and HPC scenarios. Causes can be cosmic rays, heat, or electrical crosstalk, with the latter being exploited actively through the RowHammer attack. It was shown that memory cells are more prone to bit flips than logic gates and several surveys found multi-bit flip events in main memory modules of today's data centers. Due to the shift towards in-memory data management systems, where all business related data and query intermediate results are kept solely in fast main memory, such systems are in great danger to deliver corrupt results to their users. Hardware techniques can not be scaled to compensate the exponentially increasing error rates. In other domains, there is an increasing interest in software-based solutions to this problem, but these proposed methods come along with huge runtime and/or storage overheads. These are unacceptable for in-memory data management systems. In this thesis, we investigate how to integrate bit flip detection mechanisms into in-memory data management systems. To achieve this goal, we first build an understanding of bit flip detection techniques and select two error codes, AN codes and XOR checksums, suitable to the requirements of in-memory data management systems. The most important requirement is effectiveness of the codes to detect bit flips. We meet this goal through AN codes, which exhibit better and adaptable error detection capabilities than those found in today's hardware. The second most important goal is efficiency in terms of coding latency. We meet this by introducing a fundamental performance improvements to AN codes, and by vectorizing both chosen codes' operations. We integrate bit flip detection mechanisms into the lowest storage layer and the query processing layer in such a way that the remaining data management system and the user can stay oblivious of any error detection. This includes both base columns and pointer-heavy index structures such as the ubiquitous B-Tree. Additionally, our approach allows adaptable, on-the-fly bit flip detection during query processing, with only very little impact on query latency. AN coding allows to recode intermediate results with virtually no performance penalty. We support our claims by providing exhaustive runtime and throughput measurements throughout the whole thesis and with an end-to-end evaluation using the Star Schema Benchmark. To the best of our knowledge, we are the first to present such holistic and fast bit flip detection in a large software infrastructure such as in-memory data management systems. Finally, most of the source code fragments used to obtain the results in this thesis are open source and freely available.:1 INTRODUCTION 1.1 Contributions of this Thesis 1.2 Outline 2 PROBLEM DESCRIPTION AND RELATED WORK 2.1 Reliable Data Management on Reliable Hardware 2.2 The Shift Towards Unreliable Hardware 2.3 Hardware-Based Mitigation of Bit Flips 2.4 Data Management System Requirements 2.5 Software-Based Techniques For Handling Bit Flips 2.5.1 Operating System-Level Techniques 2.5.2 Compiler-Level Techniques 2.5.3 Application-Level Techniques 2.6 Summary and Conclusions 3 ANALYSIS OF CODING TECHNIQUES 3.1 Selection of Error Codes 3.1.1 Hamming Coding 3.1.2 XOR Checksums 3.1.3 AN Coding 3.1.4 Summary and Conclusions 3.2 Probabilities of Silent Data Corruption 3.2.1 Probabilities of Hamming Codes 3.2.2 Probabilities of XOR Checksums 3.2.3 Probabilities of AN Codes 3.2.4 Concrete Error Models 3.2.5 Summary and Conclusions 3.3 Throughput Considerations 3.3.1 Test Systems Descriptions 3.3.2 Vectorizing Hamming Coding 3.3.3 Vectorizing XOR Checksums 3.3.4 Vectorizing AN Coding 3.3.5 Summary and Conclusions 3.4 Comparison of Error Codes 3.4.1 Effectiveness 3.4.2 Efficiency 3.4.3 Runtime Adaptability 3.5 Performance Optimizations for AN Coding 3.5.1 The Modular Multiplicative Inverse 3.5.2 Faster Softening 3.5.3 Faster Error Detection 3.5.4 Comparison to Original AN Coding 3.5.5 The Multiplicative Inverse Anomaly 3.6 Summary 4 BIT FLIP DETECTING STORAGE 4.1 Column Store Architecture 4.1.1 Logical Data Types 4.1.2 Storage Model 4.1.3 Data Representation 4.1.4 Data Layout 4.1.5 Tree Index Structures 4.1.6 Summary 4.2 Hardened Data Storage 4.2.1 Hardened Physical Data Types 4.2.2 Hardened Lightweight Compression 4.2.3 Hardened Data Layout 4.2.4 UDI Operations 4.2.5 Summary and Conclusions 4.3 Hardened Tree Index Structures 4.3.1 B-Tree Verification Techniques 4.3.2 Justification For Further Techniques 4.3.3 The Error Detecting B-Tree 4.4 Summary 5 BIT FLIP DETECTING QUERY PROCESSING 5.1 Column Store Query Processing 5.2 Bit Flip Detection Opportunities 5.2.1 Early Onetime Detection 5.2.2 Late Onetime Detection 5.2.3 Continuous Detection 5.2.4 Miscellaneous Processing Aspects 5.2.5 Summary and Conclusions 5.3 Hardened Intermediate Results 5.3.1 Materialization of Hardened Intermediates 5.3.2 Hardened Bitmaps 5.4 Summary 6 END-TO-END EVALUATION 6.1 Prototype Implementation 6.1.1 AHEAD Architecture 6.1.2 Diversity of Physical Operators 6.1.3 One Concrete Operator Realization 6.1.4 Summary and Conclusions 6.2 Performance of Individual Operators 6.2.1 Selection on One Predicate 6.2.2 Selection on Two Predicates 6.2.3 Join Operators 6.2.4 Grouping and Aggregation 6.2.5 Delta Operator 6.2.6 Summary and Conclusions 6.3 Star Schema Benchmark Queries 6.3.1 Query Runtimes 6.3.2 Improvements Through Vectorization 6.3.3 Storage Overhead 6.3.4 Summary and Conclusions 6.4 Error Detecting B-Tree 6.4.1 Single Key Lookup 6.4.2 Key Value-Pair Insertion 6.5 Summary 7 SUMMARY AND CONCLUSIONS 7.1 Future Work A APPENDIX A.1 List of Golden As A.2 More on Hamming Coding A.2.1 Code examples A.2.2 Vectorization BIBLIOGRAPHY LIST OF FIGURES LIST OF TABLES LIST OF LISTINGS LIST OF ACRONYMS LIST OF SYMBOLS LIST OF DEFINITION

    Needles in the Haystack - Tackling Bit Flips in Lightweight Compressed Data

    Get PDF
    Modern database systems are very often in the position to store their entire data in main memory. Aside from increased main emory capacities, a further driver for in-memory database system has been the shift to a column-oriented storage format in combination with lightweight data compression techniques. Using both mentioned software concepts, large datasets can be held and efficiently processed in main memory with a low memory footprint. Unfortunately, hardware becomes more and more vulnerable to random faults, so that e.g., the probability rate for bit flips in main memory increases, and this rate is likely to escalate in future dynamic random-access memory (DRAM) modules. Since the data is highly compressed by the lightweight compression algorithms, multi bit flips will have an extreme impact on the reliability of database systems. To tackle this reliability issue, we introduce our research on error resilient lightweight data compression algorithms in this paper. Of course, our software approach lacks the efficiency of hardware realization, but its flexibility and adaptability will play a more important role regarding differing error rates, e.g. due to hardware aging effects and aggressive processor voltage and frequency scaling. Arithmetic AN encoding is one family of codes which is an interesting candidate for effective software-based error detection. We present results of our research showing tradeoffs between compressibility and resiliency characteristics of data. We show that particular choices of the AN-code parameter lead to a moderate loss of performance. We provide evaluation for two proposed techniques, namely AN-encoded Null Suppression and AN-encoded Run Length Encoding

    1999, UMaine News Press Releases

    Get PDF
    This is an incomplete catalog of press releases posted by the University of Maine Division of Marketing and Communications online between January to September 1999

    Mediated messages: constructions of intimate communication through the use of digital technologies, and the extent to which such encounters can be conceptualised as one-to-one performance

    Get PDF
    In the 21st Century a majority of the world’s population carry in their pockets devices that promise connection to others over distance. The instant connectivity offered by technologies of communication is somewhat of mixed blessing combining the allure of interaction and the threat of availability. Much of the advertising gloss for the technologies of communication – smartphones, video conferencing and social networks – relies on selling the idea of real human connection at a distance. This study sets out to explore the nature of mediated communications between individuals in the context of a perceived opposition that conceptualises technology as either distancing or enhancing what it is to be human. The research frames mediated interactions as one-to-one performance, an approach which encourages the unexpected and playful whist embracing vulnerability. In exploring the nature of the one-to-one performance scholars and audiences stress their experiences as personal, at times intense and certainly intimate. Here intimacy is engaged with as both a subconscious technological fluency as well as intrapersonal closeness, placing such interaction in the socio-cultural context of late capitalism. It is concluded that rather than technology enframing a commodified experience of the world, intimate interrelations are possible and inevitable. Chapter 1 serves as an introduction to the research question and contextualises the inquiry in regard to my own personal and professional background. Chapter 2 details relevant concepts, scholarship, performance practice and cultural context and serves to place the work in a lineage of other practice. Chapter 3 describes, documents and interrogates the research practice, including inspirations and experiments alongside the final works. Chapter 4 conceptualises the practice within a phenomenological framework, analysing contemporary communications technologies as part of an expanding perceptual toolset with which we co-shape our reality and placing technical infrastructure within a framework of late capitalism. The final chapter concludes the complimentary writing and clearly enumerates the findings

    SPICA:revealing the hearts of galaxies and forming planetary systems : approach and US contributions

    Get PDF
    How did the diversity of galaxies we see in the modern Universe come to be? When and where did stars within them forge the heavy elements that give rise to the complex chemistry of life? How do planetary systems, the Universe's home for life, emerge from interstellar material? Answering these questions requires techniques that penetrate dust to reveal the detailed contents and processes in obscured regions. The ESA-JAXA Space Infrared Telescope for Cosmology and Astrophysics (SPICA) mission is designed for this, with a focus on sensitive spectroscopy in the 12 to 230 micron range. SPICA offers massive sensitivity improvements with its 2.5-meter primary mirror actively cooled to below 8 K. SPICA one of 3 candidates for the ESA's Cosmic Visions M5 mission, and JAXA has is committed to their portion of the collaboration. ESA will provide the silicon-carbide telescope, science instrument assembly, satellite integration and testing, and the spacecraft bus. JAXA will provide the passive and active cooling system (supporting the

    The Apertif Surveys:The First Six Months

    Get PDF
    Apertif is a new phased-array feed for the Westerbork Synthesis Radio Telescope (WSRT), greatly increasing its field of view and turning it into a natural survey instrument. In July 2019, the Apertif legacy surveys commenced; these are a time-domain survey and a two-tiered imaging survey, with a shallow and medium-deep component. The time-domain survey searches for new (millisecond) pulsars and fast radio bursts (FRBs). The imaging surveys provide neutral hydrogen (HI), radio continuum and polarization data products. With a bandwidth of 300 MHz, Apertif can detect HI out to a redshift of 0.26. The key science goals to be accomplished by Apertif include localization of FRBs (including real-time public alerts), the role of environment and interaction on galaxy properties and gas removal, finding the smallest galaxies, connecting cold gas to AGN, understanding the faint radio population, and studying magnetic fields in galaxies. After a proprietary period, survey data products will be publicly available through the Apertif Long Term Archive (ALTA, https://alta.astron.nl). I will review the progress of the surveys and present the first results from the Apertif surveys, including highlighting the currently available public data
    corecore