On the Impact of Memory Allocation on High-Performance Query Processing

Somewhat surprisingly, the behavior of analytical query engines is crucially affected by the dynamic memory allocator used. Memory allocators highly influence performance, scalability, memory efficiency and memory fairness to other processes. In this work, we provide the first comprehensive experimental analysis on the impact of memory allocation for high-performance query engines. We test five state-of-the-art dynamic memory allocators and discuss their strengths and weaknesses within our DBMS. The right allocator can increase the performance of TPC-DS (SF 100) by 2.7x on a 4-socket Intel Xeon server

A Memory Contention Responsive Hash Join Algorithm Design and Implementation on Apache AsterixDB

Efficient data management is crucial in complex computer systems, and Database Management Systems (DBMS) are indispensable for handling and processing large datasets. In DBMSs that concurrently execute multiple queries, adapting to varying workloads is desirable. Yet, predicting the fluctuating quantity and size of queries in such environments proves challenging. Over-allocating resources to a single query can impede the execution of future queries while under-allocating resources to a query expecting increased workload can lead to significant processing delays. Moreover, join operations place substantial demands on memory. This resource’s availability fluctuates as queries enter and exit the DBMS. The development of join operators capable of dynamically adapting to memory fluctuations is a complex undertaking, with few recent authors proposing memory-adaptive algorithms. This scarcity of proposals suggests the inherent difficulty in designing, implementing, and analyzing such algorithms. This thesis proposes a new memory adaptive Hash-Based join algorithm extended from designs presented by prior authors. This algorithm is implemented and experimented with in a real DBMS environment to evaluate its memory fluctuation responsiveness. A mathematical model for the increase in I/O caused by it is proposed and compared with actual results. The impacts of memory variation and frequence of memory updates reveal the importance of this thesis for further development of memory adaptive algorithms

Analisis Performansi pada In-Memory Database

ABSTRAKSI: Performansi DBMS merupakan salah satu hal yang dijadikan pertimbangan dalam pemilihan sebuah DBMS. Dalam era informasi ini, peranan DBMS sebagai perangkat lunak penyimpan dan pengelola basis data akan semakin terasa dan semakin sering bersinggungan dengan kehidupan manusia. Pada era selanjutnya dimana pertukaran informasi sangat cepat, performansi DBMS menjadi suatu hal yang mutlak diperlukan untuk menjamin ketepatan dan kecepatan penyampaian informasi Untuk menjawab tantangan akan performansi tersebut, munculah sebuah paradigma baru yang disebut dengan In-Momory Database. Dengan paradigma baru ini, diharapkan permasalahan yang berkait dengan performansi DBMS dapat diselesaikan. Melalui karakteristiknya menyimpan seluruh data yang ada di memory komputer dan menjadikan memory komputer sebagai tempat penyimpanan data utama, secara teoritis, In-Memory Database ini memiliki performansi yang lebih cepat daripada hanya menggunakan konvensional DBMS. Namun, seperti layaknya sebuah system, keunggulan yang diharapkan dari In- Momory Database ini memiliki batasan-batasan dimana batasan-batasan ini dapat membuat peningkatan performansi DBMS menjadi tidak signifikan dan tidak dapat mencapai tujuan seperti yang diinginkan.Kata Kunci : In-Momory Database, performansi, DBMS, basis dataABSTRACT: DBMS performance is one of many thing that have to be considered in choosing a DBMS. In this information era, the role of DBMS as a software which store and manage the data will be more and will be more related in the human life. In the next era where the information exchange repidly, the high performance DBMS become important to guarantee the correctness and the speed of information exchange. To fullfill the performance challenge, a new paradigm which is called In- Memory Database arise. With this new paradigm, hopefully, the problem of DBMS performance will be solve. With it characteristic store the data in computer memory (RAM) and using the computer memory as the main data store, toeretically, In-Memory Database performance better than regular DBMS. Although has a high performance, In-Memory Database has it constraint that can make the In-Memory database performance become decrease and the people who use this In-Memory database become unsatisfy.Keyword: In-Momory Database, performance, DBMS, databas

Implications of non-volatile memory as primary storage for database management systems

Traditional Database Management System (DBMS) software relies on hard disks for storing relational data. Hard disks are cheap, persistent, and offer huge storage capacities. However, data retrieval latency for hard disks is extremely high. To hide this latency, DRAM is used as an intermediate storage. DRAM is significantly faster than disk, but deployed in smaller capacities due to cost and power constraints, and without the necessary persistency feature that disks have. Non-Volatile Memory (NVM) is an emerging storage class technology which promises the best of both worlds. It can offer large storage capacities, due to better scaling and cost metrics than DRAM, and is non-volatile (persistent) like hard disks. At the same time, its data retrieval time is much lower than that of hard disks and it is also byte-addressable like DRAM. In this paper, we explore the implications of employing NVM as primary storage for DBMS. In other words, we investigate the modifications necessary to be applied on a traditional relational DBMS to take advantage of NVM features. As a case study, we have modified the storage engine (SE) of PostgreSQL enabling efficient use of NVM hardware. We detail the necessary changes and challenges such modifications entail and evaluate them using a comprehensive emulation platform. Results indicate that our modified SE reduces query execution time by up to 40% and 14.4% when compared to disk and NVM storage, with average reductions of 20.5% and 4.5%, respectively.The research leading to these results has received funding from the European Union’s 7th Framework Programme under grant agreement number 318633, the Ministry of Science and Technology of Spain under contract TIN2015-65316-P, and a HiPEAC collaboration grant awarded to Naveed Ul Mustafa.Peer ReviewedPostprint (author's final draft

Improving the Deductive System DES with Persistence by Using SQL DBMS's

This work presents how persistent predicates have been included in the in-memory deductive system DES by relying on external SQL database management systems. We introduce how persistence is supported from a user-point of view and the possible applications the system opens up, as the deductive expressive power is projected to relational databases. Also, we describe how it is possible to intermix computations of the deductive engine and the external database, explaining its implementation and some optimizations. Finally, a performance analysis is undertaken, comparing the system with current relational database systems.Comment: In Proceedings PROLE 2014, arXiv:1501.0169

Challenging Ubiquitous Inverted Files

Stand-alone ranking systems based on highly optimized inverted file structures are generally considered ‘the’ solution for building search engines. Observing various developments in software and hardware, we argue however that IR research faces a complex engineering problem in the quest for more flexible yet efficient retrieval systems. We propose to base the development of retrieval systems on ‘the database approach’: mapping high-level declarative specifications of the retrieval process into efficient query plans. We present the Mirror DBMS as a prototype implementation of a retrieval system based on this approach