Kersten, M.

Manegold, S.

English

The ability to explore huge digital resources assembled in data warehouses, databases and files, at unprecedented speed, is becoming the driver of progress in science. However, existing database management systems (DBMS) are far from capable of meeting the scientists' requirements. The Database Architectures group at CWI in Amsterdam cooperates with astronomers, seismologists and other domain experts to tackle this challenge by advancing all aspects of database technology. The group’s research results are disseminated via its open-source database system, MonetDB

International Migration, Integration and Social Cohesion online publications

UvA-DARE is a service provided by the library of the University of Amsterdam (https://dare.uva.nl)UvA-DARE (Digital Academic Repository)Revolutionary Database Technology for Data Intensive ResearchKersten, M.; Manegold, S.Publication date2012Document VersionFinal published versionPublished inERCIM NewsLink to publicationCitation for published version (APA):Kersten, M., & Manegold, S. (2012). Revolutionary Database Technology for Data IntensiveResearch. ERCIM News, 89, 16. http://ercim-news.ercim.eu/en89/special/revolutionary-database-technology-for-data-intensive-researchGeneral rightsIt is not permitted to download or to forward/distribute the text or part of it without the consent of the author(s)and/or copyright holder(s), other than for strictly personal, individual use, unless the work is under an opencontent license (like Creative Commons).Disclaimer/Complaints regulationsIf you believe that digital publication of certain material infringes any of your rights or (privacy) interests, pleaselet the Library know, stating your reasons. In case of a legitimate complaint, the Library will make the materialinaccessible and/or remove it from the website. Please Ask the Library: https://uba.uva.nl/en/contact, or a letterto: Library of the University of Amsterdam, Secretariat, Singel 425, 1012 WP Amsterdam, The Netherlands. Youwill be contacted as soon as possible.Download date:10 Mar 2023ERCIM NEWS 89   April 201216Special Theme: Big Data The heart of a scientific data warehouseis its database system, running on amodern distributed platform, and usedfor both direct interaction with datagathered from experimental devicesand management of the derived knowl-edge using workflow software.However, most (commercial) DBMSofferings cannot fulfill the demandingneeds of scientific data management.They fall short in one or more of thefollowing areas: multi-paradigm datamodels (including support for arrays),transparent data ingestion from, andseamless integration of, scientific filerepositories, complex event processing,and provenance. These topics onlyscratch the surface of the problem. Thestate of the art in scientific data explo-ration can be compared with our dailyuse of search engines. For a large part,search engines rely on guiding the userfrom their ill-phrased queries throughsuccessive refinement to the informa-tion of interest. Limited a priori knowl-edge is required. The sample answersreturned provide guidance to drilldown, chasing individual links, or toadjust the query terms.The situation in scientific databases ismore cumbersome than searching fortext, because they often contain complexobservational data, eg telescope imagesof the sky, satellite images of the earth,time series or seismograms, and little apriori knowledge exists. The prime chal-lenge is to find models that capture theessence of this data at both a macro- andmicro-scale. The answer is in the data-base, but the ‘Nobel-winning query’ isstill unknown.Next generation database managementengines should provide a much richerrepertoire and ease of use experience tocope with the deluge of observationaldata in a resource-limited setting. Goodis good enough as an answer, providedthe journey can be continued as long asthe user remains interested.We envision seven directions of longterm research in database technology:• Data Vaults. Scientific data is usuallyavailable in self-descriptive file for-mats as produced by advanced scien-tific instruments. The need to convertthese formats into relational tablesand to explicitly load all data into theDBMS forms a major hurdle for data-base-supported scientific data analy-sis. Instead, we propose the datavault, a database-attached externalfile repository. The data vault createsa true symbiosis between a DBMSand existing file-based repositories,and thus provides transparent accessto all data kept in the repositorythrough the DBMS’s (array-based)query language.• Array support. Scientific data man-agement calls for DBMSs that inte-grate the genuine scientific datamodel, multi-dimensional arrays, asfirst-class citizens next to relationaltables, and unified declarative lan-guage as symbiosis of relational andlinear algebra. Such support needs tobe beyond ‘alien’ extensions that pro-vide user defined functions. Rather,arrays need to become first-classDBMS citizens next to relationaltables.• One-minute database kernels. Such akernel differs from conventional ker-nels by identifying and avoiding per-formance degradation by answeringqueries only partly within strict timebounds. Run the query during a coffeebreak, look at the result, and continueor abandon the data exploration path.• Multi-scale query processing. Fastexploration of large datasets calls forpartitioning the database based onscience interest and resource avail-ability. It extends traditional parti-tioning schemes by taking intoaccount the areas of users’ interestand the statistical stability in samplesdrawn from the archives.• Post-processing result sets. The oftenhuge results returned should not bethrown at the user directly, but passedthrough an analytical processingpipeline to condense the informationfor human consumption. Thisinvolves computation intensive datamining techniques and harnessing thepower of GPUs in the software stackof a DBMS.• Query morphing. Given the impreci-sion of the queries, the system shouldaid in hinting at proximity resultsusing data distributions looked uponduring query evaluation. For exam-ple, aside from the traditional row set,it may suggest minor changes to thequery predicates to obtain non-emptyresults. The interesting data may be‘just around the corner’.• Queries as answers. Standing on theshoulders of your peers involveskeeping track of the queries, theirresults, and resource requirements. Itcan be used as advice to modify ill-phrased queries that could run forhours producing meaningless results.Please contact: Stefan ManegoldCWI, The NetherlandsE-mail: Stefan.Manegold@cwi.nlRevolutionary Database Technology for Data Intensive Researchby Martin Kersten and Stefan ManegoldThe ability to explore huge digital resources assembled in data warehouses, databases and files, atunprecedented speed, is becoming the driver of progress in science. However, existing databasemanagement systems (DBMS) are far from capable of meeting the scientists’ requirements. TheDatabase Architectures group at CWI in Amsterdam cooperates with astronomers, seismologistsand other domain experts to tackle this challenge by advancing all aspects of database technology.The group’s research results are disseminated via its open-source database system, MonetDB.

Revolutionary Database Technology for Data Intensive Research

https://pure.uva.nl/ws/files/1617498/159347_380003.pdf

Revolutionary Database Technology for Data Intensive Research

Abstract

Similar works

Full text

Available Versions

International Migration, Integration and Social Cohesion online publications