9 research outputs found
Here are my data files. Here are my queries. Where are my results?
Database management systems (DBMS) provide incredible flexibility and performance
when it comes to query processing, scalability and accuracy.
To fully exploit DBMS features, however, the user must \emph{define} a schema,
\emph{load} the data, \emph{tune} the system for the expected workload, and answer several questions.
Should the database use a column-store, a row-store or some hybrid format?
What indices should be created?
All these questions make for a formidable and time-consuming hurdle, often
deterring new applications or imposing high cost to existing ones.
A characteristic example is that of scientific databases with huge data sets.
The prohibitive initialization cost and complexity
still forces scientists to rely on ``ancient" tools for their data management tasks,
delaying scientific understanding and progress.
Users and applications collect their data in flat files, which have traditionally been
considered to be ``outside" a DBMS. A DBMS wants control:
always bring all data ``inside", replicate it and format it in its own ``secret" way.
The problem has been recognized and current efforts extend existing systems with
abilities such as reading information from flat files
and gracefully incorporating it into the processing engine.
This paper proposes a new generation of systems where
the only requirement from the user is \emph{a link to the raw data files}.
Queries can then immediately be fired without preparation steps in between.
Internally and in an abstract way, the system takes care of
selectively, adaptively and incrementally providing the proper environment
given the queries at hand. Only part of the data is loaded at any given time
and it is being stored and accessed in the format suitable for the current workload