1 research outputs found
Toward real-time data query systems in HEP
Exploratory data analysis tools must respond quickly to a user's questions,
so that the answer to one question (e.g. a visualized histogram or fit) can
influence the next. In some SQL-based query systems used in industry, even very
large (petabyte) datasets can be summarized on a human timescale (seconds),
employing techniques such as columnar data representation, caching, indexing,
and code generation/JIT-compilation. This article describes progress toward
realizing such a system for High Energy Physics (HEP), focusing on the
intermediate problems of optimizing data access and calculations for "query
sized" payloads, such as a single histogram or group of histograms, rather than
large reconstruction or data-skimming jobs. These techniques include direct
extraction of ROOT TBranches into Numpy arrays and compilation of Python
analysis functions (rather than SQL) to be executed very quickly. We will also
discuss the problem of caching and actively delivering jobs to worker nodes
that have the necessary input data preloaded in cache. All of these pieces of
the larger solution are available as standalone GitHub repositories, and could
be used in current analyses.Comment: 6 pages, 2 figures, proceedings for ACAT 201