Using Graph Databases

Data in High Energy Physics (HEP) usually consist of complext complex data structures stored in relational databases and files with internal schema. Such architecture exhibits many shortcomings, which could be fixed by migrating into Graph Database storage. The paper describes basic principles of the Graph Database together with an overview of existing standards and implementations. The usefulness and usability are demonstrated using the concrete example of the Event Index of the ATLAS experiment at LHC in two approaches as the full storage (all data are in the Graph Database) and meta-storage (a layer of schema-less graph-like data implemented on top of more traditional storage). The usability, the interfaces with the surrounding framework and the performance of those solutions are discussed. The possible more general usefulness for generic experiments’ storage is also discussed

Multilanguage Frameworks Towards the Ideal Multilanguage Environment

International audienceHigh Energy Physics software has been a victim of the necessity to choose one implementation language as no really usable multi-language environment existed. Even a co-existence of two languages in the same framework (typically C++ and Python) imposes a heavy burden on the system. The role of different languages was generally limited to well encapsulated domains (like Web applications, databases, graphics), with very limited connection to the central framework.The new development in the domain of the compilers and run-time environments has enabled ways for creating really multilanguage frameworks, with seamless, user-friendly and high-performance inter-operation of many languages, which traditionally live in disconnected domains (like C-based languages vs JVM languages or Web languages).Various possibilities and strategies for creation of the true multi-language frameworks are discussed, emphasizing their advantages and possible road blocks

Multidatabase The Future of the data storage in Particle Physics and Astronomy

International audienceParticle Physics experiments and Astronomy telescopes store alarge amount of data, most of those data are usually stored in simple files in various formats. Databases are used only to a limited extend, while database technology has made a tremendous progress in recent years, offering a large spectrum of applications in all possible domains. Those possibilities are still largely underused.The article demonstrates the data architecture transparently using multiple database types, namely SQL database, NoSQL database and Graph database. Each database is used for the domain where it is the most appropriate and the cross-technology access is offered to users. The strong and weak points of all three technologies are discussed. Several possible ways of organizing interactionbetweendatabases aredescribed.The architecture is illustrated on the concrete implementation of the data storage oftheFinkproject, usingJanusGraph andHBasetechnologiesto store alerts from theRubinObservatory. HBaseis used to store thebulkdata and the structuralinformationis storedinGraphs, allowing transparent access to alldata and deep analyses of relationbetween alerts and other objects. Ways for implementing heterogeneous database architecture in general HEP applications are outlined

The ATLAS Event Index: The Architecture of the Core Engine

International audienceThe global view of the ATLAS Event Index system has been presented in the 17th ACAT Conference. This article concentrates on the architecture of the system core component. This component handles the final stage of the event metadata import. It organizes its storage and provides a fast and feature-rich access to all information. A user is able to interrogate metadata in various ways, including by executing user-provided code on the data to make selections and to interpret the results. A wide spectrum of clients is available, from a set of Linux-like commands to an interactive graphical Web Service. The stored event metadata contain the basic description of the related events, the references to the experiment event storage and the full trigger record and can be extended with other event characteristics. Derived collections of events can be created. Such collections can be annotated and tagged with further information

Data-centric Graphical User Interface of the ATLAS Event Index Service

The Event Index service of the ATLAS experiment at the LHC keeps references to all real and simulated events. Hadoop Map files and HBase tables are used to store the Event Index data, a subset of data is also stored in the Oracle database. Several user interfaces are currently used to access and search the data, from a simple command line interface, through a programmable API, to sophisticated graphical web services. It provides a dynamic graph-like overview of all available data (and data collections). Data are shown together with their relations, like paternity or overlaps. Each data entity then gives users a set of actions available for the referenced data. Some actions are provided directly by the Event Index system, others are just interfaces to different ATLAS services. In many cases, specialized views are offered for detailed data inspection, such as histograms, Venn diagrams, etc. This paper documents the current status of the service, its features and performance. The future system evolution to the new Event Index architecture based on the Apache Phoenix is also described as well as possible extension to a more general framework for giving a new, more intuitive access to experiment data