The ATLAS EventIndex: a BigData catalogue for all ATLAS experiment events

The ATLAS EventIndex system comprises the catalogue of all events collected, processed or generated by the ATLAS experiment at the CERN LHC accelerator, and all associated software tools to collect, store and query this information. ATLAS records several billion particle interactions every year of operation, processes them for analysis and generates even larger simulated data samples; a global catalogue is needed to keep track of the location of each event record and be able to search and retrieve specific events for in-depth investigations. Each EventIndex record includes summary information on the event itself and the pointers to the files containing the full event. Most components of the EventIndex system are implemented using BigData open-source tools. This paper describes the architectural choices and their evolution in time, as well as the past, current and foreseen future implementations of all EventIndex components.Comment: 21 page

BigData tools for the monitoring of the ATLAS EventIndex

The ATLAS EventIndex collects event information from data both at CERN and Grid sites. It uses the Hadoop system to store the results, and web services to access them. Its successful operation depends on a number of different components, that have to be monitored constantly to ensure continuous operation of the system. Each component has completely different sets of parameters and states and requires a special approach. A scheduler runs monitoring tasks, which gather information by various methods: querying databases, web sites and storage systems, parsing logs and using CERN host monitoring services. Information is then fed to Grafana dashboards via InfluxDB. Using this platform allowed much faster performance and flexibility compared to the previously used Kibana system

BigData tools for the monitoring of the ATLAS EventIndex

The ATLAS EventIndex collects event information from data both at CERN and Grid sites. It uses the Hadoop system to store the results, and web services to access them. Its successful operation depends on a number of different components, that have to be monitored constantly to ensure continuous operation of the system. Each component has completely different sets of parameters and states and requires a special approach. A scheduler runs monitoring tasks, which gather information by various methods: querying databases, web sites and storage systems, parsing logs and using CERN host monitoring services. Information is then fed to Grafana dashboards via InfluxDB. Using this platform allowed much faster performance and flexibility compared to the previously used Kibana system

The Controls and Configuration Software of the ATLAS Data Acquisition System: evolution towards LHC Run 3

The ATLAS experiment at the Large Hadron Collider (LHC) operated very successfully in the years 2008 to 2018, in two periods identified as Run 1 and Run 2. ATLAS achieved an overall data-taking efficiency of 94%, largely constrained by the irreducible dead-time introduced to accommodate the limitations of the detector read-out electronics. Out of the 6% dead-time only about 15% could be attributed to the central trigger and DAQ system, and out of these, a negligible fraction was due to the Control and Configuration subsystem. Despite these achievements, and in order to improve even more the already excellent efficiency of the whole DAQ system in the coming Run 3, a new campaign of software updates was launched for the second long LHC shutdown (LS2). This paper presents, using a few selected examples, how the work was approached and which new technologies were introduced into the ATLAS Control and Configuration software. Despite these being specific to this system, many solutions can be considered and adapted to different distributed DAQ systems

THE CONTROLS AND CONFIGURATION SOFTWARE OF THE ATLAS DATA ACQUISITION SYSTEM FOR LHC RUN 2

The ATLAS experiment at the Large Hadron Collider (LHC) operated very successfully in the years 2008 to 2013, identified as Run 1. It achieved an overall data taking efficiency of 94%, largely constrained by the irreducible dead-time introduced to accommodate the limitations of the detector read-out electronics. Out of the 6% dead-time only about 15% could be attributed to the central trigger and DAQ system, and out of these, a negligible fraction was due to the Control and Configuration sub-system. Despite these achievements, and in order to improve the efficiency of the whole DAQ system in Run 2 (2015-2018), the first long LHC shutdown (2013-2014) was used to carry out a complete revision of the control and configuration software. The goals were three-fold: properly accommodate additional requirements that could not be seamlessly included during steady operation of the system; re-factor software that had been repeatedly modified to include new features, thus becoming less maintainable; seize the opportunity of modernizing software written at the beginning of the years 2000, thus profiting from the rapid evolution in IT technologies. This upgrade was carried out retaining the important constraint of minimally impacting the mode of operation of the system and public APIs, in order to maximize the acceptance of the changes by the large user community. This paper presents, using a few selected examples, how the work was approached and which new technologies were introduced into the ATLAS DAQ system, and how they were performing in course of Run 2. Despite these being specific to this system, many solutions can be considered and adapted to different distributed DAQ systems

Control and Configuration Software for the ATLAS DAQ system in LHC Run 2

The ATLAS experiment at the Large Hadron Collider (LHC) operated successfully from 2008 to 2018, which included Run 1 (2008-2013), a shutdown period and the Run 2 (2016-2018). In the course of the Run 2, the ATLAS data taking achieved an overall data taking efficiency of 97%, largely constraint by the irreducible dead-time introduced to accommodate the limitations of the detector read-out electronics. Less than 1% of the dead-time could be attributed to the central trigger and DAQ system, and out of these, a negligible fraction was due to the Controls and Configuration sub-system. The first long LHC shutdown (LS1) (2014-2015) was used to carry out a complete revision of the Controls and Configuration software, in order to suitably accommodate additional requirements that could not be seamlessly included during steady operation of the system. As well a refactorization of the software was carried out, software that had been repeatedly modified to include new features becoming less maintainable. Additionally, LS1 was the opportunity of modernizing software written at the beginning of the years 2000, thus profiting from the rapid evolution in IT technologies. This upgrade was carried out retaining the critical constraint of minimally impacting public APIs, and the operation mode of the system, in order to maximize the acceptance of the changes by the large user community. This paper summarizes and illustrates, at hand of a few selected examples, how the work was approached and which new technologies were introduced into the ATLAS DAQ system and were used in the course of the LHC Run 2. Despite these being specific to the system, many solutions can be considered and adapted to different distributed DAQ systems. Additionally, this paper will focus on the behavior of the Controls and Configuration services through the whole Run 2 period, putting particular emphasis on robustness, reliability and performance matters

Data-centric Graphical User Interface of the ATLAS Event Index Service

The Event Index service of the ATLAS experiment at the LHC keeps references to all real and simulated events. Hadoop Map files and HBase tables are used to store the Event Index data, a subset of data is also stored in the Oracle database. Several user interfaces are currently used to access and search the data, from a simple command line interface, through a programmable API, to sophisticated graphical web services. It provides a dynamic graph-like overview of all available data (and data collections). Data are shown together with their relations, like paternity or overlaps. Each data entity then gives users a set of actions available for the referenced data. Some actions are provided directly by the Event Index system, others are just interfaces to different ATLAS services. In many cases, specialized views are offered for detailed data inspection, such as histograms, Venn diagrams, etc. This paper documents the current status of the service, its features and performance. The future system evolution to the new Event Index architecture based on the Apache Phoenix is also described as well as possible extension to a more general framework for giving a new, more intuitive access to experiment data

The ATLAS Eventindex Using the HBase/Phoenix Storage Solution

International audienceThe ATLAS EventIndex provides a global event catalogue and event-level metadata for ATLASanalysis groups and users. The LHC Run 3, starting in 2022, will see increased data-taking andsimulation production rates, with which the current infrastructure would still cope but may bestretched to its limits by the end of Run 3. This talk describes the implementation of a new corestorage service that will provide at least the same functionality as the current one for increased dataingestion and search rates, and with increasing volumes of stored data. It is based on a set of HBasetables, coupled to Apache Phoenix for data access; in this way we will add to the advantages of aBigData based storage system the possibility of SQL as well as NoSQL data access, which allows there-use of most of the existing code for metadata integration

The ATLAS EventIndex for LHC Run 3

International audienceThe ATLAS EventIndex was designed in 2012-2013 to provide a global event catalogue and limited event-level metadata for ATLAS analysis groups and users during the LHC Run 2 (2015-2018). It provides a good and reliable service for the initial use cases (mainly event picking) and several additional ones, such as production consistency checks, duplicate event detection and measurements of the overlaps of trigger chains and derivation datasets. The LHC Run 3, starting in 2021, will see increased data-taking and simulation production rates, with which the current infrastructure would still cope but may be stretched to its limits by the end of Run 3. This proceeding describes the implementation of a new core storage service that will be able to provide at least the same functionality as the current one for increased data ingestion and search rates, and with increasing volumes of stored data. It is based on a set of HBase tables, with schemas derived from the current Oracle implementation, coupled to Apache Phoenix for data access; in this way we will add to the advantages of a BigData based storage system the possibility of SQL as well as NoSQL data access, allowing to re-use most of the existing code for metadata integration

Data-centric Graphical User Interface of the ATLAS Event Index Service

International audienceThe Event Index service of the ATLAS experiment at the LHC keeps references to all real and simulated events. Hadoop Map files and HBase tables are used to store the Event Index data, a subset of data is also stored in the Oracle database. Several user interfaces are currently used to access and search the data, from a simple command line interface, through a programmable API, to sophisticated graphical web services. It provides a dynamic graph-like overview of all available data (and data collections). Data are shown together with their relations, like paternity or overlaps. Each data entity then gives users a set of actions available for the referenced data. Some actions are provided directly by the Event Index system, others are just interfaces to different ATLAS services. In many cases, specialized views are offered for detailed data inspection, such as histograms, Venn diagrams, etc.This paper documents the current status of the service, its features and performance. The future system evolution to the new Event Index architecture based on the Apache Phoenix is also described as well as possible extension to a more general framework for giving a new, more intuitive access to experiment data