The ATLAS EventIndex: a BigData catalogue for all ATLAS experiment events

The ATLAS EventIndex system comprises the catalogue of all events collected, processed or generated by the ATLAS experiment at the CERN LHC accelerator, and all associated software tools to collect, store and query this information. ATLAS records several billion particle interactions every year of operation, processes them for analysis and generates even larger simulated data samples; a global catalogue is needed to keep track of the location of each event record and be able to search and retrieve specific events for in-depth investigations. Each EventIndex record includes summary information on the event itself and the pointers to the files containing the full event. Most components of the EventIndex system are implemented using BigData open-source tools. This paper describes the architectural choices and their evolution in time, as well as the past, current and foreseen future implementations of all EventIndex components.Comment: 21 page

A Roadmap for HEP Software and Computing R&D for the 2020s

Particle physics has an ambitious and broad experimental programme for the coming decades. This programme requires large investments in detector hardware, either to build new facilities and experiments, or to upgrade existing ones. Similarly, it requires commensurate investment in the R&D of software to acquire, manage, process, and analyse the shear amounts of data to be recorded. In planning for the HL-LHC in particular, it is critical that all of the collaborating stakeholders agree on the software goals and priorities, and that the efforts complement each other. In this spirit, this white paper describes the R&D activities required to prepare for this software upgrade.Peer reviewe

IT Lightning Talks: session #18

How can a new database system which manages billions of rows with relatively short lifetime (weeks to months) be designed ? The system should resemble a Whiteboard: - data, grouped by given properties form data collections, are added when requested - data collections are consumed by the requestor - when not needed any more, data collections are removed by a "Whiteboard sponge" process. I will show you what options I explored and which one is potentially the best in terms of efficiency

The one-million table partitions challenge in an ATLAS experiment DB application @ CERN

The presentation describes the challenges in designing a new DB system in the ATLAS experiment at CERN which needs to manage billions of rows with relatively short lifetime (weeks to months). For that system an approach of data grouping organised in table partitions might be appropriate. In the past years in ATLAS we made use of several partitioning techniques. With the broadly used range partitioning and its extension of automatic interval partitioning we add our own logic in PLSQL procedures and scheduler jobs to sustain data sliding windows in order to enforce various data retention policies. We also make use of list, reference and virtual column based partitioning. One of our tables in the production DB has 74000+ list partitions (180+ billion rows), other has 20000+ list sub-partitions. However, for a first time we challenge ourselves with the one-million partitions limit per table in the database. What options we have and which one is most suitable for our new use case will be presented to the audience

The importance of having an appropriate relational data segmentation in ATLAS

In this paper we describe specific technical solutions put in place in various database applications of the ATLAS experiment at LHC where we make use of several partitioning techniques available in Oracle 11g. With the broadly used range partitioning and its option of automatic interval partitioning we add our own logic in PLSQL procedures and scheduler jobs to sustain data sliding windows in order to enforce various data retention policies. We also make use of the new Oracle 11g reference partitioning in the Nightly Build System to achieve uniform data segmentation. However the most challenging issue was to segment the data of the new ATLAS Distributed Data Management system (Rucio), which resulted in tens of thousands list type partitions and sub-partitions. Partition and sub-partition management, index strategy, statistics gathering and queries execution plan stability are important factors when choosing an appropriate physical model for the application data management. The so-far accumulated knowledge and analysis on the new Oracle 12c version features that could be beneficial will be shared with the audience

The importance of applying an appropriate data partitioning

In this presentation are described specific technical solutions put in place in various database applications of the ATLAS experiment at LHC where we make use of several partitioning techniques available in Oracle 11g. With the broadly used range partitioning and its option of automatic interval partitioning we add our own logic in PLSQL procedures and scheduler jobs to sustain data sliding windows in order to enforce various data retention policies. We also make use of the new Oracle 11g reference partitioning in the ATLAS Nightly Build System to achieve uniform data segmentation. However the most challenging was to segment the data of the new ATLAS Distributed Data Management system, which resulted in tens of thousands list type partitions and sub-partitions. Partition and sub-partition management, index strategy, statistics gathering and queries execution plan stability are important factors when choosing an appropriate physical model for the application data management. The so-far accumulated knowledge will be shared with the audience

The one-million table partitions challenge in an ATLAS experiment DB application @ CERN

The presentation describes the challenges in designing a new DB system in the ATLAS experiment at CERN which needs to manage billions of rows with relatively short lifetime (weeks to months). For that system an approach of data grouping organised in table partitions might be appropriate. In the past years in ATLAS we made use of several partitioning techniques. With the broadly used range partitioning and its extension of automatic interval partitioning we add our own logic in PLSQL procedures and scheduler jobs to sustain data sliding windows in order to enforce various data retention policies. We also make use of list, reference and virtual column based partitioning. Some of our tables have 70000+ list partitions, others have 20000+ list sub-partitions. However for a first time we challenge ourselves with the one-million partitions limit per table in the database. What choices(options) we have and which one is potentially most suitable will be presented to the audience

The importance of having an appropriate data segmentation (partitioning)

In this presentation will be shown real life examples from database applications in the ATLAS experiment @ LHC where we make use of many Oracle partitioning techniques available in Oracle 11g. With the broadly used range partitioning and its option of automatic interval partitioning we add our own logic in PLSQL for sustaining data sliding windows in order to enforce various data retention policies. We also make use of the reference partitioning in some use cases, however the most challenging was to segment the data of a large bookkeeping system which resulted in tens of thousands list partitions and list sub-partitions. Partition and sub-partition management, index strategy, statistics gathering and queries execution plan stability are important factors when choosing an appropriate for the use case data management model. The gained experience with all of those will be shared with the audience

Evaluation of the ATLAS model for remote access to database resident information for LHC Run 3

The ATLAS model for remote access to database resident information relies upon a limited set of dedicated and distributed Oracle database repositories complemented with the deployment of Frontier system infrastructure on the WLCG. ATLAS clients with network access can get the database information they need dynamically by submitting requests to a squid server in the Frontier network which provides results from its cache or passes new requests along the network to launchpads co-located at one of the Oracle sites (the master Oracle database at CERN or one of the Tier 1 Oracle database replicas). Since the beginning of LHC Run 1, the system has evolved in terms of client, squid, and launchpad optimizations but the distribution model has remained fundamentally unchanged. On the whole, the system has been broadly successful in providing data to clients with relatively few disruptions even while site databases were down due to redundancy overall. At the same time, its quantitative performance characteristics, such as the global throughput of the system, the load distribution between sites, and the constituent interactions that make up the whole, were largely unknown. But more recently, information has been collected from launchpad and squid logs into an Elastic Search repository which has enabled a wide variety of studies of various aspects of the system. This presentation will describe dedicated studies of the data collected in Elastic Search over the previous year to evaluate the efficacy of the distribution model. Specifically, we will quantify any advantages that the redundancy of the system offers as well as related aspects such as the geographical dependence of wait times seen by clients in getting a response to its requests. These studies are essential so that during LS2 (the long shutdown between LHC Run 2 and Run 3), we can adapt the system in preparation for the expected increase in the system load in the ramp up to Run 3 operations