The Trigger and Data Acquisition system for the ATLAS experiment has to reduce the 40 MHz of LHC bunch crossing rate to ~200 Hz of recording rate. This is achieved through a complex distributed system composed by $sim$ 1.000 CPUs, about a third of the expected final size of the system. Monitoring the trigger behavior through all the trigger level is of fundamental importance to assess the quality of the data taken, to give fast feedback for the trigger configuration design and to monitor the stability of the HLT farm components. In this paper we will present the online monitoring framework and the various tools available in the ATLAS trigger system going from the ones that build the basic monitoring infrastructure and test the basic functionalities of the system to the more elaborated ones that checks the quality of the data taking looking at physics variables reconstructed online. The early experience in the 2009 cosmics data taking period will also be shown

Sidoti, A

CERN Document Server

ATL-DAQ-PROC-2010-00212January2010December 30, 2009 21:53 WSPC - Proceedings Trim Size: 9in x 6in ICATTP˙proceedings˙sidoti˙21TRIGGER MONITORINGIN ATLASA. SIDOTI∗ for the ATLAS TDAQ Collaboration†Humboldt Universita¨t zu Berlin,15, NewtonStraße,12489, Berlin, GermanyE-mail: antonio.sidoti@cern.chThe Trigger and Data Acquisition system for the ATLAS experiment has toreduce the 40 MHz of LHC bunch crossing rate (1 GHz of interaction rate atL = 1034cm2s−1) to ∼ 200 Hz of recording rate. This is achieved througha complex distributed system realized by an hardware implemented Level-1system, followed by a farm of PCs for the High Level Trigger selection, whichis performed in two steps: Level-2 and Event Filter.Monitoring the trigger behavior and its performances through all the trig-ger level is of fundamental importance to assess the quality of the data taken,to give fast feedback for the trigger configuration, to monitor the stability ofthe HLT farm components.In this paper the online monitoring tools of the ATLAS trigger system willbe presented starting with the most basic ones that monitor the technical func-tionalities of the system as for example the trigger rates. More sophisticatedtools to assess the data quality of the selected physics objects are shown too.First experience has been be collected during the cosmics data taking periodin 2009.Keywords: Trigger; Data Acquisition1. IntroductionThe ATLAS1 experiment is a multipurpose experiment at LHC investigat-ing the fundamental nature of matter and the basic interactions that ruleour universe. The ATLAS detector will analyze proton-proton collisions upto√s = 14 TeV produced by LHC. A highly selective trigger and data ac-∗Contact Address: J23910 - CERN - CH-1211 Geneve 23 Switzerland†The full list of authors is given in http://cdsweb.cern.ch/record/1207560December 30, 2009 21:53 WSPC - Proceedings Trim Size: 9in x 6in ICATTP˙proceedings˙sidoti˙22quisition system is required to reduce the LHC interaction rate of 1 GHzato the manageable recording rate of ∼ 200 Hz.To achieve these challenging requirements a large system for triggerand Data Acquisition (TDAQ) has been designed and deployed in the pastyears. The TDAQ system is monitored both in its software and hardwarecomponents and on the physics quality of the data collected.In this article, after a short introduction of the ATLAS trigger system,the various ingredients of the Online Monitoring Framework of the TDAQsystem are described.2. The ATLAS Trigger and Data Acquisition SystemThe ATLAS trigger system2 is composed of a hardware implemented Level-1 (L1) system, followed by a farm of PCs for the High Level Trigger (HLT)selection, which is performed in two steps: Level-2 (L2) and Event Filter(EF). The HLT farm is currently composed of ∼ 1000 nodes, a third of itsexpected final size.Using muon and calorimeter coarse data, the L1 trigger reduces theinput rate down to 75 kHz (upgradable to 100kHz) within 2.5 µ s of latency.The L1 is also responsible to identify the correct LHC bunch crossing andthe geometrical Region of Interest (RoI) pointing to data that will furtherbe analyzed at full granularity by the L2 system. Only data within the RoIwill be considered at the next level reducing the transfered rate by ∼ 98%of the full event size. After the acceptance from L2 with a rate of 1 kHz, theevent is build by the Event Builder (EB) and sent to the the EF. At thislevel, the full event is reconstructed with quasi oﬄine algorithms, detectorcalibrations and geometry. The output bandwidth is ∼ 300 MB/s, with anevent size of 1.5 MB this results in a recoding rate of s ∼ 200 Hz.The online selection is performed composing a chain of different triggeralgorithms at different trigger levels.For practical reasons, trigger chains selecting the same physics object(e.g. muon, electron, jet,...) are regrouped in the so called Trigger Signa-tures.3. The TDAQ Online Monitoring FrameworkThe ATLAS TDAQ Online Monitoring Framework is realized based ondifferent software tools. These applications are independent and partiallyaAt the luminosity of L = 1034cm2s−1 with 25 ns interval bunch crossing 23 additionalpile-up events are expected.December 30, 2009 21:53 WSPC - Proceedings Trim Size: 9in x 6in ICATTP˙proceedings˙sidoti˙23redundant allowing a stable supervision of the system on one hand and afast and reliable recognition of possible problems on the other. They dobasic information sharing among the various applications as well as highlevel application as complex graphical user interfaces (GUI). The focus hasbeen given to performance and scalability of the monitoring infrastructure.3.1. Basic ServicesThe Information Service (IS) application allows the sharing of informationamong the various monitoring applications. It is the backbone of the OnlineMonitoring Framework. Data shared can be simple variables and more com-plex classes. In particular, the Online Histogramming Service (OHS) basedon IS, permits the sharing of histograms data type. Information Providersinsert or update information while Information Readers read specific values.Monitoring applications can also subscribe to the repository being alertedif changes or modification in the data occur. Both IS and OHS are builton top of a common Inter Process Communication service implemented inCORBA.33.2. Monitoring ApplicationsThe monitored quantities from the L2 and EF processing tasks distributedamong the ∼ 1000 CPU of the HLT farm are stored on histograms thatneed to be summed up into a single histogram. This task is performed bythe gatherer4 that has to sum 600.000 histograms per HLT rack consideringthat each application creates 2.500 histograms and 250 applications run pereach HLT rack.The monitoring information produced for each run has to be archivedfor further cross check with oﬄine analysis. This is performed by the Mon-itoring Data Archiving.5 Histograms produced online are stored in ROOTfiles on a local data cache. The local data cache is later transfered to per-manent data storage. This is done asynchronously with respect of the endof a run to minimize the time needed to terminate a data taking run.ATLAS is a worldwide collaboration and it is vital to access the triggermonitoring information outside the ATLAS Control Rooms. One possibil-ity is to produce web pages using a Web Monitoring Interface (WMI) con-taining monitoring information. Currently three plugins exist showing runstatus, data quality and trigger rates information. Another possibility is tomirror in real time a partition with IS servers containing the available in-formation. All the online monitoring tools using that information can thusDecember 30, 2009 21:53 WSPC - Proceedings Trim Size: 9in x 6in ICATTP˙proceedings˙sidoti˙24be run outside the ATLAS Control Rooms.To efficiently display and manipulate the large number of histogramsproduced, the Online Histogram Presenter (OHP) application has beenimplemented using QT and ROOT.6 Histograms are presented in a treestructure or in configurable predefined windows that are automatically up-dated when the histograms change in OH. Also reference histograms canbe displayed superimposed to the produced histograms. OHP works witha mixed pull/push mode in the interaction with the OHS server(s). Notifi-cations are pushed from the OHS server to OHP every time a histogram isupdated and only when the histogram is actually displayed, the histogramobject is retrieved from the OHS server. A sophisticated cache mechanismminimizes the required network bandwidth.The Operational Monitoring Display (OMD) is a powerful applicationthat is able to read informations stored in IS. It performs operations (likesumming, averaging etc.) and displays the quantities elaborated in time-trends, bar charts, table etc. It is highly user friendly configurable with aDrag’n Drop approach.Monitoring the trigger rates is of fundamental importance during thecommissioning and data taking. In fact variation of trigger rates is a fastindication of changes or problems in individual ATLAS subdetectors, theTDAQ system or the beam conditions; thus a powerful and reliable ratemeasurement is required.The Trigger Rate Presenter (TRP) is a package that calculates, displays,monitors and archives the trigger rates and informations relative to theHLT farms. The TRP package is based on a Client-Server architectureand is composed by many applications. The key ideas in the design ofthe package have been the modularity of the various applications and thescalability. The adapters read the IS informations and calculate the ratesfor all the trigger levels. The trigger rates are calculated for all the triggerchains. The HLT rates are calculated for 10s interval for each L2 and EFapplication and than gathered together. The adapters perform therefore allCPU intensive applications. The calculated rates are stored in a specificIS server that keeps in memory for a couple of hours the measured triggerrates. The Graphical User Interface written in Qt displays the rates in timetrends and tables. The user can select any box to graphically display thetimetrends. A plugin exports the trigger rates to web pages that show therates in real time the rates for the last 24 hours. Using the MDA mechanism,an application archives permanently the trigger rates in ROOT TTreesthen stored for oﬄine analysis. Monitoring the trigger rates with TRP isDecember 30, 2009 21:53 WSPC - Proceedings Trim Size: 9in x 6in ICATTP˙proceedings˙sidoti˙25essential to adjust the trigger prescale factors to optimize the availablebandwidth and eventually react to changes in the LHC beam conditions orsome ATLAS subdetector.The data quality monitoring of the selected data based on the physicsobjects is performed with automatic checks by the Data Quality MonitoringFramework (DQMF). It is composed by a core package that checks theproduced histograms (width and mean of the distributions, Kolmogorov-Smirnov difference with respect to some reference histograms,...). A qualityflag for each trigger signature is calculated and stored in the Condition DBto be further used to assess the quality of data for physics analysis. A highlyconfigurable Data Quality Display shows the DQ results obtained with theDQMF and focus the attention of the shifter on the histograms failingthe automatic checks. The information is organized hierarchically goingfrom the top level, that shows the various ATLAS subsystem status, to theTrigger Signature status, down to the lowest level the checks performedon the single histograms. Within the standard reconstruction, a dedicatedtrigger monitoring is implemented, producing histograms according to theonline ones. They are also evaluated automatically using an oﬄine DQMFsystem delivering the same DQ flags for the trigger as the online DQMF.4. ConclusionThe monitoring framework for the ATLAS trigger is in place and works reli-ably. It has been used extensively during the combined cosmics data takingof 2008 and 2009 and will be of fundamental importance for a successfulcommissioning of the TDAQ system.References1. A. Collaboration, JINST 3 (2008).2. R. Hauser, The atlas trigger and data acquisition system, in these proceedings,3. R. Jones, Nuclear Science 47, 331 (2000).4. P. Renkel, The gatherer - a mechanism for integration of monitoring data inatlas, in Proc. 17th International Conference on Computing in High Energyand Nuclear Physics, (Prague, Czech Republic, 2009).5. P. F. Zema, The Monitoring Data Archiving Service for ATLAS, Tech. Rep.ATL-COM-DAQ-2006-043 (2006).6. A. Dotti, The online histogram presenter for the atlas experiment: a modularsystem for histogram visualization, in Proc. 17th International Conference onComputing in High Energy and Nuclear Physics, (Prague, Czech Republic,2009).

Trigger Monitoring at ATLAS

Abstract

Similar works

Full text

Available Versions

CERN Document Server