The foundation of the CDF Run II online event monitoring framework, placed well before the physics runs start, allowed to develop coherent monitoring software across all the different subsystems which consequently made maintenance and operation simple and efficient. Only one shift person is needed to monitor the entire CDF detector, including the trigger system. High data quality check is assured in real time and well defined monitoring results are propagated coherently to offline datasets used for physics analyzes. We describe the CDF Run II online event monitoring system and operation, with emphasis on the remote monitoring shift operation started since November 2006 with Pisa-INFN as pilot Institution and exploiting the WEB based access to the data

Arisawa, T.

Fabiani, D.

Hirschbuehl, D.

Ikado, K.

Kubo, T.

Kusakabe, Y.

Maeshima, K.

Naganoma, J.

Nakamura, K.

Plager, C.

Schmidt, E.

UNT Digital Library

Online Monitoring for the CDF Run II Experimentand the Remote Operation FacilitiesT.Arisawaa, D.Fabianib, D.Hirschbuehlc, K.Ikadoa, T.Kubod , Y.Kusakabea,K.Maeshimae, J.Naganomaa, K.Nakamuraa, C.Plagere, E.Schmidt f , F.Scurib∗†,H.Stadiec, R.Tsychiyaa, G.Veramendig, W.Wagnerc, H.Wenzelc, M.Worcester f ,K.Yoritaha Waseda University, Tokyo, Japanb Istituto Nazionale di Fisica Nucleare, Pisa, Italyc Universitaet Karlsruhe, Germanyd K.E.K, Tsukuba, Japane University of California at Los Angeles, U.S.f Fermilab, Batavia (IL), U.S.g L.B.L., Berkeley (CA), U.S.h University of Chicago, U.S.The foundation of the CDF Run II online event monitoring framework, placed well before thephysics runs start, allowed to develop coherent monitoring software across all the different sub-systems which consequently made maintenance and operation simple and efficient. Only oneshift person is needed to monitor the entire CDF detector, including the trigger system. Highdata quality check is assured in real time and well defined monitoring results are propagated co-herently to offline datasets used for physics analyzes. We describe the CDF Run II online eventmonitoring system and operation, with enphasis on the remote monitoring shift operation startedsince November 2006 with Pisa-INFN as pilot Institution and exploiting the WEB based accessto the data.XI International Workshop on Advanced Computing and Analysis Techniques in Physics ResearchApril 23-27 2007Amsterdam, the Netherlands∗Speaker.†E-mail: fabrizio.scuri@pi.infn.itc© Copyright owned by the author(s) under the terms of the Creative Commons Attribution-NonCommercial-ShareAlike Licence. http://pos.sissa.it/FERMILAB-CONF-07-765-ECDF Online Monitoring F.Scurib1. IntroductionThe CDF experiment is data taking since 2002 at the Run-II Fermilab Tevatron collider oper-ating proton-antiproton interactions at 1.96 TeV center of mass energy. The CDF detector, whosedetailed description can be found elsewhere [1], has a typical architecture for high energy exper-iments, divided into a tenth of subdetectors including: inner tracking systems and Time of Flightdetetctors, central superconducting solenoid (1.4 T), outer calorimeters and muon detectors, 3-leveltrigger system, multi-step data flow chain (250 dual processors for L3 trigger, robotic tape storage,150 dual processors for the production farm, 9 Distributed Central Analysis Facilities), and a 20PC cluster for the online operations. The online structures include the Data Aquisition system,monitors for detector calibration and data taking operation, event display, and Web servers. TheCDF detector and data flow complexity are similar to those of the experiments designed for theLHC program; in particular, the following specs summarize the CDF operation highlights:• 0.75 Million channels to be operated and monitored.• 75 Hz output of the Level 3 trigger at 20 MB/s (system limit), corresponding to about 0.5PB/year of data throughput.The main characteristics of the architecture of the CDF online system are described in section 2;some details of the new CDF remote monitoring operations are given in section 3; conclusions aresummarized in section 4.2. Architecture of the CDF online systemThe main characteristics of the CDF online system are described in this section; a more de-tailed description can be found elsewhere [2]. The online architecture was designed to match thefollowing requirements:• DAQ, Trigger, and detector performances must be monitored without interfering with thedata taking.• Monitored results must be interpreted by the crew on shift in a fast and clear way in order tomantain high quality data taking.• Different monitor (Consumer) processes can run on different machine to allow the system tobe easily expanded.• each Consumer receives only the data it needs according to the trigger selections.• Monitor and Display process types are separated; the number of active displays are limitedby the network traffic and the bandwidth.• Different Consumer processes can be combined to one executable.• Online software is developped within the CDF sofware environment as a standard detectorpartition to simplify maintainability.2CDF Online Monitoring F.Scurib• Save remote access to the monitoring operations and maintainance must be grant to the re-mote shifters and to the experts.The raw scheme of the CDF online monitoring architecture is shown in figure 1.L3 TriggerPush ProtocolConsumer ServerConsumer 1Consumer 2Consumer 3Consumer NServerServerServerServerdisplay display display displayConfig. Via “Talk to”Error ReceiverRun ControlSerious errorswww wwwState MonitorFigure 1: Raw architecture of the CDF online monitoringData relevant to the monitor operation are produced by the Consumers, receiving the inputfrom the Consumer servers, continuosly running during data aquisition, and updating the DisplayServers. Display operations are operator driven via Display Browsers which are programs passingto the user endnode the data received from the Display Servers; Display Browsers can run on thesame machine as the Display Servers or on different machines either via a socket connection or viaWEB through a plug-in for the APACHE [3] webserver to maintain ROOT [4] compatibility (figure2).Examples of monitor displays are shown in figure 3. All monitor displays are updated eachsecond; they are very useful to promptly detect new dead regions (contiguous white cells in theexample shown in figure 3). Histograms produced by the displays are available on WEB afterupdate requests and, very recently, snapshot pictures of all DAQ and online screens in the CDFcontrol room were let available on WEB (down to 5 sec. update) for remote monitoring porposes.AllConsumer andMonitor programs are based on C++ modules of Analisys Control++ (AC++)[5]; AC++ is the CDF analysis framework handling dataflow between modules; it serves input, out-put, event generation, detector simulation, event reconstruction, access to the database, and physicsanalyses; ROOT is the SW package used for event I/O and analysis tools; several version of theCDF SW package are released each year and the online modules must follow the SW evolution;each specific consumer/monitor is written according to the sub-detector specs, within the CDF SW3CDF Online Monitoring F.ScuribEVENTSCONSUMERSConsumer 1Consumer 2…SERVERTRIGGER TABLESTEMPLATESDATABASE(oracle management system)SOCKETDISPLAYFast socketconnectionFILESHTTP SERVERAPACHEDISPLAYWWWInput through web based oracle forms,oracle reports and automated cron jobsFigure 2: Element communication layoutframework; template programs are used with utility methods to access data and to produce his-tograms without interfering with the data acquisition; standard procedures are defined to compareactual and reference plots and to provide instructions to the operators.Typical monitored objects are: detector occupancies, trigger rates, luminosity, vertex position,physics objects, L3 reconstruction, and calibration results.Just one person in shift (Consumer Operator) can handle full detector and data flow monitor-ing; sub-detector experts are paged only in case of major troubles.3. Remote monitoring operationsThe main job for the Consumer Operator (CO) is to online check the detector operation anddata quality by fulfilling the following duties:a) by looking at the:- Consumer Monitor outputs;- Detector Calibration results;- Event Display.b) by submitting the result of a detailed check-list of the monitored objects.Once grant the possibility to access from remote to the full information and to the operationsrequested to serve the CO shift, there is no reason for the CO operator to be physically in the CDFcontrol room. Therefore, CDF decided to move from local to remote all owl CO shifts. This hasa strong impact for the members of the CDF collaboration of European and Asian Institutions andfor the CDF shift organization itself. Exploiting the regional time difference, owl CDF CO shiftscan be assigned to European or Asian members, making remote shifts in a more confortable daytime and, at the same time, reducing CDF the number of people on night shift to the ScientificCoordinator and DAQ operator.Tsukuba (KEK, Japan) and Pisa (Italy) were chosen as pilot Institutions for the CDF RemoteCO shift project; the project developed in cooperation with CMS-Fnal group, which is setting-up4CDF Online Monitoring F.ScuribHadronic calorimeter occupancy Monitor Event DisplayFigure 3: Examples of monitor displays.a multipurpose remote control room in the Wilson Hall at Fnal, and with the aim to acquire theknow-how for future applications of remote shifts (LHC, ILC).The remote shift project pushed the online group to make all the upgrades required to allowfull remote operation; this resulted in a general improvement and simplification of the ConsumerOperator shift; among several upgrades made to allow all quality and status checks via WEB,we just mention one relative to the CO specific operations during the shift, i.e. the possibility toremotely change the run mode (Physics/Comsmic) and to start/restart Consumers and Displays.These operations are normally made in local by running command scripts from any machine of theonline cluster. However, remote login on the online cluster from outside Fnal is forbidden by theFnal-D.o.E. policy for people other than online experts. To overcome this limitation and allow theremote shifter to operate the executables, a new GUI with action bottons was developed; the GUIwindow is exported from Fnal to the authorized remote node with a simple script launched by thelocal crew once at the beginning of the remote shift. The new GUI, named ConCon (ConsumerControl), is based on ROOT classes and merges in a user friendly way all action alternativelyperformed in local with a lot of shell commands launching script files. The ConCon GUI windowis described in figure 4.Commissioning of the remote shift procedure and first standard remote shifts were done fromPisa; the initial set-up of the remote control room is quite simple; a desktop equipped with a web-cam and running Polycom PVX under Windows XP was devoted to the video-connection between5CDF Online Monitoring F.ScuribConsumer status action buttons(with pop-up menus)Running mode switch buttonsConsumer monitors action buttons(with pop-up menus)Monitors and event display status action buttons (with pop-up menus)Access to the log filesFigure 4: The ConCon GUI window.local and remote control rooms; a second desktop with a four-screen drive card was used for thenormal shift operation, normally histogram checking from WEB browsers, running under ScientificLinux. The remote system will be upgraded with a second desktop driving 6 additional monitorswhere real time updated snapshots of the DAQ, slow control, and online screens in the local (Fnal)control room will be observed. Figure 5 shows the local (Fnal) section of the CDF control roomdevoted to the Consumer Operator monitoring (top-left) and the CDF remote control room in Pisa(botom-right).Commissioning of the remote shift procedures was completed in the period September-November2006; in this period the details of the start-up procedure for the remote shifts in Pisa and for emer-gency situation handling were defined, e.g. remote shifter replacement with a local (Fnal) one incase of any failure at the start-up and in case of network or HW failures during the shift. NewHW (Pisa and Fnal) for video-connection and monitoring (Pisa) was tested as well the new SW forremote shift (new control GUI and new WEB based procedures).Pilot remote CO shifts were made in November 2006: two 8 day regular owl shifts, Pisa andFnal in overlap were performed to check the long term network connection reliability, which wasconsidered the main issue (short network breakdowns (≤ 15 min.) are not an issue).Regular remote CO shift operation started on December 2006: at the beginning 4 shift (oneper month) were done until March 2007, then 2 shifts per month were scheduled from April 2007;6 × 8 day shift statistics is very encouraging: no HW, nor SW problems detected, no network6CDF Online Monitoring F.ScuribSection of the CDF control room at Fnaldevoted to the Consumer MonitorsCDF control room in Pisa(more monitors to be added)Figure 5: The CDF local (top-left) and remote (bottom-right) control roomsconnectivity breakdowns, just 15 min. interruption due to a scheduled maintenance.4. ConclusionsThe CDF online system is reliably operating since several years; the system architecture andSW upgrades smoothly developed according to the CDF SW environment evolution.Real time monitoring of the detector status and of data quality is now fully operated alsofrom remote both for expert feedbacks and for quality checking and monitor (Consumer Operatorsshifts); local and remote CO shifts proceed now exactly in the same way.Changes required for the remote shifts resulted in a global improvement towards a more userfriendly environment.First period operation of remote shifts from Pisa is very positive; based on the Pisa experience,CDF plans to operate other remote sites and to achieve assignment to remote 100% owl CO shifts.We encourage other worldwide experiment communities to follow this example.References[1] R. Blair at al. (CDF Collaboration), "The CDF II Detector - Technical Design Report",FERMILAB-PUB-96-390-E (1996)7CDF Online Monitoring F.Scurib[2] T. Arisawa et al.,"Online Monitoring for the CDF Run II Experiment", proceeding of CHEP 2003,March 24-28 2003, La Jolla, CaliforniaW. Wagner et al., "Online Monitoring in the CDF II Eperiment", proceedings of HEP2001, 12-18 July2001, Budapest, Hungary, hep2001/273.[3] The Apache Software Foundation, http://www.apache.org[4] R. Brun and F. Rademakers, Nucl.Instrum.Meth. A389:81-86,1997.[5] E. Sexton-Kennedy," A Users Guide to the AC++ Framework", CDF public note,CDF/DOC/CDF/COMP_UPG/PUBLIC/41788

Online monitoring for the CDF Run II experiment and the remote operation facilities

https://digital.library.unt.edu/ark:/67531/metadc899407/m2/1/high_res_d/946183.pdf

Online monitoring for the CDF Run II experiment and the remote operation facilities

Abstract

Similar works

Full text

Available Versions

UNT Digital Library