The LHC with its unprecedented complexity and criticality of beam operation will need thorough analysis of data taken from systems such as power converters, interlocks and beam instrumentation during events like magnet quenches and beam loss. The causes of beam aborts or in the worst case equipment damage have to be revealed to improve operational procedures and protection systems. The correct functioning of the protection systems with their required redundancy has to be verified after each such event. Post mortem analysis software for the control room has been prepared with automated analysis packages in view of the large number of systems and data volume. This paper recalls the requirements for the LHC Beam Post Mortem System (PM) and the necessity for highly reliable data collection. It describes in detail the redundant architecture for data collection as well as the chosen implementation of a multi-level analysis framework, allowing for automated analysis and qualification of a beam dump event based on expert provided analysis modules. It concludes with an example of the data taken during first beam tests in September 2008 with a first version of the system

Andreassen, O O

Baggiolini, V

Castaneda, A

Gorbonosov, R

Khasbulatov, D

Reymond, H

Rijllart, A

Romera Ramirez, I

Trofimov, N

Zerlauth, M

CERN Document Server

CERN-ATS-2010-00904/01/2010EUROPEAN ORGANIZATION FOR NUCLEAR RESEARCHCERN - ACCELERATORS AND TECHNOLOGY SECTORTHE LHC POST MORTEM ANALYSIS FRAMEWORKOdd Oyvind Andreassen, Vito Baggiolini, Alejandro Castaneda, Roman Gorbonosov,Denis Khasbulatov, Hubert Reymond, Adriaan Rijllart, Ivan Romera Ramirez,Nikolai Trofimov, Markus ZerlauthThe LHC with its unprecedented complexity and criticality of beam operation will need thorough analysis of datataken from systems such as power converters, interlocks and beam instrumentation during events like magnetquenches and beam loss. The causes of beam aborts or in the worst case equipment damage have to be revealed toimprove operational procedures and protection systems. The correct functioning of the protection systems with theirrequired redundancy has to be verified after each such event. Post mortem analysis software for the control room hasbeen prepared with automated analysis packages in view of the large number of systems and data volume. This paperrecalls the requirements for the LHC Beam Post Mortem System (PM) and the necessity for highly reliable datacollection. It describes in detail the redundant architecture for data collection as well as the chosen implementation ofa multi-level analysis framework, allowing for automated analysis and qualification of a beam dump event based onexpert provided analysis modules. It concludes with an example of the data taken during first beam tests in September2008 with a first version of the system.Presented at the 12th International Conference on Accelerator and Large Experimental Physics Control Systems(2009 ICALEPCS) - October 12-16, 2009 - Kobe, JapanGeneva, SwitzerlandCERN-ATS-2010-009January 2010 Abstract        THE LHC POST MORTEM ANALYSIS FRAMEWORK Odd Oyvind Andreassen, Vito Baggiolini, Alejandro Castaneda, Roman Gorbonosov, Denis Khasbulatov, Hubert Reymond, Adriaan Rijllart, Ivan Romera Ramirez, Nikolai Trofimov, Markus Zerlauth, CERN, Geneva, SwitzerlandAbstract The LHC with its unprecedented complexity and criticality of beam operation will need thorough analysis of data taken from systems such as power converters, interlocks and beam instrumentation during events like magnet quenches and beam loss. The causes of beam aborts or in the worst case equipment damage have to be revealed to improve operational procedures and protection systems. The correct functioning of the protection systems with their required redundancy has to be verified after each such event. Post mortem analysis software for the control room has been prepared with automated analysis packages in view of the large number of systems and data volume. This paper recalls the requirements for the LHC Beam Post Mortem System (PM) and the necessity for highly reliable data collection. It describes in detail the redundant architecture for data collection as well as the chosen implementation of a multi-level analysis framework, allowing for automated analysis and qualification of a beam dump event based on expert provided analysis modules. It concludes with an example of the data taken during first beam tests in September 2008 with a first version of the system. INTRODUCTION Analysis of transient data recorded by the different equipment systems of the LHC requires a powerful and reliable data collection as well as an efficient and accurate analysis for the thousands of PM data buffers arriving at the system in the case of beam dumps. As the domain knowledge of equipment experts is a key factor to the analysis of the data, they account responsible whenever possible for the implementation of the analysis modules, whereas the LHC Post Mortem Framework orchestrates the analysis flow and provides all necessary infrastructure to the analysis modules (such as read/write APIs for PM data, database/reference access, analysis configurations, etc..). In the following the key components of this analysis framework are described in detail. PM DATA COLLECTION  The Post Mortem Data collection has been realized using a redundant hardware infrastructure, using two HP proliants with RAID disk arrays being located in two different locations and connected to two different routers of the technical network at CERN. Each client system has the possibility of sending data to either of these two servers, but will be assigned to one of the two proliants being its ‘main’ server, whereas the 2nd server will be used as a fall back in case the main server cannot be reached. These main server processes for the different clients are equally distributed onto the two servers in order to optimize load sharing during normal operation. Data sending from the client systems is done through a dedicated client library, managing the sending of PM data to the PM servers transparently for the client system. In order to assure a coherent and reliable storage of the PM raw data files, the data collection will in addition create a second copy of the received file on the 2nd PM server.  Around 20 different client systems are today sending data to the PM servers, in case of beam dumps as much as 3000 individual files (containing up to 50GB of data) in a period of less than a few 10 seconds. The system is fully scalable and in case performance issues are observed in the future additional servers can be added in a transparent way. So far the experience has been very satisfying, as during 4 years of operation no file has ever been lost if being correctly sent by the client system.   PM EVENT BUILDING  Event building is one of the key components of the LHC Post Mortem Framework, as it has to detect interesting sets of PM data buffers which will be the subject of a detailed analysis.    Figure 1: Event Builder for Global PM events, showing the result for a (simulated) global beam dump event.1Hereby it is not only beam dump events which are of interest (resulting in a rather easily recognizable  avalanche of PM data) [1], but also events in the LHC Powering System which will only produce some two to five PM buffers, but which will also require for a detailed PM analysis. Event building is done based on pattern recognition within the continuous stream of incoming PM data files, using the trigger time-stamp of the PM buffer as one of the key criteria. Detected events have to be mutually exclusive, i.e. a single PM buffer can only belong to a single event. In case an event in the powering system will occur simultaneously to a global beam dump event, it will not be considered as an independent event but will be incorporated and analyzed within the global beam dump event. Figure 1 shows the example of a (simulated) global PM event (called GPM1) occurring in the Event Builder as well as of a test event (GPM2), surrounded by isolated individual system dumps (called ISODD). The latter will not trigger a global analysis session. In case an event has been detected which is judged as interesting for a detailed analysis, the event builder will forward the event details and the list of related PM data buffers to the analysis server. POST MORTEM ANALYSIS SERVER While the sole purpose of the event builder is the identification of interesting groups of PM buffers (and if possible the presumed type of event), will it be the task of the analysis server (with a simplified architecture shown in Figure 2) to trigger and present the results of the following analysis. Depending on the event type given to the analysis server by the event builder, a dedicated analysis configuration will be scheduled for execution by the PMA scheduler. The analysis configuration hereby contains the different analysis modules and their interdependencies which shall be executed with the retrieved PM data. The analysis is hereby executed twice for a given event, a first time after the preliminary identification of the event type (i.e. some 30 seconds after a beam dump, which is the time it will take most of the client systems to send PM data to the servers) in order to provide first preliminary results to the operators, and a second time once the event is finalised (typically after ~ 8 minutes, corresponding to the time it will take some of the equipment systems related to magnet powering to collect and send their PM buffers).  Figure 2: Simplified architecture of the LHC Post Mortem Framework.  ANALYSIS MODULES   The analysis modules are the sole place where domain knowledge of the LHC equipment systems and the LHC as such is present. The analysis server has been conceived in a way to be able to cope with analysis modules being implemented in different programming languages, thus profiting from existing work done by the equipment teams. In order to efficiently analyse the large amount of data awaiting analysis after a global dump the analysis modules have been arranged in three different stages, ie the data collection layer, Individual System Analysis Modules (ISA) and Global Analysis modules (GA). Data collection modules allow to filter (or convert e.g. binary data received from some equipment systems) for the following analysis modules. ISA modules are analysing exclusively data of a given equipment system, with the main task to identify and flag particularly interesting data 2for the following global GA modules and thus achieve a considerable data reduction facilitating both the automatic event qualification as well as manual analysis of the operation crews. Typical analysis modules in the ISA level are e.g. modules to calculate the beam losses or the beam orbit at the time of the beam dump. GA modules will not only use data from single equipment systems, but will correlate data across various user systems to search for the initial cause of the beam dump and to validate the correct performance of all involved machine protection systems. As an example, a quench of a main dipole magnet of the LHC will on one hand trigger a beam dump and result in a set of PM data from the Quench Protection System, but might also be causing beam losses downstream of the quenching magnet which would trigger in turn a beam dump and a set of PM data from the LHC Beam Loss Monitors. Only by correlating data of both systems and with the knowledge of the geographical location of the machine it will be possible to identify the quench as the cause and the losses as a sole consequence of this initial fault.   A final module will then assemble all the relevant details and analysis results of the analysed event and present it in a comprehensive and condensed form to the operator.  Data Manager and Libraries The PM Data Manager allows the framework and all analysis modules to easily retrieve any piece of PM data stored on the server without any detailed knowledge of the structure of the file system. It also assures the storage of event and analysis result data on the file system following the confirmation of an analysis session. The PM Data Manager in addition assures the bookkeeping in the form of detailed journal files which will also be uploaded to a controls database in order to allow for easier searches and statistics on PM data files and the related event and analysis result data.  Similar to the PM Data Manager, a number of APIs are available within the analysis server to allow modules a standardized and simple way of retrieving data from external data sources such as controls and logging databases as well as reference management systems.  Graphical User Interface  The operator interacts with the analysis server via a number of dedicated graphical user interfaces (GUI). Two main analysis modes are available, i.e. the ONLINE mode and the OFFLINE (playback) mode. In online mode the analysis server will continuously listen to the operational event builder and automatically trigger an analysis session in case an interesting event has occurred. The preliminary and final results will be presented to the engineer in charge who will then have to confirm the analysis outcome in order to allow for the next injection. As part of the analysis result he will be offered a summary of the checks performed on the different machine protection systems as well as a list of recommended actions before proceeding with the next injection.  The offline mode is primarily used to play back the analysis of a past event, e.g. to perform the analysis of a beam dump with slightly modified analysis criteria or to compare the analysis results in between two different beam dump events. Playing back an analysis can either be done using an existing event data file (i.e. the event details as they have been by the online event builder) or by re-building the complete event based on the PM data files arriving on a given day. This latter option will be useful to rebuild events with slightly different patterns or detection parameters in case of malfunctioning of the event builder or an equipment system (e.g. in case an important data file has been sent with a wrong timestamp and thus not been considered part of the event by the online event builder).  FIRST OPERATIONAL USE CASES So far only little operational experience could be gained with the LHC Post Mortem System, namely during the initial LHC operation in September 2008. Initial operation was dominated by injection tests and other machine developments, nevertheless a first emergency beam dump occurring with beam stored in ring 2 of the LHC was analyzed with a prototype version of the PM analysis framework and underlined the necessity for automated analysis. Although not yet containing global analysis modules nor the final result summary, it was capable to provide evidence for the initial cause being a faulty water cooling circuit in a power within less than ten minutes after the beam dump.  CONCLUSIONS AND OUTLOOK Parts of the LHC Post Mortem System have already been extensively exercised during the Hardware Commissioning period of the LHC [2] and valuable experience with equipment systems for magnet powering and the analysis of related data could be gathered. The territory of beam related data is however rather unexplored, and initial operation of the LHC machine will be an important period for the LHC PM system to further tune analysis modules and progress in the understanding of possible correlated failures and their identification. After the initial validation of the system it will become a more and more important challenge to provide tools to compare the results of different beam dump events against each other in order to identify similarities and to establish a knowledge database of possible failure scenarios and their detection based on the provided Post Mortem data.  REFERENCES [1] B.Goddard, M.Lamont, et al, “Post Mortem and beam Dump Data Acquisition Triggering”, LHC-TD-ES-0002, CERN, Geneva. [2] A. Vergara et al, “Software Applications for the Commissioning of the LHC Superconducting Circuits”, WEP018, these proceedings. 3

The LHC Post Mortem Analysis Framework

The LHC Post Mortem Analysis Framework

Abstract

Similar works

Full text

Available Versions

CERN Document Server