Chilingaryan, S.

Haas, D.

Kopmann, A.

Mexner, W.

Ressmann, D.

KITopen

ADEI AND TANGO ARCHIVING SYSTEM – A CONVENIENT WAY TO ARCHIVE AND REPRESENT DATA  D. Haas, S. Chilingaryan, A. Kopmann, D. Ressmann, W. Mexner, KIT, Karlsruhe, Germany  Abstract Tango offers an efficient and powerful archiving mechanism of Tango attributes in a MySQL database. The tool Mambo allows an easy configuration of all to be archived data. This approved archiving concept was successfully introduced to ANKA (Angströmquelle Karlsruhe).To provide an efficient and intuitive web-based interface instead of complex database queries, the TANGO Archiving System was integrated into the “Advanced Data Extraction Infrastructure ADEI”. ADEI is intended to manage data of distributed heterogeneous devices in large-scale physics experiments. ADEI contains internal preprocessing, data quality checks and an intuitive web interface, that guarantees fast access and visualization of huge data sets stored in the attached data sources like SQL databases or data files. ADEI and the Tango archiving system have been successfully tested at ANKA's imaging beamlines. It is intended to deploy the whole system at all ANKA beamlines.  INTRODUCTION ANKA is a third generation synchrotron light source operated by the Karlsruhe Institute for Technology (KIT). ANKA is operating sixteen beamlines and three more are under construction.  The control system of the ANKA beamlines is based on Tango [1], which has been a convenient and reliable control system. Tango offers an archiving system [2] which allows logging all Tango-attributes. In 2014 this archiving system was evaluated at ANKA to log the data of an experiment. All logged data of a beamline and the experiment should be presented and retrieved in a modern, state of the art web interface. This offers the user a convenient way to access the data. ADEI a web based interface for database query, developed by the Institute for Data Processing and Electronics (IPE) of KIT fulfils exactly these requirements. Using ADEI as a viewer respectively analysis tool and connecting it to the databases of the Tango archiving system creates a platform to track and to monitor the status of a beamline. For testing the environment, the system was developed and implemented at two beamlines Topo-Tomo [3,4] and Image at ANKA. ADEI The “Advanced Data Extraction Infrastructure (ADEI)” has been developed to provide ad-hoc data exploration capabilities to a broad range of long-running physical experiments dealing with time series data [5]. Such experiments have varying characteristics and often composed from multiple subsystems developed by different vendors. As a result, the underlying storage engines and the data formats often differ even between subsystems of a single experiment. On the other hand, the users want to get uniform access to all the data. Easy correlation of data produced by any components of the system is desirable. Beside this, operators need a tool providing the possibility to examine all collected data, checking the integrity and validity of measurements.The ADEI architecture shown in Figure 1 is modular. New data sources can easily be included. The backend provides the desired uniform access to the data. The web-based front-end allows quick inspection of data archives. The communication between front-end and back-end is realized using the AJAX (Asynchronous JavaScript + XML) paradigm.  Figure 1: Architecture of Advanced Data Extraction Infrastructure ADEI.  Data Source Access Layer unifies access to the time series stored in different formats. After data filtering and quality checks the data is aggregated and stored in intermediate caching databases. Access to the data is provided by the ADEI library and web services are used to communicate with client applications.  The backend consists of multiple components organizing the data flow from the data source to the client application.  The Data Access Layer hides details of the underlying data sources and provides other components of the system with uniform access to all types of data. This is released using independent source drivers implementing ADEI data access interface.  Furthermore, the data is passed through the chain of the configured data processing plugins which analyze the data, control the data quality, and optionally apply correction coefficients or filter out bad values. Hence, the rest of the system can fully rely on approved data quality. FPO026 Proceedings of PCaPAC2014, Karlsruhe, GermanyISBN 978-3-95450-146-5210Copyright©2014CC-BY-3.0andbytherespectiveauthorsUser Interfaces and Data DisplaysADEI is designed to deal with data sampled at high data rates and stored for long periods of time. It is still impossible to access large volumes of data interactively. Therefore, newly recorded data is continuously preprocessed. The data is aggregated over several predefined intervals, so called cache levels. For each cache level statistical information is calculated and stored in the caching database.  The higher ADEI subsystem, then, can request the raw data or to reduce amount of received data, any statistical information from the selected cache level. To illustrate the concept, the rendering module can request the mean values from 60 seconds cache level effectively receiving minute averages instead of the raw data. ADEI provides several interfaces to access the managed data. The Export module provides direct access to the data in variety of formats implemented as plugins. The Search module allows to quickly find required data channels or time intervals where certain channels possessed the specified characteristics. The Plot module is used to quickly generate preview charts.  All modules are interfaced by a Web Service interface. This interface is used by ADEI front-end but also by a number of 3rd party applications. Figure 2: Screenshot of ADEI Web Front-end. The slow control data is rendered in the plot. The data outage is indicated using a small line on top of the plot (see 5). Legend contains description of displayed graphics. The selected part of plot may be zoomed or exported using buttons 3 and 4. Axes controls and results of search are located in the left sidebar.  The ADEI front-end facilitates fast and intuitive (Google maps-style) navigation as shown in Figure 2. Based on the caching subsystem described above, the users can request overview plots over long time intervals interactively. Since the quality checks were executed during the data preprocessing, the problematic intervals are highlighted on the overview charts. User, then, can easily navigate to the interval of interest and zoom-in using mouse only. The complete plot-generation time does not normally exceed 500 ms for any type of requested data. The ADEI web service interface allows platform and programming language independent design of application-specific data management solutions. To bring an example, we have developed a WebGL application that shows the temperature distribution on the provided 3D model based on the specified mappings of the ADEI data channels onto the model. Also, we developed software libraries to use ADEI data within the National Instruments LabVIEW environment and Cern's ROOT data analysis suite. SETTING UP THE WHOLE TANGO ARCHIVING SYSTEM AT ANKA  The ANKA archiving system is based on the approved Tango archiving and ADEI (see Fig. 3).   Figure 3: Tango Archiving Infrastructure and User Interface.  Every beamline at ANKA has its own Tango database which can be archived by one server, where the Tango archiving system is running. To configure the values and the attributes which have to be logged, Mambo [2], is used. Mambo is a user friendly graphical user interface (GUI), which offers an absolute well-structured workflow to set up the archiving by the beamline scientist. To read out and view the logged data by the archiving system, ANKA was looking for a powerful tool representing it by a state of the art web interface.  To integrate Tango data, we have developed a new Tango reader which implements ADEI data access interface and exposes the data from the Tango database using it. Additionally, to make data navigation more convenient, we implemented the Tango device tree in the ADEI channel selection dialog. Now, the data from Tango archives can be interactively visualized using ADEI web interface, exported in multitude of formats supported by ADEI, and transparently accessed using ADEI web services. Other existing non-Tango data sources can be visualized side-by-side with the Tango data (see Fig. 4).    Proceedings of PCaPAC2014, Karlsruhe, Germany FPO026User Interfaces and Data DisplaysISBN 978-3-95450-146-5211 Copyright©2014CC-BY-3.0andbytherespectiveauthorsFigure 4: Screenshot of ADEI Web Front-end for the ANKA Tango Archiving System. The data from ANKA slow control system (Bragg axes at the Topo-Tomo beamline) is rendered in the plot. (1) main menu, (2) dropdown-menu for selecting server respectively beamline, (3) data selection, (4) plot of the selected data. CONCLUSION & OUTLOOK The whole system can be setup in fast and easy way with only a small amount of work. The concept has been proved to be absolute stable and reliable. The big advantage of the whole system is that it’s connecting the Tango archiving system with a total user-friendly web-based user interface to read out the logged data. Due to the ADEI architecture the visualization of huge datasets is extremely fast.   After several months of testing phase it has been proved that the beamline and experiment status can be represent in a convenient and transparent way. As it has been previously emphasized, the Tango archiving system connected to the web front-end ADEI is implemented at the Topo-Tomo as well as at the Image beamline. The remaining challenge is now to implement it to all ANKA beamlines. This is scheduled for the beginning of 2015. WHERE TO GET ADEI? ADEI and the install instructions can be found under http://adei.info REFERENCES [1] J.-M. Chaize, A. Goetz, W.-D. Klotz, J. Meyer, M. Perez, E. Taurel, “TANGO – an object oriented control system based on corba”, International Conference on Accelerator and Large Experimental Physics Control Systems, (1999): p. 475 – 479.  [2] Tango archiving system, http://www.tango-controls.org/tools/archiving-system-2/archivingsystem   [3] A. Rack, T. Weitkamp, S. Bauer Trabelsi, P. Modregger, A. Cecilia, T. dos Santos Rolo, T. Rack, D. Haas, R. Simon, R. Heldele, et al., “The micro-imaging station of the topotomo beamline at the ANKA synchrotron light source”, Nuclear Instruments and Methods in Physics Research Section B: Beam Interactions with Materials and Atoms, 267(2011): p. 1978 - 2009. [4] P. Vagovic, T. Farago, T. dos Santos Rolo, S. Chilingaryan, T. Müller, A. Cecilia, T. van de Kamp, E. Hamann, A. Riedel, D. Haas, W. Mexner, M. Fiederle, T. Baumbach, “Recent upgrades at the micro-tomography station at TopoTomo Beamline at ANKA light source”, will be published in Journal of Synchrotron Radiation. [5] S. Chilingaryan, A. Beglarian, A. Kopmann, S. Voecking, "Advanced data extraction infrastructure: Web based system for management of time series data," Journal of Physics: Conference Series, 219 (2010), Part 4, 10 pages  FPO026 Proceedings of PCaPAC2014, Karlsruhe, GermanyISBN 978-3-95450-146-5212Copyright©2014CC-BY-3.0andbytherespectiveauthorsUser Interfaces and Data Displays

ADEI and Tango archiving system. A convenient way to archive and represent data

https://publikationen.bibliothek.kit.edu/170101029/18770275

ADEI and Tango archiving system. A convenient way to archive and represent data

Abstract

Similar works

Full text

Available Versions

KITopen