HDB@ELK: another noSql customization for the HDB++ archiving system

The TANGO controls framework community has put a lot of effort in creating the HDB++ software system that is an high performance, event-driven archiving system. Its design allows storing data into traditional database management systems such as MySQL as well as NoSQL database such as Apache Cassandra. The architecture allow also to easily extend it to other noSql database like, for instance, ELK. This paper describes the step needed to extend the HDB++ and explore the possibilities to use the ELK technology in term of analysis being enabled and tools provided

Very large scale high performance computing and instrument management for high availability systems through the use of virtualization at the Square Kilometre Array (SKA) telescope

The Square Kilometer Array (SKA) Telescope, is an ongoing project set to start its building phase in 2018 and be ready for first light in 2020. The first part of the project, the SKA1 will be comprised of 130.000 low frequency antennas (50 MHz to 350 MHz) and 200 mid frequency antennas (350 MHz to 15.5 GHz). The SKA1 will produce a raw data rate of ~10 Tb/s, require a computing power of 100 Pflop/s and an archiving capacity of hundreds of PB/year. The next phase of the project, the SKA2, is going to increase the number of both low and mid antennas by a factor of 10 and increase the computing requirements accordingly. The key requirements for the project are a very demanding availability of 99.9%, computing scalability and result reproducibility. We propose an approach to enforce these requirements - with an optimal use of resources - by using highly distributed computing and virtualization technologies.publishe

SKA telescope manager: a status update

The international Square Kilometre Array (SKA) project to build two radio interferometers is approaching the end of its design phase, and gearing up for the beginning of formal construction. A key part of this distributed Observatory is the overall software control system: the Telescope Manager (TM). The two telescopes, a Low frequency dipole array to be located in Western Australia (SKA-Low) and a Mid-frequency dish array to be located in South Africa (SKA-Mid) will be operated as a single Observatory, with its global headquarters (GHQ) based in the United Kingdom at Jodrell Bank. When complete it will be the most powerful radio observatory in the world. The TM software must combine the observatory operations based at the GHQ with the monitor and control operations of each telescope, covering the range of domains from proposal submission to the coordination and monitoring of the subsystems that make up each telescope. It must also monitor itself and provide a reliable operating platform. This paper will provide an update on the design status of TM, covering the make-up of the consortium delivering the design, a brief description of the key challenges and the top level architecture, and its software development plans for tackling the construction phase of the project. It will also briefly describe the consortium’s response to the SKA Project’s decision in the second half of 2016 to adopt the processes set out by the Software Engineering Institute (SEI) for system architecture design and documentation, including a re-evaluation of its deliverables, documentation and approach to internal reviews.publishe