Today's HPC installations are highly-complex systems, and their complexity
will only increase as we move to exascale and beyond. At each layer, from
facilities to systems, from runtimes to applications, a wide range of tuning
decisions must be made in order to achieve efficient operation. This, however,
requires systematic and continuous monitoring of system and user data. While
many insular solutions exist, a system for holistic and facility-wide
monitoring is still lacking in the current HPC ecosystem. In this paper we
introduce DCDB, a comprehensive monitoring system capable of integrating data
from all system levels. It is designed as a modular and highly-scalable
framework based on a plugin infrastructure. All monitored data is aggregated at
a distributed noSQL data store for analysis and cross-system correlation. We
demonstrate the performance and scalability of DCDB, and describe two use cases
in the area of energy management and characterization.Comment: Accepted at the The International Conference for High Performance
Computing, Networking, Storage, and Analysis (SC) 201