1 research outputs found
Scaling Big Data Platform for Big Data Pipeline
Monitoring and Managing High Performance Computing (HPC) systems and
environments generate an ever growing amount of data. Making sense of this data
and generating a platform where the data can be visualized for system
administrators and management to proactively identify system failures or
understand the state of the system requires the platform to be as efficient and
scalable as the underlying database tools used to store and analyze the data.
In this paper we will show how we leverage Accumulo, d4m, and Unity to generate
a 3D visualization platform to monitor and manage the Lincoln Laboratory
Supercomputer systems and how we have had to retool our approach to scale with
our systems.Comment: Accepted to MIT Northeast Database Day 201