1 research outputs found
A Fast Lightweight Time-Series Store for IoT Data
With the advent of the Internet-of-Things (IoT), handling large volumes of
time-series data has become a growing concern. Data, generated from millions of
Internet-connected sensors, will drive new IoT applications and services. A key
requirement is the ability to aggregate, preprocess, index, store and analyze
data with minimal latency so that time-to-insight can be reduced. In the
future, we expect real-time data collection and analysis to be performed both
on small devices (e.g., in hubs and appliances) as well in server-based
infrastructure. The ability to localize sensitive data to the home, and thus
preserve privacy, is a key driver for small-device deployment.
In this paper, we present an efficient architecture for time-series data
management that provides a high data ingestion rate, while still being
sufficiently lightweight that it can be deployed in embedded environments or
small virtual machines. Our solution strives to minimize overhead and explores
what can be done without complex indexing schemes that typically, for
performance reasons, must be held in main memory. We combine a simple in-memory
hierarchical index, log-structured store and in-flight sort, with a
high-performance data pipeline architecture that is optimized for multicore
platforms. We show that our solution is able to handle streaming insertions at
over 4 million records per second (on a single x86 server) while still
retaining SQL query performance better than or comparable to existing RDBMS