1 research outputs found
Railgun: managing large streaming windows under MAD requirements
Some mission critical systems, e.g., fraud detection, require accurate,
real-time metrics over long time sliding windows on applications that demand
high throughput and low latencies. As these applications need to run 'forever'
and cope with large, spiky data loads, they further require to be run in a
distributed setting. We are unaware of any streaming system that provides all
those properties. Instead, existing systems take large simplifications, such as
implementing sliding windows as a fixed set of overlapping windows,
jeopardizing metric accuracy (violating regulatory rules) or latency (breaching
service agreements). In this paper, we propose Railgun, a fault-tolerant,
elastic, and distributed streaming system supporting real-time sliding windows
for scenarios requiring high loads and millisecond-level latencies. We
benchmarked an initial prototype of Railgun using real data, showing
significant lower latency than Flink and low memory usage independent of window
size. Further, we show that Railgun scales nearly linearly, respecting our
msec-level latencies at high percentiles (<250ms @ 99.9%) even under a load of
1 million events per second.Comment: arXiv admin note: text overlap with arXiv:2009.0036