Online Event Processing: Achieving Consistency Where Distributed Transactions Have Failed

Distributed transactions have failed as a mechanism for ensuring consistency across heterogeneous storage technologies in today's large-scale applications. Instead, we are witnessing the emergence of a programming model that relies on append-only event logs rather than transactions, and which we call OnLine Event Processing (OLEP) in contrast to OLTP. In this article we show that, although an event log is a very simple abstraction, applications that rely on such logs can efficiently provide strong consistency guarantees, such as atomicity and enforcing invariants, which are normally associated with ACID transactions. We provide case studies from real industrial data systems that have adopted the OLEP approach, demonstrating the practical advantages of building upon event logs.The Boeing Compan

The BaBar Event Building and Level-3 Trigger Farm Upgrade

The BaBar experiment is the particle detector at the PEP-II B-factory facility at the Stanford Linear Accelerator Center. During the summer shutdown 2002 the BaBar Event Building and Level-3 trigger farm were upgraded from 60 Sun Ultra-5 machines and 100MBit/s Ethernet to 50 Dual-CPU 1.4GHz Pentium-III systems with Gigabit Ethernet. Combined with an upgrade to Gigabit Ethernet on the source side and a major feature extraction software speedup, this pushes the performance of the BaBar event builder and L3 filter to 5.5kHz at current background levels, almost three times the original design rate of 2kHz. For our specific application the new farm provides 8.5 times the CPU power of the old system.Comment: Talk from the 2003 Computing in High Energy and Nuclear Physics (CHEP03), La Jolla, Ca, USA, March 2003, 4 pages, 1 eps figure, PSN MOGT00

High Level Trigger Configuration and Handling of Trigger Tables in the CMS Filter Farm

The CMS experiment at the CERN Large Hadron Collider is currently being commissioned and is scheduled to collect the first pp collision data in 2008. CMS features a two-level trigger system. The Level-1 trigger, based on custom hardware, is designed to reduce the collision rate of 40 MHz to approximately 100 kHz. Data for events accepted by the Level-1 trigger are read out and assembled by an Event Builder. The High Level Trigger (HLT) employs a set of sophisticated software algorithms, to analyze the complete event information, and further reduce the accepted event rate for permanent storage and analysis. This paper describes the design and implementation of the HLT Configuration Management system. First experiences with commissioning of the HLT system are also reported

Stepwise correlation of multivariate IoT event data based on first-order Markov chains

Correlating events in complex and dynamic IoT environments is a challenging task not only because of the amount of available data that needs to be processed but also due to the call for time efficient data processing. In this paper, we discuss the major steps that should be performed in real- or near real-time event management focusing on event detection and event correlation. We investigate the adoption of a univariate change detection algorithm for real-time event detection and we propose a stepwise event correlation scheme based on a first-order Markov model. The proposed theory is applied on the maritime domain and is validated through extensive experimentation with real sensor streams originating from large-scale sensor networks deployed in a maritime fleet of ships.Comment: arXiv admin note: substantial text overlap with arXiv:1803.0563