Consistent splitting of event streams in parallel complex event processing

Complex Event Processing (CEP) combines streams of simple events from multiple sources and infer high-level knowledge by correlating the simple events. The correlation of events is done according to the requirement of the application. This requirement is specified to the CEP system in the form of rules. State of the art implementations of CEP system deploy a distributed network of operators to execute a rule. For low latency information retrieval under high event rate of input streams, scalability of operators is required. This is done by the parallelization of operator functionality. The existing PACE system provides a framework for operator parallelization. Parameter context in a rule specify which events in the input stream can be used for correlation and are essential for application of CEP systems to many real world situations. In this thesis operator parallelization techniques, for adopting the existing PACE system to execute rules with various parameter contexts, are proposed. In cases where the existing PACE system cannot be adopted, other approaches for operator parallelization are proposed. Finally, the proposed techniques are analysed by implementing the techniques and evaluating their performance

In-memory check pointing in speculative parallel complex event processing

Parallel Complex Event Processing can be used to process high rate streams to get better latency. In order to handle out of order events, buffering and speculative processing techniques are discussed and analyzed. Another technique that merges both of them and adapts to the situation at runtime sounds promising. However, there are some inherent limitations of the technique due to which it doesn’t perform very well for parallel CEP systems. Our implementation is mainly based on this technique but we have done some optimizations to enhance the performance for parallel CEP systems. There are two major optimizations done in this technique. First of all, the internal recovery process is introduced which allows us to reduce communication overhead because we don’t forward the events to the operator instances again after recovery. Instead, we just send some messages containing commands to amend the windows, as required. This can help a lot, especially in the cases when the events contain some heavy data like high quality pictures or video clip etc. Secondly, we have divided the tasks to all the operator components, so that they can take checkpoints as well as recover back when required, irrespective of the other operator components. This allows them to run at their own pace instead of the consistency manager asking all of them to take checkpoint and send their respective states to the consistency manager. In this way, the waiting time of the consistency manager is avoided. Moreover, we have also introduced some minor optimizations like we can access the queues directly and add the out of order events to them, which avoids the recovery of the system, if possible. We have also introduced a high priority queue in the operator instance to allow the out of order events to bypass the normal queue and avoid recovery of the operator instance, if possible. Evaluations are done using synthetic data and the results show that our optimizations have increased the throughput as well as improved the latency in most of the cases