Combining neural networks with symbolic approaches to perform complex event processing on non-symbolic data

Abstract

This thesis presents three approaches to detecting situations of interest from non-symbolic data inputs such as images, audio and video through the use of Complex Event Processing (CEP) in an agile, reliable and efficient manner. These approaches must be agile, meaning that they must allow the implementation of solutions for a range of situations. We want them to be reliable, meaning that they must correctly detect the situations we are interested in. Finally, we want them to be efficient in terms of time and training data requirements. First, we present ProbCEP, an approach that combines proxy models with symbolic programming to perform CEP. We consider proxy models consisting of pre-trained neural networks, which allow the system to use non-symbolic inputs. However, the data used to train such proxy models is not necessarily related to the situations of interest (called complex events) we want to detect. Logic rules are used to define under which conditions these complex events occur, based on the output of the proxy models. We also show how the speed of the system can be significantly increased using specific optimization techniques. Then, we show two neuro-symbolic approaches, DeepProbCEP and Neuroplex. These approaches are designed to train a system to identify complex events from an input stream using small amounts of training data. Thanks to the injection of human knowledge into the system, these approaches require significantly less data than neural-only approaches. Following that, we explore the reliability of DeepProbCEP against several adversarial attacks that poison the training data. We also demonstrate that DeepProbCEP is an agile approach, allowing users to change the behaviour of the system to adapt to many situations. Finally, we discuss the potential of the research advances presented in this thesis for real-world applications, as well as possible areas for future research