Finding unexplained activities in time-stamped observation data

The activity recognition is a very big challenge for the entire research community. Thus, there are already numerous techniques able to find occurrences of activities in time-stamped observation data (e.g., a video, a sequence of transactions at a website, etc.) with each occurrence having an associated probability. However, all these techniques rely on models encoding a priori knowledge of either normal or malicious behavior. They cannot deal with events such as “zero day” attacks that have never been seen before. In practice, all these methods are incapable of quantifying how well available models explain a sequence of events observed in an observation stream. By the way, the goal of this thesis is different: in order to address the issue listed above, we want to find the subsequences of the observation data, called unexplained sequences, that known models are not able to “explain” with a certain confidence. Thus, we start with a known set A of activities (both innocuous and dangerous) that we wish to monitor and we wish to identify “unexplained” subsequences in an observation sequence that are poorly explained (e.g., because they may contain occurrences of activities that have never been seen or anticipated before, i.e. they are not in A). We formally define the probability that a sequence of observations is unexplained (totally or partially) w.r.t. A. We develop efficient algorithms to identify the top-k Totally and Partially Unexplained Activities w.r.t. A. These algorithms leverage theorems that enable us to speed up the search for totally/partially unexplained activities. We describe experiments using real-world video and cyber security datasets showing that our approach works well in practice in terms of both running time and accuracy