Multi-Observation Continuous Density Hidden Markov Models for Anomaly Detection in Full Motion Video

An increase in sensors on the battlefield produces an abundance of collected data that overwhelms the processing capability of the DoD. Automated Visual Surveillance (AVS) seeks to use machines to better exploit increased sensor data, such as by highlighting anomalies. In this thesis, we apply AVS to overhead Full Motion Video (FMV). We seek to automate the classification of soldiers in a simulated combat scenario into their agent types. To this end, we use Multi-Dimensional Continuous Density Hidden Markov Models (MOCDHMMs), a form of HMM which models a training dataset more precisely than simple HMMs. MOCDHMMs are theoretically developed but thinly applied in literature. We discover and correct three errors which occur in HMM algorithms when applied to MOCDHMMs but not when applied to simple HMMs. We offer three fixes to the errors and show analytically why they work. To show the fixes effective, we conduct experiments on three datasets: two pilot experiment datasets and a simulated combat scenario dataset. The modified MOCDHMM algorithm gives statistically significant improvement over the standard MOCDHMM: 5% improvement in accuracy for the pilot datasets and 3% for the combat scenario dataset. In addition, results suggest that increasing the number of hidden states in an MOCDHMM classifier increases the separability of the classes but also increases classifier bias. Furthermore, we find that classification based on tracked position alone is possible and that MOCDHMM classifiers are highly resistant to noise in their training data

Movement Analytics: Current Status, Application to Manufacturing, and Future Prospects from an AI Perspective

Data-driven decision making is becoming an integral part of manufacturing companies. Data is collected and commonly used to improve efficiency and produce high quality items for the customers. IoT-based and other forms of object tracking are an emerging tool for collecting movement data of objects/entities (e.g. human workers, moving vehicles, trolleys etc.) over space and time. Movement data can provide valuable insights like process bottlenecks, resource utilization, effective working time etc. that can be used for decision making and improving efficiency. Turning movement data into valuable information for industrial management and decision making requires analysis methods. We refer to this process as movement analytics. The purpose of this document is to review the current state of work for movement analytics both in manufacturing and more broadly. We survey relevant work from both a theoretical perspective and an application perspective. From the theoretical perspective, we put an emphasis on useful methods from two research areas: machine learning, and logic-based knowledge representation. We also review their combinations in view of movement analytics, and we discuss promising areas for future development and application. Furthermore, we touch on constraint optimization. From an application perspective, we review applications of these methods to movement analytics in a general sense and across various industries. We also describe currently available commercial off-the-shelf products for tracking in manufacturing, and we overview main concepts of digital twins and their applications

Video anomaly detection using deep generative models

Video anomaly detection faces three challenges: a) no explicit definition of abnormality; b) scarce labelled data and c) dependence on hand-crafted features. This thesis introduces novel detection systems using unsupervised generative models, which can address the first two challenges. By working directly on raw pixels, they also bypass the last

Predictive models of human supervisory control behavioral patterns using hidden semi-Markov models

Behavioral models of human operators engaged in complex,time-critical high-risk domains, such as those typical in Human Supervisory Control (HSC) settings, are of great value because of the high cost of operator failure. We propose that Hidden Semi-Markov Models (HSMMs) can be employed to model behaviors of operators in HSC settings where there is some intermittent human interaction with a system via a set of external controls. While regular Hidden Markov Models (HMMs) can be used to model operator behavior, HSMMs are particularly suited to time-critical supervisory control domains due to their explicit representation of state duration. Using HSMMs,we demonstrate in an unmanned vehicle supervisory control environment that such models can accurately predict future operator behavior both in terms of states and durations.This research was sponsored by the Boeing Research and Technology and the Office of Naval Research

A computational framework for unsupervised analysis of everyday human activities

In order to make computers proactive and assistive, we must enable them to perceive, learn, and predict what is happening in their surroundings. This presents us with the challenge of formalizing computational models of everyday human activities. For a majority of environments, the structure of the in situ activities is generally not known a priori. This thesis therefore investigates knowledge representations and manipulation techniques that can facilitate learning of such everyday human activities in a minimally supervised manner. A key step towards this end is finding appropriate representations for human activities. We posit that if we chose to describe activities as finite sequences of an appropriate set of events, then the global structure of these activities can be uniquely encoded using their local event sub-sequences. With this perspective at hand, we particularly investigate representations that characterize activities in terms of their fixed and variable length event subsequences. We comparatively analyze these representations in terms of their representational scope, feature cardinality and noise sensitivity. Exploiting such representations, we propose a computational framework to discover the various activity-classes taking place in an environment. We model these activity-classes as maximally similar activity-cliques in a completely connected graph of activities, and describe how to discover them efficiently. Moreover, we propose methods for finding concise characterizations of these discovered activity-classes, both from a holistic as well as a by-parts perspective. Using such characterizations, we present an incremental method to classify a new activity instance to one of the discovered activity-classes, and to automatically detect if it is anomalous with respect to the general characteristics of its membership class. Our results show the efficacy of our framework in a variety of everyday environments.Ph.D.Committee Chair: Aaron Bobick; Committee Member: Charles Isbell; Committee Member: David Hogg; Committee Member: Irfan Essa; Committee Member: James Reh

Temporal Signature Modeling and Analysis

A vast amount of digital satellite and aerial images are collected over time, which calls for techniques to extract useful high-level information, such as recognizable events. One part of this thesis proposes a framework for streaming analysis of the time series, which can recognize events without supervision and memorize them by building the temporal contexts. The memorized historical data is then used to predict the future and detect anomalies. A new incremental clustering method is proposed to recognize the event without training. A memorization method of double localization, including relative and absolute localization, is proposed to model the temporal context. Finally, the predictive model is built based on the method of memorization. The Edinburgh Pedestrian Dataset , which offers about 1000 observed trajectories of pedestrians detected in camera images each working day for several months, is used as an example to illustrate the framework. Although there is a large amount of image data captured, most of them are not available to the public. The other part of this thesis developed a method of generating spatial-spectral-temporal synthetic images by enhancing the capacity of a current tool called DIRISG (Digital Imaging and Remote Sensing Image Generation). Currently, DIRSIG can only model limited temporal signatures. In order to observe general temporal changes in a process within the scene, a process model, which links the observable signatures of interest temporally, should be developed and incorporated into DIRSIG. The sub process models could be categorized into two types. One is that the process model drives the property of each facet of the object changing over time, and the other one is to drive the geometry location of the object in the scene changing as a function of time. Two example process models are used to show how process models can be incorporated into DIRSIG

Evolutionary Inference from Admixed Genomes: Implications of Hybridization for Biodiversity Dynamics and Conservation

Hybridization as a macroevolutionary mechanism has been historically underappreciated among vertebrate biologists. Yet, the advent and subsequent proliferation of next-generation sequencing methods has increasingly shown hybridization to be a pervasive agent influencing evolution in many branches of the Tree of Life (to include ancestral hominids). Despite this, the dynamics of hybridization with regards to speciation and extinction remain poorly understood. To this end, I here examine the role of hybridization in the context of historical divergence and contemporary decline of several threatened and endangered North American taxa, with the goal to illuminate implications of hybridization for promoting—or impeding—population persistence in a shifting adaptive landscape. Chapter I employed population genomic approaches to examine potential effects of habitat modification on species boundary stability in co-occurring endemic fishes of the Colorado River basin (Gila robusta and G. cypha). Results showed how one potential outcome of hybridization might drive species decline: via a breakdown in selection against interspecific heterozygotes and subsequent genetic erosion of parental species. Chapter II explored long-term contributions of hybridization in an evolutionarily recent species complex (Gila) using a combination of phylogenomic and phylogeographic modelling approaches. Massively parallel computational methods were developed (and so deployed) to categorize sources of phylogenetic discordance as drivers of systematic bias among a panel of species tree inference algorithms. Contrary to past evidence, we found that hypotheses of hybrid origin (excluding one notable example) were instead explained by gene-tree discordance driven by a rapid radiation. Chapter III examined patterns of local ancestry in the endangered red wolf genome (Canis rufus) – a controversial taxon of a long-standing debate about the origin of the species. Analyses show how pervasive autosomal introgression served to mask signatures of prior isolation—in turn misleading analyses that led the species to be interpreted as of recent hybrid origin. Analyses also showed how recombination interacts with selection to create a non-random, structured genomic landscape of ancestries with, in the case of the red wolf, the ‘original’ species tree being retained only in low-recombination ‘refugia’ of the X chromosome. The final three chapters present bioinformatic software that I developed for my dissertation research to facilitate molecular approaches and analyses presented in Chapters I–III. Chapter IV details an in-silico method for optimizing similar genomic methods as used herein (RADseq of reduced representation libraries) for other non-model organisms. Chapter V describes a method for parsing genomic datasets for elements of interest, either as a filtering mechanism for downstream analysis, or as a precursor to targeted-enrichment reduced-representation genomic sequencing. Chapter VI presents a rapid algorithm for the definition of a ‘most parsimonious’ set of recombinational breakpoints in genomic datasets, as a method promoting local ancestry analyses as utilized in Chapter III. My three case studies and accompanying software promote three trajectories in modern hybridization research: How does hybridization impact short-term population persistence? How does hybridization drive macroevolutionary trends? and How do outcomes of hybridization vary in the genome? In so doing, my research promotes a deeper understanding of the role that hybridization has and will continue to play in governing the evolutionary fates of lineages at both contemporary and historic timescales