Pattern Recognition for Command and Control Data Systems

To analyze real-world events, researchers collect observation data from an underlying process and construct models to represent the observed situation. In this work, we consider issues that affect the construction and usage of a specific type of model. Markov models are commonly used because their combination of discrete states and stochastic transitions is suited to applications with both deterministic and stochastic components. Hidden Markov Models (HMMs) are a class of Markov model commonly used in pattern recognition. We first demonstrate how to construct HMMs using only the observation data, and no a priori information, by extending a previously developed approach from J.P. Crutchfield and C.R. Shalizi. We also show how to determine with a level of statistical confidence whether or not the model fully encapsulates the underlying process. Once models are constructed from observation data, the models are used to identify other types of observations. Traditional approaches consider the maximum likelihood that the model matches the observation, solving a classification problem. We present a new method using confidence intervals and receiver operating characteristic curves. Our method solves a detection problem by determining if observation data matches zero, one, or more than one model. To detect the occurrence of a behavior in observation data, one must consider the amount of data required. We consider behaviors to be \u27serial Markovian,\u27 when the behavior can change from one model to another at any time. When analyzing observation data, considering too much data induces high delay and could lead to confusion in the system if multiple behaviors are observed in the data stream. If too little data is used, the system has a high false positive rate and is unable to correctly detect behaviors. We demonstrate the effectiveness of all methods using illustrative examples and consumer behavior data

Using Markov Models and Statistics to Learn, Extract, Fuse, and Detect Patterns in Raw Data

Many systems are partially stochastic in nature. We have derived data driven approaches for extracting stochastic state machines (Markov models) directly from observed data. This chapter provides an overview of our approach with numerous practical applications. We have used this approach for inferring shipping patterns, exploiting computer system side-channel information, and detecting botnet activities. For contrast, we include a related data-driven statistical inferencing approach that detects and localizes radiation sources.Comment: Accepted by 2017 International Symposium on Sensor Networks, Systems and Securit

An early warning indicator for atmospheric blocking events using transfer operators

The existence of persistent midlatitude atmospheric flow regimes with time-scales larger than 5-10 days and indications of preferred transitions between them motivates to develop early warning indicators for such regime transitions. In this paper, we use a hemispheric barotropic model together with estimates of transfer operators on a reduced phase space to develop an early warning indicator of the zonal to blocked flow transition in this model. It is shown that, the spectrum of the transfer operators can be used to study the slow dynamics of the flow as well as the non-Markovian character of the reduction. The slowest motions are thereby found to have time scales of three to six weeks and to be associated with meta-stable regimes (and their transitions) which can be detected as almost-invariant sets of the transfer operator. From the energy budget of the model, we are able to explain the meta-stability of the regimes and the existence of preferred transition paths. Even though the model is highly simplified, the skill of the early warning indicator is promising, suggesting that the transfer operator approach can be used in parallel to an operational deterministic model for stochastic prediction or to assess forecast uncertainty

Statistical identification with hidden Markov models of large order splitting strategies in an equity market

Large trades in a financial market are usually split into smaller parts and traded incrementally over extended periods of time. We address these large trades as hidden orders. In order to identify and characterize hidden orders we fit hidden Markov models to the time series of the sign of the tick by tick inventory variation of market members of the Spanish Stock Exchange. Our methodology probabilistically detects trading sequences, which are characterized by a net majority of buy or sell transactions. We interpret these patches of sequential buying or selling transactions as proxies of the traded hidden orders. We find that the time, volume and number of transactions size distributions of these patches are fat tailed. Long patches are characterized by a high fraction of market orders and a low participation rate, while short patches have a large fraction of limit orders and a high participation rate. We observe the existence of a buy-sell asymmetry in the number, average length, average fraction of market orders and average participation rate of the detected patches. The detected asymmetry is clearly depending on the local market trend. We also compare the hidden Markov models patches with those obtained with the segmentation method used in Vaglica {\it et al.} (2008) and we conclude that the former ones can be interpreted as a partition of the latter ones.Comment: 26 pages, 12 figure

Experiment-friendly kinetic analysis of single molecule data in and out of equilibrium

We present a simple and robust technique to extract kinetic rate models and thermodynamic quantities from single molecule time traces. SMACKS (Single Molecule Analysis of Complex Kinetic Sequences) is a maximum likelihood approach that works equally well for long trajectories as for a set of short ones. It resolves all statistically relevant rates and also their uncertainties. This is achieved by optimizing one global kinetic model based on the complete dataset, while allowing for experimental variations between individual trajectories. In particular, neither a priori models nor equilibrium have to be assumed. The power of SMACKS is demonstrated on the kinetics of the multi-domain protein Hsp90 measured by smFRET (single molecule F\"orster resonance energy transfer). Experiments in and out of equilibrium are analyzed and compared to simulations, shedding new light on the role of Hsp90's ATPase function. SMACKS pushes the boundaries of single molecule kinetics far beyond current methods.Comment: 11 pages, 8 figure