Visual Human Tracking and Group Activity Analysis: A Video Mining System for Retail Marketing

Thesis (PhD) - Indiana University, Computer Sciences, 2007In this thesis we present a system for automatic human tracking and activity recognition from video sequences. The problem of automated analysis of visual information in order to derive descriptors of high level human activities has intrigued computer vision community for decades and is considered to be largely unsolved. A part of this interest is derived from the vast range of applications in which such a solution may be useful. We attempt to find efficient formulations of these tasks as applied to the extracting customer behavior information in a retail marketing context. Based on these formulations, we present a system that visually tracks customers in a retail store and performs a number of activity analysis tasks based on the output from the tracker. In tracking we introduce new techniques for pedestrian detection, initialization of the body model and a formulation of the temporal tracking as a global trans-dimensional optimization problem. Initial human detection is addressed by a novel method for head detection, which incorporates the knowledge of the camera projection model.The initialization of the human body model is addressed by newly developed shape and appearance descriptors. Temporal tracking of customer trajectories is performed by employing a human body tracking system designed as a Bayesian jump-diffusion filter. This approach demonstrates the ability to overcome model dimensionality ambiguities as people are leaving and entering the scene. Following the tracking, we developed a two-stage group activity formulation based upon the ideas from swarming research. For modeling purposes, all moving actors in the scene are viewed here as simplistic agents in the swarm. This allows to effectively define a set of inter-agent interactions, which combine to derive a distance metric used in further swarm clustering. This way, in the first stage the shoppers that belong to the same group are identified by deterministically clustering bodies to detect short term events and in the second stage events are post-processed to form clusters of group activities with fuzzy memberships. Quantitative analysis of the tracking subsystem shows an improvement over the state of the art methods, if used under similar conditions. Finally, based on the output from the tracker, the activity recognition procedure achieves over 80% correct shopper group detection, as validated by the human generated ground truth results

Building a person home behavior model based on data from smart house sensors 2015

Dissertação para a obtenção do grau de Mestre em Engenharia Electrotécnica Ramo de EnergiaO propósito deste trabalho é construir um modelo baseado em informação captada por sensores e câmaras em uma casa onde viva somente uma pessoa. A primeira parte do trabalho consistiu em reunir informação que ajuda-se a perceber o que já tinha sido feito neste âmbito por outros engenheiros. Na realização deste trabalho, a parte que se revelou mais acessível foi a que envolveu a procura de informação sobre os diferentes métodos a usar no sistema, tais como Lógica Fuzzy, Redes Neuronais ou o método Monte Carlo via Cadeias de Markov. Em contrapartida, investigar as diferentes maneiras de guardar a informação provou ser mais desafiante. Foi praticamente impossível encontrar informação sobre os métodos usados para guardar informação recolhida pelos sensores e câmaras. A maior parte das vezes, os engenheiros não mencionavam a parte da informação relativa a este aspecto, simplesmente apresentavam os seus resultados e as suas conclusões. Além disso, quando o uso de informação era mencionada, a mesma relacionava-se com informação online, ou só informação já recolhida por sensores e câmaras, sem quaisquer detalhes. Era espectável encontrar mais informação sobre os métodos de guardar informação, mais específica, como a informação recolhida era ligada entre si e como o sistema iria interpretá-la e que resposta iria dar. Devido à falta de informação sobre como armazenar a informação recolhida, ficou decidido que se teria de assumir uma maneira para o fazer e com isso construir o modelo. Este modelo será referido neste documento. Devido a problemas relacionados com a parte da programação e com a falta de tempo, foi impossível construir o modelo usando os, considerados, melhores métodos. Não obstante, foi possível decidir as melhores opções para construir o modelo, assim como algumas formas de o fazer. Foi decidido a maneira como armazenar informação, o protocolo de informação a usar e o método que irá inferir com as actividades e comportamentos do habitante.Abstract:The purpose of this work has been to build a model based on data collected from sensors and cameras in a house with only one inhabitant. The first part of the work consisted of gathering research in order to try to understand what was already made by other engineers. One part stood out to be less complicated, as it evolved around finding information about the different methods to use with a system like Fuzzy Logic, Neural Network or Markov Chain Monte Carlo. However, investigating different ways to store information proved to be more challenging. It was pretty much impossible to get some information about the way people store the information collected from sensors and cameras. Most of the time, other engineers never mention the part related with the data, but simply presented the results and then their conclusion. Moreover, when the use of data was mentioned, it was simply related to online data, or just data which was stored after being collected from sensors and cameras, without any further detail. It was expected to find more information about the way the data was used, more specific information, covering how all the information was connected to each other, and how the system would interpret all the data and the responses. Since the lack of information related with the data, it was decided to assume a way to store the information and with that, a model was built. This model will be referred to within this paper. Due to problems with programming and lack of time, it was impossible to build a model by using the best methods. Notwithstanding, it was possible to decide all the best options to use to build the model, along with some ways to do it. It was decided the way to store information, the Communication Protocol to use and the method to infer with the inhabitant activities and behaviors

Hybrid structural health monitoring using data-driven modal analysis and model-based Bayesian inference.

Civil infrastructures that are valuable assets for the public and owners must be adequately and periodically maintained to guarantee safety, continuous service, and avoid economic losses. Vibration-based structural health monitoring (VBSHM) has been a significant tool to assess the structural performance of civil infrastructures over the last decades. Challenges in VBSHM exist in two aspects: operational modal analysis (OMA) and Finite element model updating (FEMU). The former aims to extract natural frequency, damping ratio, and mode shapes using vibrational data under normal operation; the latter focuses on minimizing the discrepancies between measurements and model prediction. The main impediments to real-world application of VBSHM include 1) uncertainties are inevitably involved due to measurement noise and modeling error; 2) computational burden in analyzing massive data and high-fidelity model; 3) updating structural coupled parameters, e.g., mass and stiffness. Bayesian model updating approach (BMUA) is an advanced FEMU technique to update structural parameters using modal data and account for underlying uncertainties. However, traditional BMUA generally assumes mass is precisely known and only updating stiffness to circumvent the coupling effect of mass and stiffness. Simultaneously updating mass and stiffness is necessary to fully understand the structural integrity, especially when the mass has a relatively large variation. To tackle these challenges, this dissertation proposed a hybrid framework using data-driven and model-based approaches in two sequential phases: automated OMA and a BMUA with added mass/stiffness. Automated stochastic subspace identification (SSI) and Bayesian modal identification are firstly developed to acquire modal properties. Following by a novel BMUA, new eigen-equations based on two sets of modal data from the original and modified system with added mass or stiffness are derived to address the coupling effect of structural parameters, e.g., mass and stiffness. To avoid multi-dimensional integrals, an asymptotic optimization method and Differential Evolutionary Adaptive Metropolis (DREAM) sampling algorithm are employed for Bayesian inference. To alleviate computational burden, variance-based global sensitivity analysis to reduce model dimensionality and Kriging model to substitute time-consuming FEM are integrated into BMUA. The proposed VBSHM are verified and illustrated using numerical, laboratory and field test data, achieving following goals: 1) properly treating parameter uncertainties; 2) substantially reducing the computational cost; 3) simultaneously updating structural parameters with addressing the coupling effect; 4) performing the probabilistic damage identification at an accurate level

AN INFORMATION THEORETIC APPROACH TO INTERACTING MULTIPLE MODEL ESTIMATION FOR AUTONOMOUS UNDERWATER VEHICLES

Accurate and robust autonomous underwater navigation (AUV) requires the fundamental task of position estimation in a variety of conditions. Additionally, the U.S. Navy would prefer to have systems that are not dependent on external beacon systems such as global positioning system (GPS), since they are subject to jamming and spoofing and can reduce operational effectiveness. Current methodologies such as Terrain-Aided Navigation (TAN) use exteroceptive imaging sensors for building a local reference position estimate and will not be useful when those sensors are out of range. What is needed are multiple navigation filters where each can be more effective depending on the mission conditions. This thesis investigates how to combine multiple navigation filters to provide a more robust AUV position estimate. The solution presented is to blend two different filtering methodologies utilizing an interacting multiple model (IMM) estimation approach based on an information theoretic framework. The first filter is a model-based Extended Kalman Filter (EKF) that is effective under dead reckoning (DR) conditions. The second is a Particle Filter approach for Active Terrain Aided Navigation (ATAN) that is appropriate when in sensor range. Using data collected at Lake Crescent, Washington, each of the navigation filters are developed with results and then we demonstrate how an IMM information theoretic approach can be used to blend approaches to improve position and orientation estimation.Lieutenant, United States NavyApproved for public release. Distribution is unlimited

Online Visual Robot Tracking and Identification using Deep LSTM Networks

Collaborative robots working on a common task are necessary for many applications. One of the challenges for achieving collaboration in a team of robots is mutual tracking and identification. We present a novel pipeline for online visionbased detection, tracking and identification of robots with a known and identical appearance. Our method runs in realtime on the limited hardware of the observer robot. Unlike previous works addressing robot tracking and identification, we use a data-driven approach based on recurrent neural networks to learn relations between sequential inputs and outputs. We formulate the data association problem as multiple classification problems. A deep LSTM network was trained on a simulated dataset and fine-tuned on small set of real data. Experiments on two challenging datasets, one synthetic and one real, which include long-term occlusions, show promising results.Comment: IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Vancouver, Canada, 2017. IROS RoboCup Best Paper Awar

Electron spin resonance of nitrogen-vacancy centers in optically trapped nanodiamonds

Using an optical tweezers apparatus, we demonstrate three-dimensional control of nanodiamonds in solution with simultaneous readout of ground-state electron-spin resonance (ESR) transitions in an ensemble of diamond nitrogen-vacancy (NV) color centers. Despite the motion and random orientation of NV centers suspended in the optical trap, we observe distinct peaks in the measured ESR spectra qualitatively similar to the same measurement in bulk. Accounting for the random dynamics, we model the ESR spectra observed in an externally applied magnetic field to enable d.c. magnetometry in solution. We estimate the d.c. magnetic field sensitivity based on variations in ESR line shapes to be ~50 microTesla/Hz^1/2. This technique may provide a pathway for spin-based magnetic, electric, and thermal sensing in fluidic environments and biophysical systems inaccessible to existing scanning probe techniques.Comment: 29 pages, 13 figures for manuscript and supporting informatio