Visual Human Tracking and Group Activity Analysis: A Video Mining System for Retail Marketing

Thesis (PhD) - Indiana University, Computer Sciences, 2007In this thesis we present a system for automatic human tracking and activity recognition from video sequences. The problem of automated analysis of visual information in order to derive descriptors of high level human activities has intrigued computer vision community for decades and is considered to be largely unsolved. A part of this interest is derived from the vast range of applications in which such a solution may be useful. We attempt to find efficient formulations of these tasks as applied to the extracting customer behavior information in a retail marketing context. Based on these formulations, we present a system that visually tracks customers in a retail store and performs a number of activity analysis tasks based on the output from the tracker. In tracking we introduce new techniques for pedestrian detection, initialization of the body model and a formulation of the temporal tracking as a global trans-dimensional optimization problem. Initial human detection is addressed by a novel method for head detection, which incorporates the knowledge of the camera projection model.The initialization of the human body model is addressed by newly developed shape and appearance descriptors. Temporal tracking of customer trajectories is performed by employing a human body tracking system designed as a Bayesian jump-diffusion filter. This approach demonstrates the ability to overcome model dimensionality ambiguities as people are leaving and entering the scene. Following the tracking, we developed a two-stage group activity formulation based upon the ideas from swarming research. For modeling purposes, all moving actors in the scene are viewed here as simplistic agents in the swarm. This allows to effectively define a set of inter-agent interactions, which combine to derive a distance metric used in further swarm clustering. This way, in the first stage the shoppers that belong to the same group are identified by deterministically clustering bodies to detect short term events and in the second stage events are post-processed to form clusters of group activities with fuzzy memberships. Quantitative analysis of the tracking subsystem shows an improvement over the state of the art methods, if used under similar conditions. Finally, based on the output from the tracker, the activity recognition procedure achieves over 80% correct shopper group detection, as validated by the human generated ground truth results

Latitude, longitude, and beyond:mining mobile objects' behavior

Rapid advancements in Micro-Electro-Mechanical Systems (MEMS), and wireless communications, have resulted in a surge in data generation. Mobility data is one of the various forms of data, which are ubiquitously collected by different location sensing devices. Extensive knowledge about the behavior of humans and wildlife is buried in raw mobility data. This knowledge can be used for realizing numerous viable applications ranging from wildlife movement analysis, to various location-based recommendation systems, urban planning, and disaster relief. With respect to what mentioned above, in this thesis, we mainly focus on providing data analytics for understanding the behavior and interaction of mobile entities (humans and animals). To this end, the main research question to be addressed is: How can behaviors and interactions of mobile entities be determined from mobility data acquired by (mobile) wireless sensor nodes in an accurate and efficient manner? To answer the above-mentioned question, both application requirements and technological constraints are considered in this thesis. On the one hand, applications requirements call for accurate data analytics to uncover hidden information about individual behavior and social interaction of mobile entities, and to deal with the uncertainties in mobility data. Technological constraints, on the other hand, require these data analytics to be efficient in terms of their energy consumption and to have low memory footprint, and processing complexity

Graph Signal Processing: Overview, Challenges and Applications

Research in Graph Signal Processing (GSP) aims to develop tools for processing data defined on irregular graph domains. In this paper we first provide an overview of core ideas in GSP and their connection to conventional digital signal processing. We then summarize recent developments in developing basic GSP tools, including methods for sampling, filtering or graph learning. Next, we review progress in several application areas using GSP, including processing and analysis of sensor network data, biological data, and applications to image processing and machine learning. We finish by providing a brief historical perspective to highlight how concepts recently developed in GSP build on top of prior research in other areas.Comment: To appear, Proceedings of the IEE

Prediction of abnormal behaviors for intelligent video surveillance systems

IEEE Copyright Policies This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. In most cases, these works may not be reposted without the explicit permission of the copyright holder.The OBSERVER is a video surveillance system that detects and predicts abnormal behaviors aiming at the intelligent surveillance concept. The system acquires color images from a stationary video camera and applies state of the art algorithms to segment, track and classify moving objects. In this paper we present the behavior analysis module of the system. A novel method, called Dynamic Oriented Graph (DOG) is used to detect and predict abnormal behaviors, using real-time unsupervised learning. The DOG method characterizes observed actions by means of a structure of unidirectional connected nodes, each one defining a region in the hyperspace of attributes measured from the observed moving objects and having assigned a probability to generate an abnormal behavior. An experimental evaluation with synthetic data was held, where the DOG method outperforms the previously used N-ary Trees classifier.Fundação para a Ciência e a Tecnologia (FCT) - SFRH/BD/17259/2004

Data Mining in Electronic Commerce

Modern business is rushing toward e-commerce. If the transition is done properly, it enables better management, new services, lower transaction costs and better customer relations. Success depends on skilled information technologists, among whom are statisticians. This paper focuses on some of the contributions that statisticians are making to help change the business world, especially through the development and application of data mining methods. This is a very large area, and the topics we cover are chosen to avoid overlap with other papers in this special issue, as well as to respect the limitations of our expertise. Inevitably, electronic commerce has raised and is raising fresh research problems in a very wide range of statistical areas, and we try to emphasize those challenges.Comment: Published at http://dx.doi.org/10.1214/088342306000000204 in the Statistical Science (http://www.imstat.org/sts/) by the Institute of Mathematical Statistics (http://www.imstat.org

Pedestrian, Crowd, and Evacuation Dynamics

This contribution describes efforts to model the behavior of individual pedestrians and their interactions in crowds, which generate certain kinds of self-organized patterns of motion. Moreover, this article focusses on the dynamics of crowds in panic or evacuation situations, methods to optimize building designs for egress, and factors potentially causing the breakdown of orderly motion.Comment: This is a review paper. For related work see http://www.soms.ethz.c

Mining human mobility patterns from pervasive spatial and temporal data

Recent advances in communication, sensors and processors have made pervasive systems more computationally powerful and increasingly popular. These systems are deployed everywhere all the time while remaining transparent. Take smartphones as an example; they have become an integral part of human life and people carry them wherever they go. Coupled with the popularity of pervasive systems and user tracking, this has opened up excellent opportunities to analyse human mobility. This can be applied to a broad range of location-based services such as smart navigation and recommendation systems. Data from pervasive systems has temporal, spatial and spatio-temporal aspects that can be leveraged for mining human mobility patterns. Temporal data such as time series from embedded sensors on smartphones does not usually have any information about locations, while time stamps are discarded in spatial data. The list of significant locations visited by the user is an example of spatial data. The third group of data is spatio-temporal data that has both temporal and spatial aspects such as users' trajectories. In this dissertation, we analyse human mobility by mining these three kinds of data. In each chapter, we look at a specific aspect to infer key information about users’ mobility including transition time detection, movement graph summarisation, and trajectory prediction. We analyse temporal information from time series data to extract transition times in daily activities. The transition times denote when user activities change such as when the user goes to work or when the user goes shopping. In addition to applications in location-based services, extracting the transition times helps us to understand human mobility patterns across the whole day. We tackle scalability to enable processing to take place on resource-constrained devices. We introduce Shrink as a new summarisation method to compress large scale graphs. Trajectories and movements of the user can be transformed into a graph in which each node represents stay points and each edge represents distance. Since this graph is very large, Shrink is used to reduce the size of the movement graph while preserving distances between nodes. The property that is preserved in the compressed graph, also known as the coarse graph, is the distance between the nodes. Shrink is a query friendly compression, which means the compressed graph can be queried without decompression. As the complexity of distance-based queries such as shortest path queries is highly dependent on the size of the graph, Shrink improves performance in terms of time and storage. We also investigate the effect of compression on the human mobility mining algorithms and show that the summarisation provides a trade-off between efficiency and granularity. We also analyse spatial-temporal data by predicting user trajectory based on historical data. Specifically, given the historical data and the user’s trajectory in the first part of the current day (e.g. trajectory in the morning), we predict how users will complete their trajectory in that particular day (e.g. predicting the trajectory for the rest of the day or the afternoon). The granularity of the predicted trajectory is the same as the granularity of the given trajectories. We emphasize that the predicted trajectory includes the sequence of future locations, the stay times, and the departure times. This enhances the user experience because by having the detailed trajectory in advance, location-based services can notify users about the consequence of the movement. In summary, this thesis contains efficient algorithms that can be applied to diverse aspects of pervasive signals for mining human mobility. The new algorithms are aimed at problems in transition time detection, summarisation, and prediction. The solutions address the scalability issues and can work in big pervasive temporal and spatial data effectively and accurately

Semantic Trajectories: Mobility Data Computation and Annotation

With the large-scale adoption of GPS equipped mobile sensing devices, positional data generated by moving objects (e.g., vehicles, people, animals) are being easily collected. Such data are typically modeled as streams of spatio-temporal (x,y,t) points, called ''trajectories''. In recent years trajectory management research has progressed significantly towards efficient storage and indexing techniques, as well as suitable knowledge discovery. These works focused on the geometric aspect of the raw mobility data. We are now witnessing a growing demand in several application sectors (e.g., from shipment tracking to geo-social networks) on understanding the {\it semantic'' behavior of moving objects. Semantic behavior refers to the use of semantic abstractions of the raw mobility data, including not only geometric patterns but also knowledge extracted jointly from the mobility data and the underlying geographic and application domains information. The core contribution of this paper lies in a ''Semantic Model'' and a ''Computation and Annotation Platform'' for developing a semantic approach that progressively transforms the raw mobility data into semantic trajectories enriched with annotations and segmentations. We also analyze a number of experiments we did with semantic trajectories in different domains