Human activity recognition for pervasive interaction

PhD ThesisThis thesis addresses the challenge of computing food preparation context in the kitchen. The automatic recognition of fine-grained human activities and food ingredients is realized through pervasive sensing which we achieve by instrumenting kitchen objects such as knives, spoons, and chopping boards with sensors. Context recognition in the kitchen lies at the heart of a broad range of real-world applications. In particular, activity and food ingredient recognition in the kitchen is an essential component for situated services such as automatic prompting services for cognitively impaired kitchen users and digital situated support for healthier eating interventions. Previous works, however, have addressed the activity recognition problem by exploring high-level-human activities using wearable sensing (i.e. worn sensors on human body) or using technologies that raise privacy concerns (i.e. computer vision). Although such approaches have yielded significant results for a number of activity recognition problems, they are not applicable to our domain of investigation, for which we argue that the technology itself must be genuinely “invisible”, thereby allowing users to perform their activities in a completely natural manner. In this thesis we describe the development of pervasive sensing technologies and algorithms for finegrained human activity and food ingredient recognition in the kitchen. After reviewing previous work on food and activity recognition we present three systems that constitute increasingly sophisticated approaches to the challenge of kitchen context recognition. Two of these systems, Slice&Dice and Classbased Threshold Dynamic Time Warping (CBT-DTW), recognize fine-grained food preparation activities. Slice&Dice is a proof-of-concept application, whereas CBT-DTW is a real-time application that also addresses the problem of recognising unknown activities. The final system, KitchenSense is a real-time context recognition framework that deals with the recognition of a more complex set of activities, and includes the recognition of food ingredients and events in the kitchen. For each system, we describe the prototyping of pervasive sensing technologies, algorithms, as well as real-world experiments and empirical evaluations that validate the proposed solutions.Vietnamese government’s 322 project, executed by the Vietnamese Ministry of Education and Training

Spatiotemporal analysis of human actions using RGB-D cameras

Markerless human motion analysis has strong potential to provide cost-efficient solution for action recognition and body pose estimation. Many applications including humancomputer interaction, video surveillance, content-based video indexing, and automatic annotation among others will benefit from a robust solution to these problems. Depth sensing technologies in recent years have positively changed the climate of the automated vision-based human action recognition problem, deemed to be very difficult due to the various ambiguities inherent to conventional video. In this work, first a large set of invariant spatiotemporal features is extracted from skeleton joints (retrieved from depth sensor) in motion and evaluated as baseline performance. Next we introduce a discriminative Random Decision Forest-based feature selection framework capable of reaching impressive action recognition performance when combined with a linear SVM classifier. This approach improves upon the baseline performance obtained using the whole feature set with a significantly less number of features (one tenth of the original). The approach can also be used to provide insights on the spatiotemporal dynamics of human actions. A novel therapeutic action recognition dataset (WorkoutSU-10) is presented. We took advantage of this dataset as a benchmark in our tests to evaluate the reliability of our proposed methods. Recently the dataset has been published publically as a contribution to the action recognition community. In addition, an interactive action evaluation application is developed by utilizing the proposed methods to help with real life problems such as 'fall detection' in the elderly people or automated therapy program for patients with motor disabilities

Behaviour Profiling using Wearable Sensors for Pervasive Healthcare

In recent years, sensor technology has advanced in terms of hardware sophistication and miniaturisation. This has led to the incorporation of unobtrusive, low-power sensors into networks centred on human participants, called Body Sensor Networks. Amongst the most important applications of these networks is their use in healthcare and healthy living. The technology has the possibility of decreasing burden on the healthcare systems by providing care at home, enabling early detection of symptoms, monitoring recovery remotely, and avoiding serious chronic illnesses by promoting healthy living through objective feedback. In this thesis, machine learning and data mining techniques are developed to estimate medically relevant parameters from a participant‘s activity and behaviour parameters, derived from simple, body-worn sensors. The first abstraction from raw sensor data is the recognition and analysis of activity. Machine learning analysis is applied to a study of activity profiling to detect impaired limb and torso mobility. One of the advances in this thesis to activity recognition research is in the application of machine learning to the analysis of 'transitional activities': transient activity that occurs as people change their activity. A framework is proposed for the detection and analysis of transitional activities. To demonstrate the utility of transition analysis, we apply the algorithms to a study of participants undergoing and recovering from surgery. We demonstrate that it is possible to see meaningful changes in the transitional activity as the participants recover. Assuming long-term monitoring, we expect a large historical database of activity to quickly accumulate. We develop algorithms to mine temporal associations to activity patterns. This gives an outline of the user‘s routine. Methods for visual and quantitative analysis of routine using this summary data structure are proposed and validated. The activity and routine mining methodologies developed for specialised sensors are adapted to a smartphone application, enabling large-scale use. Validation of the algorithms is performed using datasets collected in laboratory settings, and free living scenarios. Finally, future research directions and potential improvements to the techniques developed in this thesis are outlined

Early Abstraction of Inertial Sensor Data for Long-Term Deployments

Advances in microelectronics over the last decades have led to miniaturization of computing devices and sensors. A driving force to use these in various application scenarios is the desire to grasp physical phenomena from the environment, objects and living entities. We investigate sensing in two particularly challenging applications: one where small sensor modules are worn by people to detect their activities, and one where wirelessly networked sensors observe events over an area. This thesis takes a data-driven approach, focusing on human motion and vibrations caused by trains that are captured by accelerometer sensors as time series and shall be analyzed for characteristic patterns. For both, the acceleration sensor must be sampled at relatively high rates in order to capture the essence of the phenomena, and remain active for long stretches of time. The large amounts of gathered sensor data demand novel approaches that are able to swiftly process the data while guaranteeing accurate classification results. The following contributions are made in particular: * A data logger that would suit the requirements of long-term deployments is designed and evaluated. In a power profiling study both hardware components and firmware parameters are thoroughly tested, revealing that the sensor is able to log acceleration data at a sampling rate of 100 Hertz for up to 14 full days on a single battery charge. * A technique is proposed that swiftly and accurately abstracts an original signal with a set of linear segments, thus preserving its shape, while being twice as fast as a similar method. This allows for more efficient pattern matching, since for each pattern only a fraction of data points must be considered. A second study shows that this algorithm can perform data abstraction directly on a data logger with limited resources. * The railway monitoring scenario requires streaming vibration data to be analyzed for particular sparse and complex events directly on the sensor node, extracting relevant information such as train type or length from the shape of the vibration footprint. In a study conducted on real-world data, a set of efficient shape features is identified that facilitates train type prediction and length estimation with very high accuracies. * To achieve fast and accurate activity recognition for long-term bipolar patients monitoring scenarios, we present an approach that relies on the salience of motion patterns (motifs) that are characteristic for the target activity. These motifs are accumulated by using a symbolic abstraction that encodes the shape of the original signal. A large-scale study shows that a simple bag-of-words classifier trained with extracted motifs is on par with traditional approaches regarding the accuracy, while being much faster. * Some activities are hard to predict from acceleration data alone with the aforementioned approach. We argue that human-object interactions, captured as human motion and grasped objects through RFID, are an ideal supplement. A custom bracelet-like antenna to detect objects from up to 14 cm is proposed, along with a novel benchmark to evaluate such wearable setups. By aiming for wearable and wirelessly networked sensor systems, these contributions apply for particularly challenging applications that require long-term deployments of miniature sensors in general. They form the basis of a framework towards efficient event detection that relies heavily on early data abstraction and shape-based features for time series, while focusing less on the classification techniques

Activity recognition in naturalistic environments using body-worn sensors

Phd ThesisThe research presented in this thesis investigates how deep learning and feature learning can address challenges that arise for activity recognition systems in naturalistic, ecologically valid surroundings such as the private home. One of the main aims of ubiquitous computing is the development of automated recognition systems for human activities and behaviour that are sufficiently robust to be deployed in realistic, in-the-wild environments. In most cases, the targeted application scenarios are people’s daily lives, where systems have to abide by practical usability and privacy constraints. We discuss how these constraints impact data collection and analysis and demonstrate how common approaches to the analysis of movement data effectively limit the practical use of activity recognition systems in every-day surroundings. In light of these issues we develop a novel approach to the representation and modelling of movement data based on a data-driven methodology that has applications in activity recognition, behaviour imaging, and skill assessment in ubiquitous computing. A number of case studies illustrate the suitability of the proposed methods and outline how study design can be adapted to maximise the benefit of these techniques, which show promising performance for clinical applications in particular.SiDE research hu

WSN based sensing model for smart crowd movement with identification: a conceptual model

With the advancement of IT and increase in world population rate, Crowd Management (CM) has become a subject undergoing intense study among researchers. Technology provides fast and easily available means of transport and, up-to-date information access to the people that causes crowd at public places. This imposes a big challenge for crowd safety and security at public places such as airports, railway stations and check points. For example, the crowd of pilgrims during Hajj and Ummrah while crossing the borders of Makkah, Kingdom of Saudi Arabia. To minimize the risk of such crowd safety and security identification and verification of people is necessary which causes unwanted increment in processing time. It is observed that managing crowd during specific time period (Hajj and Ummrah) with identification and verification is a challenge. At present, many advanced technologies such as Internet of Things (IoT) are being used to solve the crowed management problem with minimal processing time. In this paper, we have presented a Wireless Sensor Network (WSN) based conceptual model for smart crowd movement with minimal processing time for people identification. This handles the crowd by forming groups and provides proactive support to handle them in organized manner. As a result, crowd can be managed to move safely from one place to another with group identification. The group identification minimizes the processing time and move the crowd in smart way