Multi-label Class-imbalanced Action Recognition in Hockey Videos via 3D Convolutional Neural Networks

Automatic analysis of the video is one of most complex problems in the fields of computer vision and machine learning. A significant part of this research deals with (human) activity recognition (HAR) since humans, and the activities that they perform, generate most of the video semantics. Video-based HAR has applications in various domains, but one of the most important and challenging is HAR in sports videos. Some of the major issues include high inter- and intra-class variations, large class imbalance, the presence of both group actions and single player actions, and recognizing simultaneous actions, i.e., the multi-label learning problem. Keeping in mind these challenges and the recent success of CNNs in solving various computer vision problems, in this work, we implement a 3D CNN based multi-label deep HAR system for multi-label class-imbalanced action recognition in hockey videos. We test our system for two different scenarios: an ensemble of $k$ binary networks vs. a single $k$-output network, on a publicly available dataset. We also compare our results with the system that was originally designed for the chosen dataset. Experimental results show that the proposed approach performs better than the existing solution.Comment: Accepted to IEEE/ACIS SNPD 2018, 6 pages, 3 figure

Towards event analysis in time-series data: Asynchronous probabilistic models and learning from partial labels

In this thesis, we contribute in two main directions: modeling asynchronous time-series data and learning from partial labelled data. We first propose novel probabilistic frameworks to improve flexibility and expressiveness of current approaches in modeling complex real-world asynchronous event sequence data. Second, we present a scalable approach to end-to-end learn a deep multi-label classifier with partial labels. To evaluate the effectiveness of our proposed frameworks, we focus on visual recognition application, however, our proposed frameworks are generic and can be used in modeling general settings of learning event sequences, and learning multi-label classifiers from partial labels. Visual recognition is a fundamental piece for achieving machine intelligence, and has a wide range of applications such as human activity analysis, autonomous driving, surveillance and security, health-care monitoring, etc. With a wide range of experiments, we show that our proposed approaches help to build more powerful and effective visual recognition frameworks

Weakly Supervised Learning for Activity Recognition from Time Series Data

The thesis focuses on activity recognition from sensor data, which has spurred a great deal of interest due to its impact on health care and security. Previous work on activity recognition from multivariate time series data has mainly applied supervised learning techniques which require a high degree of annotation effort to label the training data with the start and end times of each activity. Multi-instance learning (MIL) and multi-instance multi-label learning (MIML) are weakly supervised learning alternatives to the standard supervised learning framework, in which ambiguity in the labeling can reduce the annotation effort needed to produce labeled training data. We introduce generative graphical models for MIL and MIML based on auto-regressive processes. Our first work, based on a mixture of auto-regressive processes, assumes that instances within a bag are independent. We then relax the i.i.d. assumption for instances and extend the model by considering the sequential structure within a bag. Finally we take a MIML approach to predict the presence of multiple activities within a time interval. For each approach, we evaluate against other state-of-the-art algorithms to demonstrate the effectiveness of the proposed approach on multiple real-world data sets

CALDA: Improving Multi-Source Time Series Domain Adaptation with Contrastive Adversarial Learning

Unsupervised domain adaptation (UDA) provides a strategy for improving machine learning performance in data-rich (target) domains where ground truth labels are inaccessible but can be found in related (source) domains. In cases where meta-domain information such as label distributions is available, weak supervision can further boost performance. We propose a novel framework, CALDA, to tackle these two problems. CALDA synergistically combines the principles of contrastive learning and adversarial learning to robustly support multi-source UDA (MS-UDA) for time series data. Similar to prior methods, CALDA utilizes adversarial learning to align source and target feature representations. Unlike prior approaches, CALDA additionally leverages cross-source label information across domains. CALDA pulls examples with the same label close to each other, while pushing apart examples with different labels, reshaping the space through contrastive learning. Unlike prior contrastive adaptation methods, CALDA requires neither data augmentation nor pseudo labeling, which may be more challenging for time series. We empirically validate our proposed approach. Based on results from human activity recognition, electromyography, and synthetic datasets, we find utilizing cross-source information improves performance over prior time series and contrastive methods. Weak supervision further improves performance, even in the presence of noise, allowing CALDA to offer generalizable strategies for MS-UDA. Code is available at: https://github.com/floft/caldaComment: Under review at IEEE Transactions on Pattern Analysis and Machine Intelligenc

FedOpenHAR: Federated Multi-Task Transfer Learning for Sensor-Based Human Activity Recognition

Motion sensors integrated into wearable and mobile devices provide valuable information about the device users. Machine learning and, recently, deep learning techniques have been used to characterize sensor data. Mostly, a single task, such as recognition of activities, is targeted, and the data is processed centrally at a server or in a cloud environment. However, the same sensor data can be utilized for multiple tasks and distributed machine-learning techniques can be used without the requirement of the transmission of data to a centre. This paper explores Federated Transfer Learning in a Multi-Task manner for both sensor-based human activity recognition and device position identification tasks. The OpenHAR framework is used to train the models, which contains ten smaller datasets. The aim is to obtain model(s) applicable for both tasks in different datasets, which may include only some label types. Multiple experiments are carried in the Flower federated learning environment using the DeepConvLSTM architecture. Results are presented for federated and centralized versions under different parameters and restrictions. By utilizing transfer learning and training a task-specific and personalized federated model, we obtained a similar accuracy with training each client individually and higher accuracy than a fully centralized approach.Comment: Subimtted to Asian Conference in Machine Learning (ACML) 2023, Pattern Recognition in Health Analysis Workshop, 7 pages, 3 figure

Machine learning techniques for sensor-based household activity recognition and forecasting

Thanks to the recent development of cheap and unobtrusive smart-home sensors, ambient assisted living tools promise to offer innovative solutions to support the users in carrying out their everyday activities in a smoother and more sustainable way. To be effective, these solutions need to constantly monitor and forecast the activities of daily living carried out by the inhabitants. The Machine Learning field has seen significant advancements in the development of new techniques, especially regarding deep learning algorithms. Such techniques can be successfully applied to household activity signal data to benefit the user in several applications. This thesis therefore aims to produce a contribution that artificial intelligence can make in the field of activity recognition and energy consumption. The effective recognition of common actions or the use of high-consumption appliances would lead to user profiling, thus enabling the optimisation of energy consumption in favour of the user himself or the energy community in general. Avoiding wasting electricity and optimising its consumption is one of the main objectives of the community. This work is therefore intended as a forerunner for future studies that will allow, through the results in this thesis, the creation of increasingly intelligent systems capable of making the best use of the user's resources for everyday life actions. Namely, this thesis focuses on signals from sensors installed in a house: data from position sensors, door sensors, smartphones or smart meters, and investigates the use of advanced machine learning algorithms to recognize and forecast inhabitant activities, including the use of appliances and the power consumption. The thesis is structured into four main chapters, each of which represents a contribution regarding Machine Learning or Deep Learning techniques for addressing challenges related to the aforementioned data from different sources. The first contribution highlights the importance of exploiting dimensionality reduction techniques that can simplify a Machine Learning model and increase its efficiency by identifying and retaining only the most informative and predictive features for activity recognition. In more detail, it is presented an extensive experimental study involving several feature selection algorithms and multiple Human Activity Recognition benchmarks containing mobile sensor data. In the second contribution, we propose a machine learning approach to forecast future energy consumption considering not only past consumption data, but also context data such as inhabitants’ actions and activities, use of household appliances, interaction with furniture and doors, and environmental data. We performed an experimental evaluation with real-world data acquired in an instrumented environment from a large user group. Finally, the last two contributions address the Non-Intrusive-Load-Monitoring problem. In one case, the aim is to identify the operating state (on/off) and the precise energy consumption of individual electrical loads, considering only the aggregate consumption of these loads as input. We use a Deep Learning method to disaggregate the low-frequency energy signal generated directly by the new generation smart meters being deployed in Italy, without the need for additional specific hardware. In the other case, driven by the need to build intelligent non-intrusive algorithms for disaggregating electrical signals, the work aims to recognize which appliance is activated by analyzing energy measurements and classifying appliances through Machine Learning techniques. Namely, we present a new way of approaching the problem by unifying Single Label (single active appliance recognition) and Multi Label (multiple active appliance recognition) learning paradigms. This combined approach, supplemented with an event detector, which suggests the instants of activation, would allow the development of an end-to-end NILM approach