Human Activity Recognition and Prediction using RGBD Data

Being able to predict and recognize human activities is an essential element for us to effectively communicate with other humans during our day to day activities. A system that is able to do this has a number of appealing applications, from assistive robotics to health care and preventative medicine. Previous work in supervised video-based human activity prediction and detection fails to capture the richness of spatiotemporal data that these activities generate. Convolutional Long short-term memory (Convolutional LSTM) networks are a useful tool in analyzing this type of data, showing good results in many other areas. This thesis’ focus is on utilizing RGB-D Data to improve human activity prediction and recognition. A modified Convolutional LSTM network is introduced to do so. Experiments are performed on the network and are compared to other models in-use as well as the current state-of-the-art system. We show that our proposed model for human activity prediction and recognition outperforms the current state-of-the-art models in the CAD-120 dataset without giving bounding frames or ground-truths about objects

Human activity recognition and prediction in RGB-D videos

Reconhecimento de atividade humana é uma área de investigação multidisciplinar que tem atraído o interesse de investigadores especializados em aprendizagem automática, visão por computador e medicina. Esta área tem diversas aplicações: sistemas de vigilância, interação homem-máquina, análise de desportos, robôs colaborativos, saúde e automóveis autónomos. Capturar atividade humana apresenta dificuldades técnicas como oclusão, iluminação insuficiente, seguimento erróneo e questões éticas. O movimento humano pode ser ambíguo e com múltiplas intenções. A forma como interagimos com outros seres humanos e objetos cria uma combinação quase infinita de variações de como fazemos as coisas. O objetivo desta dissertação é desenvolver um sistema capaz de reconhecer e prever a atividade humana usando técnicas de aprendizagem automática para extrair significado de características calculadas a partir de articulações do corpo humano capturado pela câmara Kinect. Propomos uma arquitetura hierárquica e modular que realiza segmentação temporal de sequências de ações, anotação semi-supervisionada de sub-atividades utilizando técnicas de clustering, reconhecimento de sub-atividade frame-a-frame em tempo real usando classificadores binários de random decision forests logo a partir dos primeiros instantes da ação e previsão de atividade em tempo real baseada em conditional random fields para modelar a estrutura das sequências de ações para obter as futuras possibilidades. Gravámos um novo conjunto de dados contendo sequências de ações agressivas com um total de 72 sequências, 360 amostras de 8 ações distintas realizadas por 12 sujeitos. Efetuamos testes extensivos com dois conjuntos de dados, comparando o desempenho de reconhecimento de vários classificadores supervisionados treinados com dados anotados manualmente ou com dados anotados de forma semi-supervisionada. Aprendemos como a qualidade dos conjuntos de treino afeta os resultado que dependem também da complexidade das ações que estão a ser reconhecidas. Conseguímos obter melhores resultados que algumas das abordagens existentes na literatura em reconhecimento de atividade, efetuamos o reconhecimento de forma antecipada e obtivemos resultados encorajadores na previsão de atividades.Human Activity Recognition is an interdisciplinary research area that has been attracting interest from several research communities specialized in machine learning, computer vision, and medical research. The potential applications range from surveillance systems, human computer interfaces, sports analysis, digital assistants, collaborative robots, health-care and self-driving cars. Capturing human activity presents technical difficulties like occlusion, insufficient lighting, unreliable tracking and ethical concerns. Human motion can be ambiguous and have multiple intents. The complexity of our lives and how we interact with other humans and objects prompt to a nearly infinite combination of variations in how we do things. The focus of this dissertation is to develop a system capable of recognizing and predicting human activity using machine learning techniques to extract meaning from features computed from relevant joints of the human body captured by the skeleton tracker of the Kinect sensor. We propose a modular framework that performs off-line temporal segmentation of sequences of actions, off-line semi unsupervised labeling of sub-activities via clustering techniques, real-time frame by-frame sub-activity recognition using random decision forest binary classifiers right from the very first frames of the action and real-time activity prediction with conditional random fields to model the sequential structure of sequences of actions to reason about future possibilities. We recorded a new dataset containing long sequences of aggressive actions with a total of 72 sequences, 360 samples of 8 distinct actions performed by 12 subjects. We experimented extensively with two different datasets, compared the recognition performance of several supervised classifiers trained with manually labeled data versus semi-unsupervised labeled data. We learned how the quality of the training data affects the results which also depends on the complexity of the actions being recognized. We outperformed state-ofthe-art activity recognition approaches, performed early action recognition and obtained encouraging results in activity prediction

SVMDnet: A Novel Framework for Elderly Activity Recognition based on Transfer Learning

Elderly Activity Recognition has become very crucial now-a-days because majority of elderly people are living alone and are vulnerable. Despite the fact that several researchers employ ML (machine learning) and DL (deep learning) techniques to recognize elderly actions, relatively lesser research specifically aimed on transfer learning based elderly activity recognition. Even transfer learning is not sufficient to handle the complexity levels in the HAR related problems because it is a more general approach. A novel transfer leaning based framework SVMDnet is proposed in which pre-trained deep neural network extracts essential action features and to classify actions, Support Vector Machine (SVM) is used as a classifier. The proposed model is evaluated on Stanford-40 Dataset and self-made dataset. The older volunteers over the age of 60 were recruited for the main dataset, which was compiled from their responses in a uniform environment with 10 kinds of activities. Results from SVMDnet on the two datasets shows that our model behaves well with human recognition and human-object interactions as well

An Efficient Activity Detection System based on Skeleton Joints Identification

The increasing criminal activities in the current world has drawn lot of interest activity recognition techniques which helps to perform the sophistical analytical operations on human activity and also helps to interface the human and computer interactions. From the existing review analysis it is found that most of the existing systems are not emphasize on computational performance but are more application specific by identifying specific problems. Hence, it is found that all the features are not required for accurate and cost effective human activity detection. Thus, the human skelton action can be considered and presented a simple and accurate process to identify the significant joints only. From the outcomes it is found that the proposed system is cost effective and computational efficient activity recognition technique for human actions

Monitoring Quality of Life Indicators at Home from Sparse and Low-Cost Sensor Data.

Supporting older people, many of whom live with chronic conditions or cognitive and physical impairments, to live independently at home is of increasing importance due to ageing demographics. To aid independent living at home, much effort is being directed at reliably detecting activities from sensor data to monitor people’s quality of life or to enhance self-management of their own health. Current efforts typically leverage smart homes which have large numbers of sensors installed to overcome challenges in the accurate detection of activities. In this work, we report on the results of machine learning models based on data collected with a small number of low-cost, off-the-shelf passive sensors that were retrofitted in real homes, some with more than a single occupant. Models were developed from the sensor data collected to recognize activities of daily living, such as eating and dressing as well as meaningful activities, such as reading a book and socializing. We evaluated five algorithms and found that a Recurrent Neural Network was most accurate in recognizing activities. However, many activities remain difficult to detect, in particular meaningful activities, which are characterized by high levels of individual personalization. Our work contributes to applying smart healthcare technology in real-world home settings