Gestural Human-Machine-Interface (HMI) for an autonomous wheelchair for kids

El Trabajo de Fin de Master (TFM) se desarrolla a partir de una plataforma de ayuda a la movilidad destinada a niños. La arquitectura general de la plataforma se describe en anteriores trabajos. La plataforma consta de distintos nodos para suplir todas las funciones, alimentación, electrónica de potencia, control y navegación, interacción con el entorno e interfaz humano-maquina. Este TFM se centra en el nodo PC, el cual se basa en un ordenador con sistema operativo Linux y caracterizado por el uso de Robot Operating System (ROS). Sobre esta base se asienta la interfaz humano máquina gestual que se desarrolla en este trabajo. Este integra en el sistema existente una cámara RGBD Intel Realsense D435, ya que esta aplicación necesita tanto imagen RGB como imagen en profundidad. La información que proporciona la cámara se utiliza por medio de los paquetes que ofrece el fabricante de la cámara en ROS. Posteriormente se realiza la detección de personas. Para ello se utiliza una red neuronal entrenada para la detección de objetos basada en Tensorflow. A partir de los resultados de detección de la red, se obtiene la posición de las personas detectadas, transformando la posición en el plano de la persona su localización en el entorno virtual de la aplicación. Además se aplican técnicas de filtrado y tracking para mejorar esta localización. Por último, se implementa un sistema de reconocimiento de gestos, mediante el cual se pueda seleccionar fácilmente que usuario que desea interactuar con la plataforma y ejecutar una aplicación determinada. En el caso de este trabajo, la aplicacion elegida se basa en una estrategia denominada Follow Me, en la que la plataforma interactúe con el usuario y navegue por el entorno siguiéndole. La aplicación se incluye dentro del entorno de ROS, compatibilizando de esta forma su actuación con el resto de funciones de la plataforma.The Master's thesis is based on a mobility support platform for children. The general architecture of the platform is described in previous works. The platform consists of different nodes to provide all functions, power supply, power electronics, control and navigation, interaction with the environment and human-machine interface. This Master's thesis focuses on the PC node, which is based on a computer with a Linux operating system and characterised by the use of Robot Operating System (ROS). This is the basis for the gestural human-machine interface developed in this work. An Intel Realsense D435 RGBD camera is integrated into the existing system, as both RGB image and depth image are required for this application. The information provided by the camera is used by means of the packages offered by the camera manufacturer in ROS. Subsequently, the detection of persons is carried out. For this purpose, a neural network trained for object detection based on Tensorflow is used. From the detection results of the network, the position of the detected persons is obtained, transforming the position in the plane of the person to the location in the virtual environment of the application. In addition, f ltering and tracking techniques are applied to improve this localisation. Finally, a gesture recognition system is implemented, by means of which the user can easily select which user wants to interact with the platform and execute a given application. In the case of this work, the chosen application is based on a navigation strategy called Follow Me, in which the platform follows the user and navigates the environment in this way. The application is merged within the ROS environment, thus making it compatible with the rest of the platform's functions.Máster Universitario en Ingeniería Industrial (M141

Quaternion-based gesture recognition using wireless wearable motion capture sensors

This work presents the development and implementation of a unified multi-sensor human motion capture and gesture recognition system that can distinguish between and classify six different gestures. Data was collected from eleven participants using a subset of five wireless motion sensors (inertial measurement units) attached to their arms and upper body from a complete motion capture system. We compare Support Vector Machines and Artificial Neural Networks on the same dataset under two different scenarios and evaluate the results. Our study indicates that near perfect classification accuracies are achievable for small gestures and that the speed of classification is sufficient to allow interactivity. However, such accuracies are more difficult to obtain when a participant does not participate in training, indicating that more work needs to be done in this area to create a system that can be used by the general population

SEMANTIC ANALYSIS AND UNDERSTANDING OF HUMAN BEHAVIOUR IN VIDEO STREAMING

This thesis investigates the semantic analysis of the human behaviour captured by video streaming, both from the theoretical and technological points of view. The video analysis based on the semantic content is in fact still an open issue for the computer vision research community, especially when real-time analysis of complex scenes is concerned. Automated video analysis can be described and performed at different abstraction levels, from the pixel analysis up to the human behaviour understanding. Similarly, the organisation of computer vision systems is often hierarchical with low-level image processing techniques feeding into tracking algorithms and, then, into higher level scene analysis and/or behaviour analysis modules. Each level of this hierarchy has its open issues, among which the main ones are: - motion and object detection: dynamic background modelling, ghosts, suddenly changes in illumination conditions; - object tracking: modelling and estimating the dynamics of moving objects, presence of occlusions; - human behaviour identification: human behaviour patterns are characterized by ambiguity, inconsistency and time-variance. Researchers proposed various approaches which partially address some aspects of the above issues from the perspective of the semantic analysis and understanding of the video streaming. Many progresses were achieved, but usually not in a comprehensive way and often without reference to the actual operating situations. A popular class of approaches has been devised to enhance the quality of the semantic analysis by exploiting some background knowledge about scene and/or the human behaviour, thus narrowing the huge variety of possible behavioural patterns by focusing on a specific narrow domain. In general, the main drawback of the existing approaches to semantic analysis of the human behaviour, even in narrow domains, is inefficiency due to the high computational complexity related to the complex models representing the dynamics of the moving objects and the patterns of the human behaviours. In this perspective this thesis explores an innovative, original approach to human behaviour analysis and understanding by using the syntactical symbolic analysis of images and video streaming described by means of strings of symbols. A symbol is associated to each area of the analysed scene. When a moving object enters an area, the corresponding symbol is appended to the string describing the motion. This approach allows for characterizing the motion of a moving object with a word composed by symbols. By studying and classifying these words we can categorize and understand the various behaviours. The main advantage of this approach consists in the simplicity of the scene and motion descriptions so that the behaviour analysis will have limited computational complexity due to the intrinsic nature both of the representations and the related operations used to manipulate them. Besides, the structure of the representations is well suited for possible parallel processing, thus allowing for speeding up the analysis when appropriate hardware architectures are used. The theoretical background, the original theoretical results underlying this approach, the human behaviour analysis methodology, the possible implementations, and the related performance are presented and discussed in the thesis. To show the effectiveness of the proposed approach, a demonstrative system has been implemented and applied to a real indoor environment with valuable results. Furthermore, this thesis proposes an innovative method to improve the overall performance of the object tracking algorithm. This method is based on using two cameras to record the same scene from different point of view without introducing any constraint on cameras\u2019 position. The image fusion task is performed by solving the correspondence problem only for few relevant points. This approach reduces the problem of partial occlusions in crowded scenes. Since this method works at a level lower than that of semantic analysis, it can be applied also in other systems for human behaviour analysis and it can be seen as an optional method to improve the semantic analysis (because it reduces the problem of partial occlusions)