Portuguese sign language recognition via computer vision and depth sensor

Sign languages are used worldwide by a multitude of individuals. They are mostly used by the deaf communities and their teachers, or people associated with them by ties of friendship or family. Speakers are a minority of citizens, often segregated, and over the years not much attention has been given to this form of communication, even by the scientific community. In fact, in Computer Science there is some, but limited, research and development in this area. In the particular case of sign Portuguese Sign Language-PSL that fact is more evident and, to our knowledge there isn’t yet an efficient system to perform the automatic recognition of PSL signs. With the advent and wide spreading of devices such as depth sensors, there are new possibilities to address this problem. In this thesis, we have specified, developed, tested and preliminary evaluated, solutions that we think will bring valuable contributions to the problem of Automatic Gesture Recognition, applied to Sign Languages, such as the case of Portuguese Sign Language. In the context of this work, Computer Vision techniques were adapted to the case of Depth Sensors. A proper gesture taxonomy for this problem was proposed, and techniques for feature extraction, representation, storing and classification were presented. Two novel algorithms to solve the problem of real-time recognition of isolated static poses were specified, developed, tested and evaluated. Two other algorithms for isolated dynamic movements for gesture recognition (one of them novel), have been also specified, developed, tested and evaluated. Analyzed results compare well with the literature.As Línguas Gestuais são utilizadas em todo o Mundo por uma imensidão de indivíduos. Trata-se na sua grande maioria de surdos e/ou mudos, ou pessoas a eles associados por laços familiares de amizade ou professores de Língua Gestual. Tratando-se de uma minoria, muitas vezes segregada, não tem vindo a ser dada ao longo dos anos pela comunidade científica, a devida atenção a esta forma de comunicação. Na área das Ciências da Computação existem alguns, mas poucos trabalhos de investigação e desenvolvimento. No caso particular da Língua Gestual Portuguesa - LGP esse facto é ainda mais evidente não sendo nosso conhecimento a existência de um sistema eficaz e efetivo para fazer o reconhecimento automático de gestos da LGP. Com o aparecimento ou massificação de dispositivos, tais como sensores de profundidade, surgem novas possibilidades para abordar este problema. Nesta tese, foram especificadas, desenvolvidas, testadas e efectuada a avaliação preliminar de soluções que acreditamos que trarão valiosas contribuições para o problema do Reconhecimento Automático de Gestos, aplicado às Línguas Gestuais, como é o caso da Língua Gestual Portuguesa. Foram adaptadas técnicas de Visão por Computador ao caso dos Sensores de Profundidade. Foi proposta uma taxonomia adequada ao problema, e apresentadas técnicas para a extração, representação e armazenamento de características. Foram especificados, desenvolvidos, testados e avaliados dois algoritmos para resolver o problema do reconhecimento em tempo real de poses estáticas isoladas. Foram também especificados, desenvolvidos, testados e avaliados outros dois algoritmos para o Reconhecimento de Movimentos Dinâmicos Isolados de Gestos(um deles novo).Os resultados analisados são comparáveis à literatura.Las lenguas de Signos se utilizan en todo el Mundo por una multitud de personas. En su mayoría son personas sordas y/o mudas, o personas asociadas con ellos por vínculos de amistad o familiares y profesores de Lengua de Signos. Es una minoría de personas, a menudo segregadas, y no se ha dado en los últimos años por la comunidad científica, la atención debida a esta forma de comunicación. En el área de Ciencias de la Computación hay alguna pero poca investigación y desarrollo. En el caso particular de la Lengua de Signos Portuguesa - LSP, no es de nuestro conocimiento la existencia de un sistema eficiente y eficaz para el reconocimiento automático. Con la llegada en masa de dispositivos tales como Sensores de Profundidad, hay nuevas posibilidades para abordar el problema del Reconocimiento de Gestos. En esta tesis se han especificado, desarrollado, probado y hecha una evaluación preliminar de soluciones, aplicada a las Lenguas de Signos como el caso de la Lengua de Signos Portuguesa - LSP. Se han adaptado las técnicas de Visión por Ordenador para el caso de los Sensores de Profundidad. Se propone una taxonomía apropiada para el problema y se presentan técnicas para la extracción, representación y el almacenamiento de características. Se desarrollaran, probaran, compararan y analizan los resultados de dos nuevos algoritmos para resolver el problema del Reconocimiento Aislado y Estático de Posturas. Otros dos algoritmos (uno de ellos nuevo) fueran también desarrollados, probados, comparados y analizados los resultados, para el Reconocimiento de Movimientos Dinámicos Aislados de los Gestos

Design of Intangible Interface for Human Computer Interaction- Hand Gestures Controlled Mouse

The current generation is the touch generation. In this generation, everything works with the touch of a finger, be it our cell phones, computers or any day to day appliance. But the so called 'touch generation' has reached its ultimate state and is soon likely to decline. The generation of 'intangible interfaces' is just at its dawn and is soon going to take over the touch generation. The problem statement for this project is to improve the human computer interaction with the use of intangible interface. The aim is to control all operations of a mouse without the actual use of mouse.This would allow the user to navigate his laptop while sitting far away from it. This is achieved by recognition of hand gestures. A computer vision application is made which captures images of the hand gestures from the laptop's web camera, processes it, and performs the required output.Several algorithms of computer vision including image segmentation, feature extraction have been used. The basic functions of a mouse i.e. left click, right click and double click using hand gestures are performed in this project. The main aim is to improve the interaction of human with the computer i.e. to make it more natural but keeping the cost factor in mind. The purpose is to develop an application that must be easy as well as economic to use

Hand Gesture Based Surveillance Robot

In this work, a hardware and software based integrated system is developed for hand gesture based surveillance robot. The proposed system is a non-invasive technique and software part of the system uses gesture based image processing technique. The hardware part is developed based on AVR microcontroller platform. The captured image of hand is segmented and its contour is determined. The convexity defects are computed to detect the number of fingers used by the subject. The number of fingers directs the path to robot that is to be followed. The camera placed on the robot capture the images of its surrounding, wherever it travels and send it back to the PC for monitoring. In this way, it can be used as a surveillance system. Experimental results show that the overall accuracy obtained above 90% for gesture recognition by which robot will be directed to follow the path. The system can be directly applied to defence grounds for detection of enemy, for spying purpose where the human reach is avoided or not recommended. This unit can be used for overcoming physical handicaps by helping in development of gesture-based wheel chairs, for control of home devices and appliances for persons with physical handicaps and/or elderly users with impaired mobility

A wearable, wireless sensor system for sports medicine

Thesis (S.M.)--Massachusetts Institute of Technology, School of Architecture and Planning, Program in Media Arts and Sciences, 2008.Includes bibliographical references (p. 135-139).This thesis describes a compact, wireless, wearable system that measures, for purposes of biomechanical analysis, signals indicative of forces, torques and other descriptive and evaluative features that the human body undergoes during bursts of extreme physical activity (such as during athletic performance). Standard approaches leverage high speed camera systems, which need significant infrastructure and provide limited update rates and dynamic accuracy, to make these measurements. This project uses 6 degree-of-freedom inertial measurement units worn on various segments of an athlete's body to directly make these dynamic measurements. A combination of low and high G sensors enables sensitivity for slow and fast motion, and the addition of a compass helps in tracking joint angles. Data from the battery-powered nodes is acquired using a custom wireless protocol over an RF link. This data, along with rigorous calibration data, is processed on a PC, with an end product being precise angular velocities and accelerations that can be employed during biomechanical analysis to gain a better understanding of what occurs during activity. The focus of experimentation was baseball pitching and batting at the professional level. Several pitchers and batters were instrumented with the system and data was gathered during several pitches or swings. The data was analyzed, and the results of this analysis are presented in this thesis. The dynamic results are more precise than from other camera based systems and also offer the measurement of metrics that are not available from any other system, providing the opportunity for furthering sports medicine research. System performance and results are evaluated, and ideas for future work and system improvements are presented.by Michael Tomasz Lapinski.S.M

Multimodal access to social media services

Tese de mestrado integrado. Engenharia Informática e Computação. Faculdade de Engenharia. Universidade do Porto, Microsoft Language Development Center. 201

Sociometric badges : wearable technology for measuring human behavior

Thesis (S.M.)--Massachusetts Institute of Technology, School of Architecture and Planning, Program in Media Arts and Sciences, 2007.Includes bibliographical references (p. 137-144).We present the design, implementation and deployment of a wearable computing research platform for measuring and analyzing human behavior in a variety of settings and applications. We propose the use of wearable sociometric badges capable of automatically measuring the amount of face-to-face interaction, conversational time, physical proximity to other people, and physical activity levels using social signals derived from vocal features, body motion, and relative location to capture individual and collective patterns of behavior. Our goal is to be able to understand how patterns of behavior shape individuals and organizations. We attempt to use on-body sensors in large groups of people for extended periods of time in naturalistic settings for the purpose of identifying, measuring, and quantifying social interactions, information flow, and organizational dynamics. We deployed this research platform in a group of 22 employees working in a real organization over a period of one month. Using these automatic measurements we were able to predict employees' self-assessment of productivity, job satisfaction, and their own perception of group interaction quality. An initial exploratory data analysis indicates that it is possible to automatically capture patterns of behavior using this wearable platform.by Daniel Olguín Olguín.S.M

Children’s Fitness and Quality of Movement

Introduction: Movement is essential to life and plays a key role in development throughout childhood. Movement can be assessed by its quantity and quality. Movement is important to measure as it can aid early intervention. Current research suggests that global levels of fitness are declining, with a lack of research surrounding children’s natural fitness levels as they get older. Quantity of movement is commonly studied, however quality is becoming increasingly popular. A clear understanding of the methods of technology used to measure quality of movement is important as understanding this area will aid in designing appropriate interventions.Methods: This thesis comprises of two experimental studies. Study one is a repeated measures design using previously collected Swanlinx data to investigate how components of children’s fitness change over a one-year period. Study two is a scoping review investigating the measurement of quality of movement with technology in the form of MEM’s devices, while aiming to gain clarity on the definition of quality.Results: Study one revealed that children’s fitness levels increase across a one-year period, in all components of fitness, except sit and reach. Boys performed significantly better in all fitness components, apart from sit and reach. Study two demonstrated the broad field that is included under the term of quality, showing clarity is needed in this area. A large number of devices, movements and populations are being observed, with multiple definitions of quality which is dependent on the metrics collected.Conclusion: Study one concludes that children’s fitness levels increase over one-year, with boys performing better than girls. This can be used to understand children’s natural fitness levels and aid future interventions in participation. Study two concludes that there are multiple ways to assess quality of movement however a clear definition of the quality should be stated, aiding comparison of quality

Étude et conception de systèmes de validation de tâches d'assemblage : participation à la conception d'un robot collaboratif destiné à une chaîne de montage automobile

La robotique collaborative connaît ces dernières années une forte croissance, grâce à la pertinence de la collaboration entre les humains et les robots dans un même espace de travail. Des applications industrielles se développent, aussi bien dans les grandes multinationales que dans les PME. Dans le contexte d’une chaîne d’assemblage automobile, de nombreuses problématiques se posent pour permettre la collaboration efficace entre robot et opérateur. Un robot collaboratif conçu dans ce cadre est destiné à effectuer des tâches d’assemblages dans des zones à l’accessibilité complexe. Étant à distance, l’opérateur n’aura pas nécessairement de visuel direct sur l’accomplissement de la tâche, et pour éviter des pertes de temps, il est souhaitable que le robot soit capable de vérifier et de valider automatiquement son bon accomplissement. En découle la problématique à l’étude dans ce mémoire, qui est la vérification automatique du bon accomplissement des tâches d’assemblage effectuées par le robot. Afin d’y répondre, une pré-étude est effectuée, visant à déterminer les possibilités en termes de capteurs et d’algorithmes de détection, et permettant de répondre aux besoins et contraintes de General Motors, commanditaire de cette étude. Chaque chapitre du mémoire présente donc un système de détection, composé d’un capteur et d’un ou plusieurs algorithmes de décision. Les différences en termes de traitement du signal et d’algorithme sont présentées et comparées, en s’appuyant sur des expérimentations. Parmi les types d’assemblages sélectionnés par General Motors, ceux utilisés pour la conception et l’expérimentation des solutions sont les assemblages clipsés (« snap fit ») et les assemblages de connecteurs électriques. Le premier chapitre présente une solution fondée sur l’utilisation de photodétecteurs, secondée par un algorithme vérifiant en temps réel les valeurs renvoyées par le photodétecteur. Lorsque cette valeur descend sous une valeur seuil préétablie, l’algorithme déduit que l’assemblage a été correctement effectué. Cette méthode est simple et robuste, ayant obtenu un taux de détection correcte de la situation d’assemblage de 100% lors des phases de tests. Cependant cette robustesse est contrebalancée par le caractère relativement spécifique de la solution et son peu de flexibilité pour l’adaptation à d’autres assemblages. Le second chapitre porte sur un système employant un accéléromètre destiné à enregistrer les différents mouvements survenant lors de l’assemblage. Suit une méthode de traitement du signal obtenu, permettant de caractériser et de différencier les signaux émanant d’assemblages réussis de ceux provenant d’assemblages ratés grâce à des paramètres statistiques. Ces derniers sont finalement récupérés par deux algorithmes de reconnaissance différents, qui sont ensuite comparés. Le premier algorithme consiste à vérifier la correspondance des paramètres statistiques d’un signal avec des valeurs seuil fixées à l’avance. L’autre méthode, issue de l’apprentissage automatique, utilise les Séparateurs à Vaste Marge pour distinguer deux classes : les assemblages réussis et les assemblages ratés. Après expérimentation, il ressort que ces deux méthodes ont des performances assez similaires , de l’ordre de 90%. Ainsi ce sera le contexte et la complexité du signal à étudier qui présideront au choix d’une méthode. Enfin, le dernier chapitre se penche sur l’emploi d’un microphone et de techniques de reconnaissance sonore pour permettre la bonne identification d’un assemblage de connecteurs électriques. Deux méthodes de traitement, la transformée en ondelettes et les Mel Frequency Cepstrum Coefficients (MFCC), sont comparées. La reconnaissance est assurée par un algorithme basé sur les modèles de mélange gaussiens. A l’issue d’essais où les deux prétraitements ont été mis à l’épreuve, en tenant compte de la résistance face au bruit ambiant, il est possible de constater que la transformée en ondelettes présente des performances supérieures aux MFCC, dans le cadre précis de l’étude. La reconnaissance sonore ne nécessitant qu’un son distinct à identifier pour pouvoir fonctionner, cette solution présente un fort potentiel d’adaptation à d’autres types d’assemblages, mais aussi d’autres problématiques. Ces trois systèmes permettent de balayer un certain éventail de possibilités, et les solutions présentées sont amenées à être croisées et bonifiées afin d’aboutir à des systèmes robustes et fiables, aux standards de l’industrie