Development and comparison of dataglove and sEMG signal-based algorithms for the improvement of a hand gestures recognition system.

openHand gesture recognition is a topic widely discussed in literature, where several techniques are analyzed both in terms of input signal types and algorithms. The main bottleneck of the field is the generalization ability of the classifier, which becomes harder as the number of gestures to classify increases. This project has two purposes: first, it aims to develop a reliable and high-generalizability classifier, evaluating the difference in performances when using Dataglove and sEMG signals; finally, it makes considerations regarding the difficulties and advantages of developing a sEMG signal-based hand gesture recognition system, with the objective of providing indications for its improvement. To design the algorithms, data coming from a public available dataset were considered; the information were referred to 40 healthy subjects (not amputees), and for each of the 17 gestures considered, 6 repetitions were done. Finally, both conventional machine learning and deep learning approaches were used, comparing their efficiency. The results showed better performances for dataglove-based classifier, highlighting the signal informative power, while the sEMG could not provide high generalization. Interestingly, the latter signal gives better performances if it’s analyzed with classical machine learning approaches which allowed, performing feature selection, to underline both the most significative features and the most informative channels. This study confirmed the intrinsic difficulties in using the sEMG signal, but it could provide hints for the improvement of sEMG signal-based hand gesture recognition systems, by reduction of computational cost and electrodes position optimization

Biometric walk recognizer. Research and results on wearable sensor-based gait recognition

Gait is a biometric trait that can allow user authentication, though being classified as a "soft" one due to a certain lack in permanence, and to sensibility to specific conditions. The earliest research relies on computer vision-based approaches, especially applied in video surveillance. More recently, the spread of wearable sensors, especially those embedded in mobile devices, which are able to capture the dynamics of the walking pattern through simpler 1D signals, has spurred a different research line. This capture modality can avoid some problems related to computer vision-based techniques, but suffers from specific limitations. Related research is still in a less advanced phase with respect to other biometric traits. However, the promising results achieved so far, the increasing accuracy of sensors, the ubiquitous presence of mobile devices, and the low cost of related techniques, make this biometrics attractive and suggest to continue the investigations in this field. The first Chapters of this thesis deal with an introduction to biometrics, and more specifically to gait trait. A comprehensive review of technologies, approaches and strategies exploited by gait recognition proposals in the state-of-the-art is also provided. After such introduction, the contributions of this work are presented in details. Summarizing, it improves preceding result achieved during my Master Degree in Computer Science course of Biometrics and extended in my following Master Degree Thesis. The research deals with different strategies, including preprocessing and recognition techniques, applied to the gait biometrics, in order to allow both an automatic recognition and an improvement of the system accuracy

Real-time Immersive human-computer interaction based on tracking and recognition of dynamic hand gestures

With fast developing and ever growing use of computer based technologies, human-computer interaction (HCI) plays an increasingly pivotal role. In virtual reality (VR), HCI technologies provide not only a better understanding of three-dimensional shapes and spaces, but also sensory immersion and physical interaction. With the hand based HCI being a key HCI modality for object manipulation and gesture based communication, challenges are presented to provide users a natural, intuitive, effortless, precise, and real-time method for HCI based on dynamic hand gestures, due to the complexity of hand postures formed by multiple joints with high degrees-of-freedom, the speed of hand movements with highly variable trajectories and rapid direction changes, and the precision required for interaction between hands and objects in the virtual world. Presented in this thesis is the design and development of a novel real-time HCI system based on a unique combination of a pair of data gloves based on fibre-optic curvature sensors to acquire finger joint angles, a hybrid tracking system based on inertia and ultrasound to capture hand position and orientation, and a stereoscopic display system to provide an immersive visual feedback. The potential and effectiveness of the proposed system is demonstrated through a number of applications, namely, hand gesture based virtual object manipulation and visualisation, hand gesture based direct sign writing, and hand gesture based finger spelling. For virtual object manipulation and visualisation, the system is shown to allow a user to select, translate, rotate, scale, release and visualise virtual objects (presented using graphics and volume data) in three-dimensional space using natural hand gestures in real-time. For direct sign writing, the system is shown to be able to display immediately the corresponding SignWriting symbols signed by a user using three different signing sequences and a range of complex hand gestures, which consist of various combinations of hand postures (with each finger open, half-bent, closed, adduction and abduction), eight hand orientations in horizontal/vertical plans, three palm facing directions, and various hand movements (which can have eight directions in horizontal/vertical plans, and can be repetitive, straight/curve, clockwise/anti-clockwise). The development includes a special visual interface to give not only a stereoscopic view of hand gestures and movements, but also a structured visual feedback for each stage of the signing sequence. An excellent basis is therefore formed to develop a full HCI based on all human gestures by integrating the proposed system with facial expression and body posture recognition methods. Furthermore, for finger spelling, the system is shown to be able to recognise five vowels signed by two hands using the British Sign Language in real-time

Character Recognition

Character recognition is one of the pattern recognition technologies that are most widely used in practical applications. This book presents recent advances that are relevant to character recognition, from technical topics such as image processing, feature extraction or classification, to new applications including human-computer interfaces. The goal of this book is to provide a reference source for academic research and for professionals working in the character recognition field

State of the art of audio- and video based solutions for AAL

Working Group 3. Audio- and Video-based AAL ApplicationsIt is a matter of fact that Europe is facing more and more crucial challenges regarding health and social care due to the demographic change and the current economic context. The recent COVID-19 pandemic has stressed this situation even further, thus highlighting the need for taking action. Active and Assisted Living (AAL) technologies come as a viable approach to help facing these challenges, thanks to the high potential they have in enabling remote care and support. Broadly speaking, AAL can be referred to as the use of innovative and advanced Information and Communication Technologies to create supportive, inclusive and empowering applications and environments that enable older, impaired or frail people to live independently and stay active longer in society. AAL capitalizes on the growing pervasiveness and effectiveness of sensing and computing facilities to supply the persons in need with smart assistance, by responding to their necessities of autonomy, independence, comfort, security and safety. The application scenarios addressed by AAL are complex, due to the inherent heterogeneity of the end-user population, their living arrangements, and their physical conditions or impairment. Despite aiming at diverse goals, AAL systems should share some common characteristics. They are designed to provide support in daily life in an invisible, unobtrusive and user-friendly manner. Moreover, they are conceived to be intelligent, to be able to learn and adapt to the requirements and requests of the assisted people, and to synchronise with their specific needs. Nevertheless, to ensure the uptake of AAL in society, potential users must be willing to use AAL applications and to integrate them in their daily environments and lives. In this respect, video- and audio-based AAL applications have several advantages, in terms of unobtrusiveness and information richness. Indeed, cameras and microphones are far less obtrusive with respect to the hindrance other wearable sensors may cause to one’s activities. In addition, a single camera placed in a room can record most of the activities performed in the room, thus replacing many other non-visual sensors. Currently, video-based applications are effective in recognising and monitoring the activities, the movements, and the overall conditions of the assisted individuals as well as to assess their vital parameters (e.g., heart rate, respiratory rate). Similarly, audio sensors have the potential to become one of the most important modalities for interaction with AAL systems, as they can have a large range of sensing, do not require physical presence at a particular location and are physically intangible. Moreover, relevant information about individuals’ activities and health status can derive from processing audio signals (e.g., speech recordings). Nevertheless, as the other side of the coin, cameras and microphones are often perceived as the most intrusive technologies from the viewpoint of the privacy of the monitored individuals. This is due to the richness of the information these technologies convey and the intimate setting where they may be deployed. Solutions able to ensure privacy preservation by context and by design, as well as to ensure high legal and ethical standards are in high demand. After the review of the current state of play and the discussion in GoodBrother, we may claim that the first solutions in this direction are starting to appear in the literature. A multidisciplinary 4 debate among experts and stakeholders is paving the way towards AAL ensuring ergonomics, usability, acceptance and privacy preservation. The DIANA, PAAL, and VisuAAL projects are examples of this fresh approach. This report provides the reader with a review of the most recent advances in audio- and video-based monitoring technologies for AAL. It has been drafted as a collective effort of WG3 to supply an introduction to AAL, its evolution over time and its main functional and technological underpinnings. In this respect, the report contributes to the field with the outline of a new generation of ethical-aware AAL technologies and a proposal for a novel comprehensive taxonomy of AAL systems and applications. Moreover, the report allows non-technical readers to gather an overview of the main components of an AAL system and how these function and interact with the end-users. The report illustrates the state of the art of the most successful AAL applications and functions based on audio and video data, namely (i) lifelogging and self-monitoring, (ii) remote monitoring of vital signs, (iii) emotional state recognition, (iv) food intake monitoring, activity and behaviour recognition, (v) activity and personal assistance, (vi) gesture recognition, (vii) fall detection and prevention, (viii) mobility assessment and frailty recognition, and (ix) cognitive and motor rehabilitation. For these application scenarios, the report illustrates the state of play in terms of scientific advances, available products and research project. The open challenges are also highlighted. The report ends with an overview of the challenges, the hindrances and the opportunities posed by the uptake in real world settings of AAL technologies. In this respect, the report illustrates the current procedural and technological approaches to cope with acceptability, usability and trust in the AAL technology, by surveying strategies and approaches to co-design, to privacy preservation in video and audio data, to transparency and explainability in data processing, and to data transmission and communication. User acceptance and ethical considerations are also debated. Finally, the potentials coming from the silver economy are overviewed.publishedVersio

Peripheral interaction

In our everyday life we carry out a multitude of activities in parallel without focusing our attention explicitly on them. We drink a cup of tea while reading a book, we signal a colleague passing by with a hand gesture, that we are concentrated right now and that he should wait one moment, or we walk a few steps backwards while taking photos. Many of these interactions - like drinking, sending signals via gestures or walking - are rather complex by themselves. By means of learning and training, however, these interactions become part of our routines and habits and therefore only consume little or no attentional resources. In contrast, when interacting with digital devices, we are often asked for our full attention. To carry out - even small and marginal tasks - we are regularly forced to switch windows, do precise interactions (e.g., pointing with the mouse) and thereby these systems trigger context and focus switches, disrupting us in our main focus and task. Peripheral interaction aims at making use of human capabilities and senses like divided attention, spatial memory and proprioception to support interaction with digital devices in the periphery of the attention, consequently quasi-parallel to another primary task. In this thesis we investigate peripheral interaction in the context of a standard desktop computer environment. We explore three interaction styles for peripheral interaction: graspable interaction, touch input and freehand gestures. StaTube investigates graspable interaction in the domain of instant messaging, while the Appointment Projection uses simple wiping gestures to access information about upcoming appointments. These two explorations focus on one interaction style each and offer first insights into the general benefits of peripheral interaction. In the following we carried out two studies comparing all three interaction styles (graspable, touch, freehand) for audio player control and for dealing with notifications. We found that all three interaction styles are generally fit for peripheral interaction but come with different advantages and disadvantages. The last set of explorative studies deals with the ability to recall spatial locations in 2D as well as 3D. The Unadorned Desk makes use of the physical space around the desktop computer and thereby offers an extended interaction space to store and retrieve virtual items such as commands, applications or tools. Finally, evaluation of peripheral interaction is not straightforward as the systems are designed to blend into the environment and not draw attention on them. We propose an additional evaluation method for the lab to complement the current evaluation practice in the field. The main contributions of this thesis are (1) an exhaustive classification and a more detailed look at manual peripheral interaction for tangible, touch and freehand interaction. Based on these exploration with all three interaction styles, we offer (2) implications in terms of overall benefits of peripheral interaction, learnability and habituation, visual and mental attention, feedback and handedness for future peripheral interaction design. Finally, derived from a diverse set of user studies, we assess (3) evaluation strategies enriching the design process for peripheral interaction.In unserem täglichen Leben führen wir eine große Anzahl an Aktivitäten parallel aus ohne uns explizit darauf zu konzentrieren. Wir trinken Tee während wir ein Buch lesen, wir signalisieren einem Kollegen durch eine Handgeste, dass wir gerade konzentriert sind und er einen Moment warten soll oder wir gehen ein paar Schritte rückwärts während wir fotografieren. Viele dieser Aktivitäten - wie beispielsweise Trinken, Gestikulieren und Laufen - sind an sich komplex. Durch Training werden diese Tätigkeiten allerdings Teil unserer Routinen und Gewohnheiten, und beanspruchen daher nur noch wenig oder sogar keine Aufmerksamkeit. Im Gegensatz dazu, verlangen digitale Geräte meist unsere volle Aufmerksamkeit während der Interaktion. Um - oftmals nur kleine - Aufgaben durchzuführen, müssen wir Fenster wechseln, präzise Aktionen durchführen (z.B. mit dem Mauszeiger zielen) und werden dabei durch die Systeme zu einem Kontext- und Fokuswechsel gezwungen. Periphere Interaktion hingegen macht sich menschliche Fähigkeiten wie geteilte Aufmerksamkeit, das räumliche Gedächtnis und Propriozeption zu Nutze um Interaktion mit digitalen Geräten am Rande der Aufmerksamkeit also der Peripherie zu ermöglichen -- quasi-parallel zu einem anderen Primärtask. In dieser Arbeit untersuchen wir Periphere Interaktion am Computerarbeitsplatz. Dabei betrachten wir drei verschiedene Interaktionsstile: Begreifbare Interaktion (graspable), Touch Eingabe und Freiraum Gestik (freehand). StaTube untersucht Begreifbare Interaktion am Beispiel von Instant Messaging, während die Appointment Projection einfache Wischgesten nutzt, um Informationen nahender Termine verfügbar zu machen. Diese beiden Untersuchungen betrachten jeweils einen Interaktionsstil und beleuchten erste Vorteile, die durch Periphere Interaktion erzielt werden können. Aufbauend darauf führen wir zwei vergleichende Studien zwischen allen drei Interaktionsstilen durch. Als Anwendungsszenarien dienen Musiksteuerung und der Umgang mit Benachrichtigungsfenstern. Alle drei Interaktionsstile können erfolgreich für Periphere Interaktion eingesetzt werden, haben aber verschiedene Vor- und Nachteile. Die letzte Gruppe von Studien befasst sich mit dem räumlichen Gedächtnis in 2D und 3D. Das Unadorned Desk nutzt den physikalischen Raum neben dem Desktop Computer um virtuelle Objekte, beispielsweise Funktionen, Anwendungen oder Werkzeuge, zu lagern. Darüber hinaus ist die Evaluation von Peripherer Interaktion anspruchsvoll, da sich die Systeme in die Umwelt integrieren und gerade keine Aufmerksamkeit auf sich ziehen sollen. Wir schlagen eine Evaluationsmethode für das Labor vor, um die derzeitig vorherrschenden Evaluationsmethoden in diesem Forschungsfeld zu ergänzen. Die Kernbeiträge dieser Arbeit sind eine (1) umfassende Klassifizierung und ein detaillierter Blick auf manuelle Periphere Interaktion, namentlich Begreifbare Interaktion, Touch Eingabe und Freiraum Gestik. Basierend auf unseren Untersuchungen ziehen wir (2) Schlussfolgerungen, die den generellen Nutzen von Peripherer Interaktion darlegen und Bereiche wie die Erlernbarkeit und Gewöhnung, visuelle und mentale Aufmerksamkeit, Feedback so wie Händigkeit beleuchten um zukünftige Projekte im Bereich der Peripheren Interaktion zu unterstützen. Aufbauend auf den verschiedenen Nutzerstudien, diskutieren wir Evaluationsstrategien um den Entwicklungsprozess Peripherer Interaktion zu unterstützen

Real-time immersive human-computer interaction based on tracking and recognition of dynamic hand gestures

With fast developing and ever growing use of computer based technologies, human-computer interaction (HCI) plays an increasingly pivotal role. In virtual reality (VR), HCI technologies provide not only a better understanding of three-dimensional shapes and spaces, but also sensory immersion and physical interaction. With the hand based HCI being a key HCI modality for object manipulation and gesture based communication, challenges are presented to provide users a natural, intuitive, effortless, precise, and real-time method for HCI based on dynamic hand gestures, due to the complexity of hand postures formed by multiple joints with high degrees-of-freedom, the speed of hand movements with highly variable trajectories and rapid direction changes, and the precision required for interaction between hands and objects in the virtual world. Presented in this thesis is the design and development of a novel real-time HCI system based on a unique combination of a pair of data gloves based on fibre-optic curvature sensors to acquire finger joint angles, a hybrid tracking system based on inertia and ultrasound to capture hand position and orientation, and a stereoscopic display system to provide an immersive visual feedback. The potential and effectiveness of the proposed system is demonstrated through a number of applications, namely, hand gesture based virtual object manipulation and visualisation, hand gesture based direct sign writing, and hand gesture based finger spelling. For virtual object manipulation and visualisation, the system is shown to allow a user to select, translate, rotate, scale, release and visualise virtual objects (presented using graphics and volume data) in three-dimensional space using natural hand gestures in real-time. For direct sign writing, the system is shown to be able to display immediately the corresponding SignWriting symbols signed by a user using three different signing sequences and a range of complex hand gestures, which consist of various combinations of hand postures (with each finger open, half-bent, closed, adduction and abduction), eight hand orientations in horizontal/vertical plans, three palm facing directions, and various hand movements (which can have eight directions in horizontal/vertical plans, and can be repetitive, straight/curve, clockwise/anti-clockwise). The development includes a special visual interface to give not only a stereoscopic view of hand gestures and movements, but also a structured visual feedback for each stage of the signing sequence. An excellent basis is therefore formed to develop a full HCI based on all human gestures by integrating the proposed system with facial expression and body posture recognition methods. Furthermore, for finger spelling, the system is shown to be able to recognise five vowels signed by two hands using the British Sign Language in real-time.EThOS - Electronic Theses Online ServiceGBUnited Kingdo

Design of a Multi-biometric Platform, based on physical traits and physiological measures: Face, Iris, Ear, ECG and EEG

Security and safety is one the main concerns both for governments and for private companies in the last years so raising growing interests and investments in the area of biometric recognition and video surveillance, especially after the sad happenings of September 2001. Outlays assessments of the U.S. government for the years 2001-2005 estimate that the homeland security spending climbed from $56.0 billions of dollars in 2001 to almost$100 billion of 2005. In this lapse of time, new pattern recognition techniques have been developed and, even more important, new biometric traits have been investigated and refined; besides the well-known physical and behavioral characteristics, also physiological measures have been studied, so providing more features to enhance discrimination capabilities of individuals. This dissertation proposes the design of a multimodal biometric platform, FAIRY, based on the following biometric traits: ear, face, iris EEG and ECG signals. In the thesis the modular architecture of the platform has been presented, together with the results obtained for the solution to the recognition problems related to the different biometrics and their possible fusion. Finally, an analysis of the pattern recognition issues concerning the area of videosurveillance has been discussed

Design of a Multi-biometric Platform, based on physical traits and physiological measures: Face, Iris, Ear, ECG and EEG

Security and safety is one the main concerns both for governments and for private companies in the last years so raising growing interests and investments in the area of biometric recognition and video surveillance, especially after the sad happenings of September 2001. Outlays assessments of the U.S. government for the years 2001-2005 estimate that the homeland security spending climbed from $56.0 billions of dollars in 2001 to almost$100 billion of 2005. In this lapse of time, new pattern recognition techniques have been developed and, even more important, new biometric traits have been investigated and refined; besides the well-known physical and behavioral characteristics, also physiological measures have been studied, so providing more features to enhance discrimination capabilities of individuals. This dissertation proposes the design of a multimodal biometric platform, FAIRY, based on the following biometric traits: ear, face, iris EEG and ECG signals. In the thesis the modular architecture of the platform has been presented, together with the results obtained for the solution to the recognition problems related to the different biometrics and their possible fusion. Finally, an analysis of the pattern recognition issues concerning the area of videosurveillance has been discussed