Computer vision methods for unconstrained gesture recognition in the context of sign language annotation

Cette thèse porte sur l'étude des méthodes de vision par ordinateur pour la reconnaissance de gestes naturels dans le contexte de l'annotation de la Langue des Signes. La langue des signes (LS) est une langue gestuelle développée par les sourds pour communiquer. Un énoncé en LS consiste en une séquence de signes réalisés par les mains, accompagnés d'expressions du visage et de mouvements du haut du corps, permettant de transmettre des informations en parallèles dans le discours. Même si les signes sont définis dans des dictionnaires, on trouve une très grande variabilité liée au contexte lors de leur réalisation. De plus, les signes sont souvent séparés par des mouvements de co-articulation. Cette extrême variabilité et l'effet de co-articulation représentent un problème important dans les recherches en traitement automatique de la LS. Il est donc nécessaire d'avoir de nombreuses vidéos annotées en LS, si l'on veut étudier cette langue et utiliser des méthodes d'apprentissage automatique. Les annotations de vidéo en LS sont réalisées manuellement par des linguistes ou experts en LS, ce qui est source d'erreur, non reproductible et extrêmement chronophage. De plus, la qualité des annotations dépend des connaissances en LS de l'annotateur. L'association de l'expertise de l'annotateur aux traitements automatiques facilite cette tâche et représente un gain de temps et de robustesse. Le but de nos recherches est d'étudier des méthodes de traitement d'images afin d'assister l'annotation des corpus vidéo: suivi des composantes corporelles, segmentation des mains, segmentation temporelle, reconnaissance de gloses. Au cours de cette thèse nous avons étudié un ensemble de méthodes permettant de réaliser l'annotation en glose. Dans un premier temps, nous cherchons à détecter les limites de début et fin de signe. Cette méthode d'annotation nécessite plusieurs traitements de bas niveau afin de segmenter les signes et d'extraire les caractéristiques de mouvement et de forme de la main. D'abord nous proposons une méthode de suivi des composantes corporelles robuste aux occultations basée sur le filtrage particulaire. Ensuite, un algorithme de segmentation des mains est développé afin d'extraire la région des mains même quand elles se trouvent devant le visage. Puis, les caractéristiques de mouvement sont utilisées pour réaliser une première segmentation temporelle des signes qui est par la suite améliorée grâce à l'utilisation de caractéristiques de forme. En effet celles-ci permettent de supprimer les limites de segmentation détectées en milieu des signes. Une fois les signes segmentés, on procède à l'extraction de caractéristiques visuelles pour leur reconnaissance en termes de gloses à l'aide de modèles phonologiques. Nous avons évalué nos algorithmes à l'aide de corpus internationaux, afin de montrer leur avantages et limitations. L'évaluation montre la robustesse de nos méthodes par rapport à la dynamique et le grand nombre d'occultations entre les différents membres. L'annotation résultante est indépendante de l'annotateur et représente un gain de robustese important.This PhD thesis concerns the study of computer vision methods for the automatic recognition of unconstrained gestures in the context of sign language annotation. Sign Language (SL) is a visual-gestural language developed by deaf communities. Continuous SL consists on a sequence of signs performed one after another involving manual and non-manual features conveying simultaneous information. Even though standard signs are defined in dictionaries, we find a huge variability caused by the context-dependency of signs. In addition signs are often linked by movement epenthesis which consists on the meaningless gesture between signs. The huge variability and the co-articulation effect represent a challenging problem during automatic SL processing. It is necessary to have numerous annotated video corpus in order to train statistical machine translators and study this language. Generally the annotation of SL video corpus is manually performed by linguists or computer scientists experienced in SL. However manual annotation is error-prone, unreproducible and time consuming. In addition de quality of the results depends on the SL annotators knowledge. Associating annotator knowledge to image processing techniques facilitates the annotation task increasing robustness and speeding up the required time. The goal of this research concerns on the study and development of image processing technique in order to assist the annotation of SL video corpus: body tracking, hand segmentation, temporal segmentation, gloss recognition. Along this PhD thesis we address the problem of gloss annotation of SL video corpus. First of all we intend to detect the limits corresponding to the beginning and end of a sign. This annotation method requires several low level approaches for performing temporal segmentation and for extracting motion and hand shape features. First we propose a particle filter based approach for robustly tracking hand and face robust to occlusions. Then a segmentation method for extracting hand when it is in front of the face has been developed. Motion is used for segmenting signs and later hand shape is used to improve the results. Indeed hand shape allows to delete limits detected in the middle of a sign. Once signs have been segmented we proceed to the gloss recognition using lexical description of signs. We have evaluated our algorithms using international corpus, in order to show their advantages and limitations. The evaluation has shown the robustness of the proposed methods with respect to high dynamics and numerous occlusions between body parts. Resulting annotation is independent on the annotator and represents a gain on annotation consistency

Designing a visual component of communication within 3D avatar virtual worlds

Merged with duplicate record 10026.1/2600 on 08.20.2017 by CS (TIS)Over the last few years 3D avatar virtual worlds (AVW) have emerged on the Internet. These are computer generated, multi-user, graphical spaces within which people meet, form social groups and interact with each other in real time, typically through the exchange of text or audio messages. Each user is represented within the space by a digital image known as an avatar, which is usually humanoid in form, and is predominantly under the control of the person it represents. This thesisd escribesa creativep roject that is concernedw ith aspectso f social communication between users of "Ws. In particular, an avatar is designed that is capable of performing body language, and a set of useful gestures are implemented that support aspects of social interaction and integrate with verbal discourse in a meaningful way. In addition to this, a number of scenic properties are derived that enable better comprehension of the non verbal communication, e. g. spatial arrangement, camera position and lighting effects. The research consists of a number of interrelated design activities which include reviewing the literature on avatar design in order to locate goals and variety of the project, therefore building on the on the work of others; a comparative review of three popular 3D AVWs to explore the design problem; a study that aims to gain an understanding of the social dynamics involved; the adaptation of a diagrammatic technique for the purpose of modelling social interaction; the development of 2D and 3D prototype techniques exploring the application of the social interaction modelling technique; a body of creative work developing ideas for conveying non verbal communication and the appraisal of the effectiveness of this creative work. The research contributes to the field of avatar design in a number of ways. Firstly, it develops our understanding of social dynamics in virtual worlds. Secondly, it postulates modes of non verbal communication for both individuals and social groups that supports multi-participatory social discourse. Additionally, a number of useful research techniques have been devised, such as a linear diagramming technique that can be used to represent the structure of conversation thereby facilitating the exploration and understanding of the dynamics of AVW social discourse. The work is of interest to those working in the field of avatar and multi-user virtual world design. It may also be of interest to anyone thinking of using an avatar virtual world for the application of collaborative leaming, collaborative games and conferencing

Building Embodied Conversational Agents:Observations on human nonverbal behaviour as a resource for the development of artificial characters

"Wow this is so cool!" This is what I most probably yelled, back in the 90s, when my first computer program on our MSX computer turned out to do exactly what I wanted it to do. The program contained the following instruction: COLOR 10(1.1) After hitting enter, it would change the screen color from light blue to dark yellow. A few years after that experience, Microsoft Windows was introduced. Windows came with an intuitive graphical user interface that was designed to allow all people, so also those who would not consider themselves to be experienced computer addicts, to interact with the computer. This was a major step forward in human-computer interaction, as from that point forward no complex programming skills were required anymore to perform such actions as adapting the screen color. Changing the background was just a matter of pointing the mouse to the desired color on a color palette. "Wow this is so cool!". This is what I shouted, again, 20 years later. This time my new smartphone successfully skipped to the next song on Spotify because I literally told my smartphone, with my voice, to do so. Being able to operate your smartphone with natural language through voice-control can be extremely handy, for instance when listening to music while showering. Again, the option to handle a computer with voice instructions turned out to be a significant optimization in human-computer interaction. From now on, computers could be instructed without the use of a screen, mouse or keyboard, and instead could operate successfully simply by telling the machine what to do. In other words, I have personally witnessed how, within only a few decades, the way people interact with computers has changed drastically, starting as a rather technical and abstract enterprise to becoming something that was both natural and intuitive, and did not require any advanced computer background. Accordingly, while computers used to be machines that could only be operated by technically-oriented individuals, they had gradually changed into devices that are part of many people’s household, just as much as a television, a vacuum cleaner or a microwave oven. The introduction of voice control is a significant feature of the newer generation of interfaces in the sense that these have become more "antropomorphic" and try to mimic the way people interact in daily life, where indeed the voice is a universally used device that humans exploit in their exchanges with others. The question then arises whether it would be possible to go even one step further, where people, like in science-fiction movies, interact with avatars or humanoid robots, whereby users can have a proper conversation with a computer-simulated human that is indistinguishable from a real human. An interaction with a human-like representation of a computer that behaves, talks and reacts like a real person would imply that the computer is able to not only produce and understand messages transmitted auditorily through the voice, but also could rely on the perception and generation of different forms of body language, such as facial expressions, gestures or body posture. At the time of writing, developments of this next step in human-computer interaction are in full swing, but the type of such interactions is still rather constrained when compared to the way humans have their exchanges with other humans. It is interesting to reflect on how such future humanmachine interactions may look like. When we consider other products that have been created in history, it sometimes is striking to see that some of these have been inspired by things that can be observed in our environment, yet at the same do not have to be exact copies of those phenomena. For instance, an airplane has wings just as birds, yet the wings of an airplane do not make those typical movements a bird would produce to fly. Moreover, an airplane has wheels, whereas a bird has legs. At the same time, an airplane has made it possible for a humans to cover long distances in a fast and smooth manner in a way that was unthinkable before it was invented. The example of the airplane shows how new technologies can have "unnatural" properties, but can nonetheless be very beneficial and impactful for human beings. This dissertation centers on this practical question of how virtual humans can be programmed to act more human-like. The four studies presented in this dissertation all have the equivalent underlying question of how parts of human behavior can be captured, such that computers can use it to become more human-like. Each study differs in method, perspective and specific questions, but they are all aimed to gain insights and directions that would help further push the computer developments of human-like behavior and investigate (the simulation of) human conversational behavior. The rest of this introductory chapter gives a general overview of virtual humans (also known as embodied conversational agents), their potential uses and the engineering challenges, followed by an overview of the four studies

Building Embodied Conversational Agents:Observations on human nonverbal behaviour as a resource for the development of artificial characters

"Wow this is so cool!" This is what I most probably yelled, back in the 90s, when my first computer program on our MSX computer turned out to do exactly what I wanted it to do. The program contained the following instruction: COLOR 10(1.1) After hitting enter, it would change the screen color from light blue to dark yellow. A few years after that experience, Microsoft Windows was introduced. Windows came with an intuitive graphical user interface that was designed to allow all people, so also those who would not consider themselves to be experienced computer addicts, to interact with the computer. This was a major step forward in human-computer interaction, as from that point forward no complex programming skills were required anymore to perform such actions as adapting the screen color. Changing the background was just a matter of pointing the mouse to the desired color on a color palette. "Wow this is so cool!". This is what I shouted, again, 20 years later. This time my new smartphone successfully skipped to the next song on Spotify because I literally told my smartphone, with my voice, to do so. Being able to operate your smartphone with natural language through voice-control can be extremely handy, for instance when listening to music while showering. Again, the option to handle a computer with voice instructions turned out to be a significant optimization in human-computer interaction. From now on, computers could be instructed without the use of a screen, mouse or keyboard, and instead could operate successfully simply by telling the machine what to do. In other words, I have personally witnessed how, within only a few decades, the way people interact with computers has changed drastically, starting as a rather technical and abstract enterprise to becoming something that was both natural and intuitive, and did not require any advanced computer background. Accordingly, while computers used to be machines that could only be operated by technically-oriented individuals, they had gradually changed into devices that are part of many people’s household, just as much as a television, a vacuum cleaner or a microwave oven. The introduction of voice control is a significant feature of the newer generation of interfaces in the sense that these have become more "antropomorphic" and try to mimic the way people interact in daily life, where indeed the voice is a universally used device that humans exploit in their exchanges with others. The question then arises whether it would be possible to go even one step further, where people, like in science-fiction movies, interact with avatars or humanoid robots, whereby users can have a proper conversation with a computer-simulated human that is indistinguishable from a real human. An interaction with a human-like representation of a computer that behaves, talks and reacts like a real person would imply that the computer is able to not only produce and understand messages transmitted auditorily through the voice, but also could rely on the perception and generation of different forms of body language, such as facial expressions, gestures or body posture. At the time of writing, developments of this next step in human-computer interaction are in full swing, but the type of such interactions is still rather constrained when compared to the way humans have their exchanges with other humans. It is interesting to reflect on how such future humanmachine interactions may look like. When we consider other products that have been created in history, it sometimes is striking to see that some of these have been inspired by things that can be observed in our environment, yet at the same do not have to be exact copies of those phenomena. For instance, an airplane has wings just as birds, yet the wings of an airplane do not make those typical movements a bird would produce to fly. Moreover, an airplane has wheels, whereas a bird has legs. At the same time, an airplane has made it possible for a humans to cover long distances in a fast and smooth manner in a way that was unthinkable before it was invented. The example of the airplane shows how new technologies can have "unnatural" properties, but can nonetheless be very beneficial and impactful for human beings. This dissertation centers on this practical question of how virtual humans can be programmed to act more human-like. The four studies presented in this dissertation all have the equivalent underlying question of how parts of human behavior can be captured, such that computers can use it to become more human-like. Each study differs in method, perspective and specific questions, but they are all aimed to gain insights and directions that would help further push the computer developments of human-like behavior and investigate (the simulation of) human conversational behavior. The rest of this introductory chapter gives a general overview of virtual humans (also known as embodied conversational agents), their potential uses and the engineering challenges, followed by an overview of the four studies

Space and place as expressive categories in videogames

This thesis was submitted for the degree of Doctor of Philosophy and awarded by Brunel University.This thesis sets out to explore some of the ways in which videogames use space as a means of expression. This expression takes place in two registers: representation and embodiment. Representation is understood as a form of expression in which messages and ideas are communicated. Embodiment is understood as a form of expression in which the player is encouraged to take up a particular position in relation to the game. This distinction between representation and embodiment is useful analytically but the thesis attempts to synthesise these modes in order to account for the experience of playing videogames, where representation and embodiment are constantly happening and constantly influencing and shaping each other. Several methods are developed to analyse games in a way that brings these two modes to the fore. The thesis attempts to arrive at a number of spatial aesthetics of videogames by adapting methods from game studies, literary criticism, phenomenology, onomastics (the study of names), cartographic theory, choreography and architectural and urban formation analysis

On Myth & Music, Legend & Landscape, Science, Stars, & Story; A Portfolio of Compositions: An Animist Aesthetic Argument on Symbolist Sound-Synergies, with Musings on Messiaen and the Taoism of Takemitzu, in the work of Mulvany.

The accompanying portfolio contains eight major works for disparate forces: Brú na Sídhe is a work for Large Orchestra; Bog Bodies is a work commissioned for a heterogenous Quintet; Gáeth Ard Úar is a work written for a Solo Bass/Contrabass Clarinet in B-flat; a Piano Trio, SzyzygyS, follows; a work entitled Blackwater is written for Solo Guitar; a diptych, named Sigil, written for a Dectet of Harp and Vibraphone with double String Quartet, follows; then, O Unworn World for a ten part Choral ensemble is presented as a meditation on a poem by Patrick Kavanagh; and, the last of the works in the portfolio, is a String Quartet entitled Wodwo. These pieces are conceived using an aesthetic guide provided by two substantially different composers: Tōru Takemitsu and Olivier Messiaen. The connection with these composers is explored in a chapter entitled the Aesthetic Review, Chapter One; and, indeed, throughout the commentaries provided in the subsequent chapters dealing with each of the works in the order stated. It is noted that, while these composers provide a guide to the formulation and execution, and even the subject matter, of ideas in music, it is not the case that the music will sound in any way similar to either of these composers. The divergent techniques of composition described within are substantiated with reference to these composers, and fall into three broadly defined categories: the Constructivist, the Impressionist, and the Deductivist approaches to style. The works also contain a similar approach to the subject matter of the inspiration. A thread of Pagan Spirituality and Mythology can be traced through all of the works, to varying degrees. They contain, in addition, a preponderance on duality as a concept; the duality of Science and Natural Spirituality; the duality of defined and undefined sounds; and the duality of composer and performer. With these works, the composer wishes to demonstrate a mastery of compositional techniques and a thorough and methodical approach to the realisation of a conceptual framework

Embodiment and the Arts: Views from South Africa

Embodiment and the Arts: Views from South Africa presents a diversity of views on the nature and status of the body in relation to acting, advertisements, designs, films, installations, music, photographs, performance, typography, and video works. Applying the methodologies of phenomenology, hermeneutic phenomenology, embodied perception, ecological psychology, and sense-based research, the authors place the body at the centre of their analyses. The cornerstone of the research presented here is the view that aesthetic experience is active and engaged rather than passive and disinterested. This novel volume offers a rich and diverse range of applications of the paradigm of embodiment to the arts in South Africa.Publishe

Machine Sensation

Emphasising the alien qualities of anthropomorphic technologies, Machine Sensation makes a conscious effort to increase rather than decrease the tension between nonhuman and human experience. In a series of rigorously executed cases studies, including natural user interfaces, artificial intelligence as well as sex robots, Leach shows how object-oriented ontology enables one to insist upon the unhuman nature of technology while acknowledging its immense power and significance in human life. Machine Sensation meticulously engages OOO, Actor Network Theory, the philosophy of technology, cybernetics and posthumanism in innovative and gripping ways