Building a sign language corpus for use in machine translation

In recent years data-driven methods of machine translation (MT) have overtaken rule-based approaches as the predominant means of automatically translating between languages. A pre-requisite for such an approach is a parallel corpus of the source and target languages. Technological developments in sign language (SL) capturing, analysis and processing tools now mean that SL corpora are becoming increasingly available. With transcription and language analysis tools being mainly designed and used for linguistic purposes, we describe the process of creating a multimedia parallel corpus specifically for the purposes of English to Irish Sign Language (ISL) MT. As part of our larger project on localisation, our research is focussed on developing assistive technology for patients with limited English in the domain of healthcare. Focussing on the first point of contact a patient has with a GP’s office, the medical secretary, we sought to develop a corpus from the dialogue between the two parties when scheduling an appointment. Throughout the development process we have created one parallel corpus in six different modalities from this initial dialogue. In this paper we discuss the multi-stage process of the development of this parallel corpus as individual and interdependent entities, both for our own MT purposes and their usefulness in the wider MT and SL research domains

Multi-Level Sensory Interpretation and Adaptation in a Mobile Cube

Signals from sensors are often analyzed in a sequence of steps, starting with the raw sensor data and eventually ending up with a classification or abstraction of these data. This paper will give a practical example of how the same information can be trained and used to initiate multiple interpretations of the same data on different, application-oriented levels. Crucially, the focus is on expanding embedded analysis software, rather than adding more powerful, but possibly resource-hungry, sensors. Our illustration of this approach involves a tangible input device the shape of a cube that relies exclusively on lowcost accelerometers. The cube supports calibration with user supervision, it can tell which of its sides is on top, give an estimate of its orientation relative to the user, and recognize basic gestures

Representation of Samba dance gestures, using a multi-modal analysis approach

In this paper we propose an approach for the representation of dance gestures in Samba dance. This representation is based on a video analysis of body movements, carried out from the viewpoint of the musical meter. Our method provides the periods, a measure of energy and a visual representation of periodic movement in dance. The method is applied to a limited universe of Samba dances and music, which is used to illustrate the usefulness of the approach

Computer-based tracking, analysis, and visualization of linguistically significant nonmanual events in American Sign Language (ASL)

Our linguistically annotated American Sign Language (ASL) corpora have formed a basis for research to automate detection by computer of essential linguistic information conveyed through facial expressions and head movements. We have tracked head position and facial deformations, and used computational learning to discern specific grammatical markings. Our ability to detect, identify, and temporally localize the occurrence of such markings in ASL videos has recently been improved by incorporation of (1) new techniques for deformable model-based 3D tracking of head position and facial expressions, which provide significantly better tracking accuracy and recover quickly from temporary loss of track due to occlusion; and (2) a computational learning approach incorporating 2-level Conditional Random Fields (CRFs), suited to the multi-scale spatio-temporal characteristics of the data, which analyses not only low-level appearance characteristics, but also the patterns that enable identification of significant gestural components, such as periodic head movements and raised or lowered eyebrows. Here we summarize our linguistically motivated computational approach and the results for detection and recognition of nonmanual grammatical markings; demonstrate our data visualizations, and discuss the relevance for linguistic research; and describe work underway to enable such visualizations to be produced over large corpora and shared publicly on the Web

From Autonomous to Performative Control of Timbral Spatialisation

Timbral spatialisation is one such process that requires the independent control of potentially thousands of parameters (Torchia, et al., 2003). Current research on controlling timbral spatialisation has focussed either on automated generative systems, or suggested that to design trajectories in software is to write every movement line by line (Normandeau, 2009). This research proposes that Wave Terrain Synthesis may be used as an effective bridging control structure for timbral spatialisation, enabling the performative control of large numbers of parameter sets associated with software. This methodology also allows for compact interactive mapping possibilities for a physical controller, and may also be effectively mapped gesturall

Intérprete de lenguaje de signos en español multidispositivo

Versión electrónica de la ponencia presentada en Conferencia Ibero-Americana IADIS WWW/Internet 2006, celebrado en Murcia en 2006En este artículo presentamos un transcriptor de texto a lenguaje de signos distribuido y multidispositivo. La presentación al usuario final del lenguaje de signos es realizada por un personaje animado en tres dimensiones. Este transcriptor está creado para adaptar su salida a la capacidad de proceso del dispositivo receptor. Por lo que puede ser utilizado por un usuario en un ordenador personal para transcribir una página Web, o en un teléfono móvil para transcribir una conversación (utilizando un reconocedor de voz). La flexibilidad del sistema permite adaptarlo a varios idiomas o usarlo como un simple elemento para mejorar una interfaz multimedia

Intelligent Interactive Multimedia by Converging the Intention of Spectator and Multimedia Creator

In this research, we propose a new approach on how human and technology interact with each other. Here, by enhancing the current HCI framework, it will enable interaction between human and technology become more effective and ideally. The aim of this research is to create an Intelligent Interactive Multimedia by converging the intention of spectator and multimedia creator. Several methods are proposed to achieve the conception of Intelligent Interactive Multimedia. Digital Drawing Block is the interactive multimedia with the initial intention of multimedia creator and it forms an interaction with spectator. Spectator intention has been categorized into four common categories, additionally, five features of hand gesture recognition is proposed to deduce the spectator intention. All these five features will be captured by the web-cam during the spectator’s interaction with the Digital Drawing Block. Moreover, captured features will be sent to the machine learning for analyzing. Proposed user models are to assist the machine learning to evaluate the most appropriate category of human behaviour which matches the spectator actual intention. Lastly, graphic that represents spectator intention will be generated together with the initial intention of multimedia creator. The new creation from spectator and multimedia creator will be displayed through the Digital Drawing Block. The conception of Intelligent Interactive Multimedia can represent as 70%'s effort of Multimedia Creator + 30%'s effort of spectator