Multimodal Dialogue Management for Multiparty Interaction with Infants

We present dialogue management routines for a system to engage in multiparty agent-infant interaction. The ultimate purpose of this research is to help infants learn a visual sign language by engaging them in naturalistic and socially contingent conversations during an early-life critical period for language development (ages 6 to 12 months) as initiated by an artificial agent. As a first step, we focus on creating and maintaining agent-infant engagement that elicits appropriate and socially contingent responses from the baby. Our system includes two agents, a physical robot and an animated virtual human. The system's multimodal perception includes an eye-tracker (measures attention) and a thermal infrared imaging camera (measures patterns of emotional arousal). A dialogue policy is presented that selects individual actions and planned multiparty sequences based on perceptual inputs about the baby's internal changing states of emotional engagement. The present version of the system was evaluated in interaction with 8 babies. All babies demonstrated spontaneous and sustained engagement with the agents for several minutes, with patterns of conversationally relevant and socially contingent behaviors. We further performed a detailed case-study analysis with annotation of all agent and baby behaviors. Results show that the baby's behaviors were generally relevant to agent conversations and contained direct evidence for socially contingent responses by the baby to specific linguistic samples produced by the avatar. This work demonstrates the potential for language learning from agents in very young babies and has especially broad implications regarding the use of artificial agents with babies who have minimal language exposure in early life

Speech to sign language translation system for Spanish

This paper describes the development of and the first experiments in a Spanish to sign language translation system in a real domain. The developed system focuses on the sentences spoken by an official when assisting people applying for, or renewing their Identity Card. The system translates official explanations into Spanish Sign Language (LSE: Lengua de Signos Espan¿ola) for Deaf people. The translation system is made up of a speech recognizer (for decoding the spoken utterance into a word sequence), a natural language translator (for converting a word sequence into a sequence of signs belonging to the sign language), and a 3D avatar animation module (for playing back the hand movements). Two proposals for natural language translation have been evaluated: a rule-based translation module (that computes sign confidence measures from the word confidence measures obtained in the speech recognition module) and a statistical translation module (in this case, parallel corpora were used for training the statistical model). The best configuration reported 31.6% SER (Sign Error Rate) and 0.5780 BLEU (BiLingual Evaluation Understudy). The paper also describes the eSIGN 3D avatar animation module (considering the sign confidence), and the limitations found when implementing a strategy for reducing the delay between the spoken utterance and the sign sequence animation

Training Language Mediators and Interpreters through Embodied Cognition, Immersive Learning and Virtual Reality: Didactic, Organizational and Cost Benefits

The use of Virtual Reality (VR) for training purposes has expanded exponentially in the last decade. This paper specifically examines the use of VR in the training of language mediators and interpreters. By way of exemplification, the paper describes a training project at the University of Palermo, the “Coopera” project. The physically-cognitively immersive ‘learning by doing’ factor that characterizes a VR educational experience is particularly well-suited to skill- based activities. The paper explains how the immersive nature of VR and the notion of ‘embodied cognition’ is particularly suited to mediator/interpreter training by plausibly reproducing daily life situations and providing a ‘risk- free’ environment in which students can safely practice (and make mistakes) without causing harm to themselves and others, and also to contain the high costs of mediator/interpreter training. We hope the Coopera project will help foster further research into new technologies in immersive language-mediation pedagogy

Co-designing a chatbot for and with refugees and migrants

An information portal, HandbookGermany.de, is developed to support the integration of refugees and migrants into society in Germany. However, the information-seeking process is exhausting, cumbersome, and even confusing if refugees and migrants are not proficient at using web services. In light of this, a chatbot-based conversational service is considered as an alternative to enhance the information-seeking experience. For the purpose of designing products and services for refugees and migrants, a great deal of research proposes employing co-design methods as an effective means. The overall aim of this thesis is to explore, understand, and define possibilities of improving refugees and migrants’ experiences of social integration by proposing an engaging and efficient chatbot solution. Furthermore, this thesis aims to explore the necessity of co-design approach as a critical methodology to develop solutions. Therefore, the main research question in this thesis is how can a co-design approach contribute to designing a chatbot supporting social integration within the context of refugees and migrants. User experience, problems, and needs are unveiled in depth by listening to migrants and refugees’ problems, behaviors, and expectations (i.e., document studies, questionnaires, cultural probes, and expert interviews), and observing how migrants interact with the chatbot (i.e., participant observations and empathy probes). The research findings are then transformed into design questions. The designer, developers, and migrants jointly generate concepts leveraging generative toolkits in co-design workshops. By using surveys, the Method for the Assessment of eXperience (MAX), and property checklists, the resulting concepts are later validated with refugees and migrants. As research through design, this thesis draws three conclusions. Firstly, the co-design approach benefits defining problems in the complex context of refugees and migrants by supporting them in expressing ideas and thoughts. The defined problems can then be converted into design questions that promote the proceeding of the design process. Secondly, the co-design approach helps to develop mature concepts, which lays a foundation for the final design. Thirdly, the utilization of co-design tools plays an essential role in validating and refining the solution efficiently, as they make ideas concrete and visible so that refugees and migrants can easily reflect on them throughout the whole design process

Methodology for developing an advanced communications system for the Deaf in a new domain

A methodology for developing an advanced communications system for the Deaf in a new domain is presented in this paper. This methodology is a user-centred design approach consisting of four main steps: requirement analysis, parallel corpus generation, technology adaptation to the new domain, and finally, system evaluation. During the requirement analysis, both the user and technical requirements are evaluated and defined. For generating the parallel corpus, it is necessary to collect Spanish sentences in the new domain and translate them into LSE (Lengua de Signos Española: Spanish Sign Language). LSE is represented by glosses and using video recordings. This corpus is used for training the two main modules of the advanced communications system to the new domain: the spoken Spanish into the LSE translation module and the Spanish generation from the LSE module. The main aspects to be generated are the vocabularies for both languages (Spanish words and signs), and the knowledge for translating in both directions. Finally, the field evaluation is carried out with deaf people using the advanced communications system to interact with hearing people in several scenarios. In this evaluation, the paper proposes several objective and subjective measurements for evaluating the performance. In this paper, the new considered domain is about dialogues in a hotel reception. Using this methodology, the system was developed in several months, obtaining very good performance: good translation rates (10% Sign Error Rate) with small processing times, allowing face-to-face dialogues

Profile, Spring 2008

Interactive Arts and Media Department Newsletter. 20 pages. Includes student, faculty, and alumni profiles, book and game reviews, calendar of events, internship experience, faculty article, and interviews with working professionals.https://digitalcommons.colum.edu/profile/1002/thumbnail.jp

Interactive Food and Beverage Marketing: Targeting Children and Youth in the Digital Age

Looks at the practices of food and beverage industry marketers in reaching youth via digital videos, cell phones, interactive games and social networking sites. Recommends imposing governmental regulations on marketing to children and adolescents

Translating bus information into sign language for deaf people

This paper describes the application of language translation technologies for generating bus information in Spanish Sign Language (LSE: Lengua de Signos Española). In this work, two main systems have been developed: the first for translating text messages from information panels and the second for translating spoken Spanish into natural conversations at the information point of the bus company. Both systems are made up of a natural language translator (for converting a word sentence into a sequence of LSE signs), and a 3D avatar animation module (for playing back the signs). For the natural language translator, two technological approaches have been analyzed and integrated: an example-based strategy and a statistical translator. When translating spoken utterances, it is also necessary to incorporate a speech recognizer for decoding the spoken utterance into a word sequence, prior to the language translation module. This paper includes a detailed description of the field evaluation carried out in this domain. This evaluation has been carried out at the customer information office in Madrid involving both real bus company employees and deaf people. The evaluation includes objective measurements from the system and information from questionnaires. In the field evaluation, the whole translation presents an SER (Sign Error Rate) of less than 10% and a BLEU greater than 90%