Say-it: Design of a Multimodal Game Interface for Children Based on CMU Sphinx 4 Framework

Nowadays, computer games are involved in a child’s education as tools for learning or practicing some academic skills. However, most educational games are designed without any considerations about children with motor system difficulties. Thus, the main benefit of multimodal interfaces is to allow for inclusive design, which will enable children with motor disabilities to use the same applications other children use. Since focus and concentration are major skills for children in the learning process, many physical games can be performed with assist from parents or teachers to practice these skills. This project aims to explore the effectiveness and the use of multimodal applications as computer-based exercises by implementing a multimodal system that offers an inclusive design for three interactive games to practice the skill to focus. In this project, Say-it is implemented as a functional multimodal prototype to demonstrate the value of multimodal interfaces in the education of young children. It is designed to provide the advantages of both current physical games and computer exercises and make these exercises available for children with wide rang of abilities. Also, Say-it can be considered as an experimental prototype to explore the performance of using CMU Sphinx 4 framework as an underlying speech recognition tool for the application. The results of using Say-it show that multimodal games can be designed using the existing speech recognition technologies, such as CMU Sphinx 4 framework. Also, using different input modalities in the proposed prototype makes the games more enjoyable and challenging for children

Digital interaction: where are we going?

In the framework of the AVI 2018 Conference, the interuniversity center ECONA has organized a thematic workshop on "Digital Interaction: where are we going?". Six contributions from the ECONA members investigate different perspectives around this thematic

Towards the Use of Dialog Systems to Facilitate Inclusive Education

Continuous advances in the development of information technologies have currently led to the possibility of accessing learning contents from anywhere, at anytime, and almost instantaneously. However, accessibility is not always the main objective in the design of educative applications, specifically to facilitate their adoption by disabled people. Different technologies have recently emerged to foster the accessibility of computers and new mobile devices, favoring a more natural communication between the student and the developed educative systems. This chapter describes innovative uses of multimodal dialog systems in education, with special emphasis in the advantages that they provide for creating inclusive applications and learning activities

New Multimodal Designs for Foreign Language Learning

Semiotiske multimodale teorier taler om nye affordanser (”handlemuligheder”) i medie- og læringslandskaber, hvilket både teoretisk og empirisk møder genklang i Universal Design for Learning (UDL) -og Computer-Assisted Language Learning (CALL)-litteraturen, men pga. de to tilganges henholdsvis neurodidaktiske og teknologiske orientering mangler begge et teoretisk fundament inden for semiotisk multimodalitet og læringsøkologi. Beriget med multimodalitetsteori og økologiske perspektiver kan UDL og CALL krydsbefrugtes til at danne et multimodalt og økologisk bevidst inkluderende design for sprogindlæring. Denne hypotese undersøges teoretisk som en del af et igangværende projekt og dernæst i et empirisk undersøgelsesdesign, der udforsker digital stilladsering. Multimodale og økologiske perspektiver anvendes til at analysere affordanser og økologier i CALL- og UDL-baserede læringsdesign. På baggrund af denne analyse opbygges et principstyret UDL-CALL læringsdesign. Til den empiriske afprøvning foreslås et mixed-methods-undersøgelsesdesign, og foreløbige undersøgelsesresultater præsenteres, der antyder UDL-CALL-designets gangbarhed.Semiotic multimodality theory speaks of new learning affordances in media ecologies, which is both theoretically and empirically echoed in UDL and in CALL literature, but owing to their neuro-didactic respectively technology-driven standpoints both approaches lack theoretical underpinnings for ecology and semiotic multimodality. Enhanced with multimodality theory and ecological perspectives UDL and CALL can crossbreed, forming a multimodally and ecologically aware inclusive design for language learning. This study from an ongoing project investigates the hypothesis from a theoretical and an empirical perspective, examining digital scaffolds. Multimodal-semiotic and ecological perspectives are used to analyse affordances and ecologies in CALL and UDL learning designs. From this analysis, a principled UDL-CALL learning design is constructed. For empirical testing, a mixed-methods research design is proposed, presenting preliminary results indicative of the design’s viability