Designing multimodal interaction for the visually impaired

Although multimodal computer input is believed to have advantages over unimodal input, little has been done to understand how to design a multimodal input mechanism to facilitate visually impaired users\u27 information access. This research investigates sighted and visually impaired users\u27 multimodal interaction choices when given an interaction grammar that supports speech and touch input modalities. It investigates whether task type, working memory load, or prevalence of errors in a given modality impact a user\u27s choice. Theories in human memory and attention are used to explain the users\u27 speech and touch input coordination. Among the abundant findings from this research, the following are the most important in guiding system design: (1) Multimodal input is likely to be used when it is available. (2) Users select input modalities based on the type of task undertaken. Users prefer touch input for navigation operations, but speech input for non-navigation operations. (3) When errors occur, users prefer to stay in the failing modality, instead of switching to another modality for error correction. (4) Despite the common multimodal usage patterns, there is still a high degree of individual differences in modality choices. Additional findings include: (I) Modality switching becomes more prevalent when lower working memory and attentional resources are required for the performance of other concurrent tasks. (2) Higher error rates increases modality switching but only under duress. (3) Training order affects modality usage. Teaching a modality first versus second increases the use of this modality in users\u27 task performance. In addition to discovering multimodal interaction patterns above, this research contributes to the field of human computer interaction design by: (1) presenting a design of an eyes-free multimodal information browser, (2) presenting a Wizard of Oz method for working with visually impaired users in order to observe their multimodal interaction. The overall contribution of this work is that of one of the early investigations into how speech and touch might be combined into a non-visual multimodal system that can effectively be used for eyes-free tasks

On intelligible multimodal visual analysis

Analyzing data becomes an important skill in a more and more digital world. Yet, many users are facing knowledge barriers preventing them to independently conduct their data analysis. To tear down some of these barriers, multimodal interaction for visual analysis has been proposed. Multimodal interaction through speech and touch enables not only experts, but also novice users to effortlessly interact with such kind of technology. However, current approaches do not take the user differences into account. In fact, whether visual analysis is intelligible ultimately depends on the user. In order to close this research gap, this dissertation explores how multimodal visual analysis can be personalized. To do so, it takes a holistic view. First, an intelligible task space of visual analysis tasks is defined by considering personalization potentials. This task space provides an initial basis for understanding how effective personalization in visual analysis can be approached. Second, empirical analyses on speech commands in visual analysis as well as used visualizations from scientific publications further reveal patterns and structures. These behavior-indicated findings help to better understand expectations towards multimodal visual analysis. Third, a technical prototype is designed considering the previous findings. Enriching the visual analysis by a persistent dialogue and a transparency of the underlying computations, conducted user studies show not only advantages, but address the relevance of considering the user’s characteristics. Finally, both communications channels – visualizations and dialogue – are personalized. Leveraging linguistic theory and reinforcement learning, the results highlight a positive effect of adjusting to the user. Especially when the user’s knowledge is exceeded, personalizations helps to improve the user experience. Overall, this dissertations confirms not only the importance of considering the user’s characteristics in multimodal visual analysis, but also provides insights on how an intelligible analysis can be achieved. By understanding the use of input modalities, a system can focus only on the user’s needs. By understanding preferences on the output modalities, the system can better adapt to the user. Combining both directions imporves user experience and contributes towards an intelligible multimodal visual analysis

Touching creativity; a review and early pilot test of haptic tooling to support design practice, within a distance learning curriculum

Machine haptics has been shown to assist and enhance human–computer interactions. Research from previous studies in the field of haptics has focused on developing a user’s sense of realism of touch when using a haptic device. This paper examines the use of haptics for education, specifically for creative online education. The paper is presented in two parts. First, a review of literature was conducted and used to aid the rationale and underpin the design of a pilot test. Second, a pilot test was designed using a single-point kinaesthetic haptic device with a haptic rendered interface, to support the assembly of a virtual design prototype. The pilot test proved to be extremely valuable in creating and developing a rich virtual environment for non-sighted and sighted participants to use. The results from the initial pilot test showed that although users were positive about their experience of using the haptic device, there were improvements to be made to the interface to enhance the user experience in the next phase of testing

Exploring young people's collaborative and creative processes using keyboard and computer based music technologies in formal and non-formal settings

In many UK music education settings, young people (11-17 years old) make music collectively. Despite this we currently lack understanding of the processes involved when collaboratively creating music, particularly when working around music technologies. To date, research has tended to focus on classroom-based collaborative interactions on well-defined tasks, where there is only one correct solution. As a result we know little about 1) the kinds of learning practices that emerge outside of school settings and 2) the processes young people engage in when working on open-ended, creative tasks. Addressing these areas, this research specifically set out to explore the nature of the creative process when composing music collaboratively using keyboards and sampling software, in school, community centre and music camp settings. The contextual relations or features of these different settings, such as the task setting, instruction and technology used and their influence on the creative music-making processes were examined. This was achieved through analysis of the young people's verbal dialogues, which resulted in greater understanding of the relations between context and creativity. The findings show that how the creative and musical content is organised, rather than the physical setting within which it takes place, plays a fundamental role in the types of talk and creative processes that emerge. Drawing on the results of the studies carried out, creativity was conceptualised as a cyclic process, with interdependent phases of exploration, discovery, elaboration, critical listening, refining and editing, recording and saving; with problem finding and discovery being central underlying drivers. Finally, the kinds of verbal dialogues that emerged across all the settings strongly indicated that traditional logical-deductive types of reasoning and talk are not necessary and may even be inappropriate for certain phases of the creative process. This finding is interesting and presents some challenges to our current understanding of collaborative learning. Consequently, it warrants further investigation. In sum, given the contemporary educational emphasis on self-directed and creative learners, the questions addressed in this thesis and the findings on the context and nature of the creative processes, and informal and formal learning, are considered timely and relevant

A CASE STUDY ON THE IMPLEMENTATION OF DISCIPLINARY LITERACY WITH MULTIMODAL DESIGN IN SOCIAL STUDIES

Social studies education provides education researchers with a less common opportunity to discover multimodal instructional methods for disciplinary literacy. During a 12-week period in 2021, four social studies teachers with at least one history course in a large suburban Mercer County, New Jersey school district participated in a case study to showcase how disciplinary literacy can be implemented using multimodal design. Given the existing lack of research on instructional literacy design for secondary grades, this study provides researchers and practitioners with multiple perspectives on how to maximize teaching practices in underrepresented areas of education. Significantly, social studies teachers will thus be able to build a highly effective set of disciplinary literacy activities that incorporate multimodalities, whether in a remote, hybrid, or traditional setting. The main research question for this study is: What are the experiences of teachers implementing disciplinary reading instruction using multimodality? In turn, the research sub-questions consider: How does the prior academic and/or professional background of participating social studies teachers influence the implementation of disciplinary literacy with multimodalities? To what extent are teachers reflecting on the effectiveness of the implementation of this practice? The instructional design for this study is based on Lave and Wenger’s (1991) situated learning: within the series of mini lessons, learned literacy takes place for students through participation while the whole person acts in the world. Participants were selected through purposive sampling in conjunction with the district’s central office administration. With the district looking to advance disciplinary literacy for all subjects over the next decade, the most collaborative-oriented social studies were selected, specifically either honors- or academic-level history courses. Through the case study, participant educators developed their understanding of how disciplinary literacy is attained for students, particularly through relationship-building and social practice. The overall findings highlight positive experiences from teachers after implementing a series of disciplinary literacy-focused mini-units with multimodalities. Since this study uses entirely qualitative methods, the data invites further analysis using different qualitative, quantitative, or mixed-method approaches to determine the best means to implement disciplinary literacy within history education, both during and after the Covid-19 pandemic

Multimodal Learning Approach to Improve Listening Skills in Sophomore Students at Victor Mideros High School in the Academic Period 2022- 2023

To describe the multimodal learning approach used to improve listening skills in sophomore students at “Victor Mideros” high school in the academic year 2022 – 2023.The present research work arises as an alternative to help improve listening skills in the English language. Its general objective is to describe the multimodal learning approach for the improvement of listening skills in sophomore students at Victor Mideros high school in the academic period 2022- 2023. In order to carry out this research, a mixed approach was applied, both in the quantitative and qualitative aspects of the investigated issue. In addition, two types of research techniques with their respective instruments were used to obtain truthful and quality information: a survey with a questionnaire of 8 questions to the target population and a semi-structured interview made of 7 questions to the teachers. These instruments revealed very clear results that served as a key and fundamental basis for the tabulation and analysis of the data. After this procedure and after socializing the didactic guide proposed in this research study, it was concluded that the multimodal learning approach provides several strategies that help students improve their listening skills in the English language.Licenciatur

Creative pedagogies in early years science: thematic planning and sustained scientific dialogues

This paper draws on the ‘See the Science’ Project funded by the Primary Science Teaching Trust that took place in South West England in 2012-13. It aims to illuminate the process of transforming a curriculum document into a valuable learning experience for children through the use of classroom talk. The project was grounded in concerns that the increased use of thematic, ‘creative’ curricula in England was leading to a loss of scientific learning for children in the early years (five to seven year olds). It presents our findings that in the context of the twelve primary schools with which we worked, the type of curriculum used (e.g. thematic, cross-curricular) had less impact on teaching than we anticipated, whereas the teacher’s immediate responsiveness to children’s ideas and interests and their development of a repertoire of different forms of talk for different purposes in learning science was crucial. A framework emerged to characterise the form of talk we aimed to develop; sustained scientific dialogues, encompassing the essence of sustained, shared thinking in the early years (Siraj-Blatchford et al., 2008) with a science focus and drawing on characteristics of dialogic talk (Mortimer and Scott 2003; Alexander, 2008).Through the development of case studies using qualitative approaches including transcription and analysis of classroom talk we examine how teachers developed their practice and we argue that sustained scientific dialogues play a key role in the development of creative pedagogies. Three cases presented here are representative of the typical project findings: one inexperienced teacher developed talk that was more sustained and two experienced teachers developed talk that was more dialogic, the latter exemplifying how this maintained a strong science focus. The fourth case shows how an early years teacher with a strong dialogic pedagogy expanded his repertoire to include more authoritative episodes focussed on scientific knowledge