Development of Dual-Modal Presentations of Textual Information

Delivering information across sensory modalities is often supported by the independent nature of multi-modal information processing, which assumes that there is no interference between tasks and thus no degradation in performance. However, research in cognitive psychology shows that visual and auditory perceptual processing is closely linked. Problems related to memory and cognitive workload are found in current applications with voice-based interface. For instance, mental integration of disparate information tends to cause a heavy cognitive memory load, and switching attention between modalities may be slow and have a high cost. This study focuses on how to design a visual-auditory information presentation to: (1) minimize the interference in information processing between the visual and auditory channels; and (2) improve the effectiveness of mental integration of information from different modalities. Baddeley\u27s working memory model suggests that imagery spatial information and verbal information can be concurrently held in different subsystems within human working memory. Based on this model and research on human attention, this study proposes a method to convert textual information into a “graphics + voice” representation and hypothesizes that this dual-modal presentation will result in superior comprehension performance and higher satisfaction as compared to pure textual display. Simple T-tests will be used to test the hypothesis. Results of this study will benefit interface design of generic computer systems by alleviating information overload in the visual display. Findings may also help to address usability problems associated with hand-held devices

Literacy practices in the learning careers of childcare students

This paper draws from the Literacies for Learning in Further Education research project, funded through the Teaching and Learning Research Programme. Drawing on the empirical study of literacy practices in eight Childcare courses in Scotland and England, we seek to demonstrate that, integral to the learning careers of students are literacy careers through which their learning is mediated. In the process, by drawing upon the lens of literacy, we also challenge some of the common sense understandings of learning in childcare. In particular we suggest that the literacy practices of lower level courses can be more diverse than those of higher level courses, producing confusing literacy careers for the students involved. We also point to the complexity of the literacy careers in childcare, given that students are required to mediate different aspects of their experience through literacy. In particular there are the mediations made possible by the use of information technology and those entailed in relating work placements to classroom practice. We argue that students on vocational courses have complex literacy careers and that a literacies approach to learning helps to reveal this complexity

Narration and Speech and Thought Presentation in Comics

The purpose of this study was to test the application of two linguistic models of narration and one linguistic model of speech and thought presentation on comic texts: Fowler's (1986) internal and external narration types, Simpson's (1993) narrative categories from his 'modal grammar of point of view' and Leech and Short's (1981) speech and thought presentation scales. These three linguistic models of narration and speech and thought presentation, originally designed and used for the analysis of prose texts, were applied to comics, a multimodal medium that tells stories through a combination of both words and images. Through examples from comics, I demonstrate in this thesis that Fowler's (1986) basic distinction between internal and external narration types and Simpson's (1993) narrative categories (categories A, B(N) and B(R) narration) can be identified in both visual and textual forms in the pictures and the words of comics. I also demonstrate the potential application of Leech and Short's (1981) speech and thought presentation scales on comics by identifying instances of the scales' categories (NPV/NPT, NPSA/NPTA, DS/DT and FDS/FDT) from comics, but not all of the speech and thought presentation categories existed in my comic data (there was no evidence of IS/IT and the ategorisation of FIS/FIT was debatable). In addition, I identified other types of discourse that occurred in comics which were not accounted for by Leech and Short's (1981) speech and thought presentation categories: internally and externally-located DS and DT (DS and DT that are presented within (internally) or outside of (externally) the scenes that they originate from), narratorinfluenced forms of DS and DT (where narrator interference seems to occur in DS and DT), visual presentations of speech and thought (where speech and thought are represented by pictorial or symbolic content in balloons) and non-verbal balloons (where no speech or thought is being presented, but states of mind and emphasized pauses or silence are represented by punctuation marks and other symbols in speech balloons)

Taking the Next Step with Generative Artificial Intelligence: The Transformative Role of Multimodal Large Language Models in Science Education

The integration of Artificial Intelligence (AI), particularly Large Language Model (LLM)-based systems, in education has shown promise in enhancing teaching and learning experiences. However, the advent of Multimodal Large Language Models (MLLMs) like GPT-4 with vision (GPT-4V), capable of processing multimodal data including text, sound, and visual inputs, opens a new era of enriched, personalized, and interactive learning landscapes in education. Grounded in theory of multimedia learning, this paper explores the transformative role of MLLMs in central aspects of science education by presenting exemplary innovative learning scenarios. Possible applications for MLLMs could range from content creation to tailored support for learning, fostering competencies in scientific practices, and providing assessment and feedback. These scenarios are not limited to text-based and uni-modal formats but can be multimodal, increasing thus personalization, accessibility, and potential learning effectiveness. Besides many opportunities, challenges such as data protection and ethical considerations become more salient, calling for robust frameworks to ensure responsible integration. This paper underscores the necessity for a balanced approach in implementing MLLMs, where the technology complements rather than supplants the educator's role, ensuring thus an effective and ethical use of AI in science education. It calls for further research to explore the nuanced implications of MLLMs on the evolving role of educators and to extend the discourse beyond science education to other disciplines. Through the exploration of potentials, challenges, and future implications, we aim to contribute to a preliminary understanding of the transformative trajectory of MLLMs in science education and beyond

The SAE Architecture Analysis & Design Language (AADL) A Standard for Engineering Performance Critical Systems

International audienceThe Society of Automotive Engineers (SAE) Architecture Analysis & Design Language, AS5506, provides a means for the formal specification of the hardware and software architecture of embedded computer systems and system of systems. It was designed to support a full Model Based Development lifecycle including system specification, analysis, system tuning, integration, and upgrade over the lifecycle. It was designed to support the integration of multiple forms of analyses and to be extensible in a standard way for additional analysis approaches. A system can be automatically integrated from AADL models when fully specified and when source code is provided for the software components. Analysis of large complex systems has been demonstrated in the avionics domain

MOG 2007:Workshop on Multimodal Output Generation: CTIT Proceedings

This volume brings together presents a wide variety of work offering different perspectives on multimodal generation. Two different strands of work can be distinguished: half of the gathered papers present current work on embodied conversational agents (ECA’s), while the other half presents current work on multimedia applications. Two general research questions are shared by all: what output modalities are most suitable in which situation, and how should different output modalities be combined

When humans form media and media form humans: an experimental study examining the effects different digital media have on the learning outcomes of students who have different learning styles

A set of computer-based experiments are reported that investigate the understanding achieved by learners when studying a complex domain (statistics) in a real E-learning environment using three different media combinations – Text only, Text and Diagrams and Spoken Text and Diagrams, and the results agree with earlier work carried out on more limited domains. The work is then extended to examine how student interaction and student learning styles affect the learning outcomes. Different responses to the media combinations are observed and significant differences occur between learners classified as Sensing and Reflective learners. The experiment also identified some important differences in performance with the different media combinations by students registered as Dyslexic. The experiment was therefore repeated with a much larger sample of Dyslexic learners and the earlier effects were found to be significant. The results were surprising and may provide useful guidance for the design of material for Dyslexic students