Gaze transitions when learning with multimedia

Eye tracking methodology is used to examine the influence of interactive multimedia on the allocation of visual attention and its dynamics during learning. We hypothesized that an interactive simulation promotes more organized switching of attention between different elements of multimedia learning material, e.g., textual description and pictorial visualization. Participants studied a description of an algorithm accompanied either by an interactive simulation, self-paced animation, or static illustration. Using a novel framework for entropy-based comparison of gaze transition matrices, results showed that the interactive simulation elicited more careful visual investigation of the learning material as well as reading of the problem description through to its completion

Exploration of concise redundancy in online multimedia learning

With the rapid growth of multimedia in education, the importance of investigating the effect of redundancy, repeating instructional messages to enhance conceptualization in instructional material design, is becoming more important. Various studies have been conducted recently regarding the effects of different forms of redundancy. A multimedia lesson presenting concurrent on-screen text, still graphics or animations, and narration is a typical setting in redundancy research. Concise redundancy is the revision of the on-screen text into a concise form which is presented to the learners concurrently with visualizations and narration. The purpose of this study was to investigate, while controlling for spatial ability, the effects of concise redundancy on students\u27 retention and confidence when learning with highly complex multimedia materials. In addition, the effects of animation or still graphics along with text redundancy were examined. No significant differences were found between the graphic presentations (animation or series of stills) and text redundancy groups (full, concise, or none) on retention or levels of confidence. When examining the results taking into account high and low spatial abilities, no significant differences were found in terms of different graphic presentation (animation or series of stills) and different text redundancy groups (full, concise, or none). However, in one condition, low spatial ability learners exhibited significantly higher levels of confidence than high spatial ability learners when learning with narrated static graphics and concise redundancy. The current study should provide further guidance for researchers who are interested in examining narrated multimedia lessons containing concise redundancy when comparing static graphics to animated graphics

Comparison of Written and Spoken Instruction to Foster Coordination between Diagram and Equation in Undergraduate Physics Education

Visual–graphical representations are used to visualise information and are therefore key components of learning materials. An important type of convention-based representation in everyday contexts as well as in science, technology, engineering, and math (STEM) disciplines are vector field plots. Based on the cognitive theory of multimedia learning, we aim to optimize an instruction with symbolical-mathematical and visual-graphical representations in undergraduate physics education through spoken instruction combined with dynamic visual cues. For this purpose, we conduct a pre-post study with 38 natural science students who are divided into two groups and instructed via different modalities and with visual cues on the graphical interpretation of vector field plots. Afterward, the students rate their cognitive load. During the computer-based experiment, we record the participants’ eye movements. Our results indicate that students with spoken instruction perform better than students with written instruction. This suggests that the modality effect is also applicable to mathematical-symbolical and convention-based visual-graphical representations. The differences in visual strategies imply that spoken instruction might lead to increased effort in organising and integrating information. The finding of the modality effect with higher performance during spoken instruction could be explained by deeper cognitive processing of the material

Macht der kognitive Stil einen Unterschied?: Folgen verschiedener Visualisierungsarten und Modalitäten für den Lernerfolg in Bezug auf den visuellen und verbalen kognitiven Stil

This doctoral thesis is based on three quantitative studies conducted on 464 participants. The main goal was to investigate the role of visual-verbal cognitive style when learning with dynamic and non-dynamic learning materials. The first study revealed important differences regarding the way in which visualizers and verbalizers observe static picture/text combinations in order to learn from them. That is to say that visualizers concentrated mostly on pictures while verbalizers on texts, exhibiting an active way of learning but mostly within textual (verbalizers) or pictorial (visualizers) areas of stimuli. Contradictory to expectations, visualizers did not directly show any supremacy in dealing with pictures, as they did not identify relevant areas sooner than verbalizers. Indirectly though, the results confirmed that verbalizers are less proficient in decoding pictorial information, as they switched to non-informative parts of it sooner than visualizers. Although the retention test did not show any differences on learning outcomes between both groups, visualizers achieved better results on a comprehension test. The results of the second study confirmed that, when learning with system controlled multimedia environments, spoken explanatory narration brings better results than a written one does. Additionally, an influence of the visual cognitive style on learning with written explanatory text was found. That is, when using a combination of static pictures and written text, higher visual cognitive style comes along with better learning outcome. On the other hand, a combination of higher visual cognitive style, animation and written modality of explanations results in deterioration of learning outcome. The study did not provide any significant results regarding an influence of verbal cognitive style on learning. In the third study the issues of learner/ system control were addressed, when learning with spoken modality of explanatory text. The results revealed that spoken modality design yielded better outcomes when learning in system-paced design rather than self-paced and with animation rather than static pictures. In the group of highly developed visualizers though, the combination of static pictures, self-pacing and spoken narration led to a decline of learning outcomes and to cognitive overload. Again, there were no significant results regarding the verbal cognitive style. The results of the three studies support the assumption of an important role of cognitive style in learning. They indicate a moderating role of visual cognitive style when learning with dynamic and non-dynamic multimedia. This role depends on the design (self-controlled, system-controlled) and modality of explanations (spoken, written) though. Additionally, the differences in gaze patterns between visualizers and verbalizers shed more light on the way in which these two groups of learners retrieve information from multimedia materials. This doctoral research makes a contribution to theoretical research on multimedia learning and cognitive styles, as well as to practical implications on learning materials’ design and efficient education.Diese Dissertation umfasst drei quantitative Studien mit insgesamt 464 Teilnehmern. Dabei war das Hauptziel, die Rolle visueller und verbaler kognitiver Stile beim Lernen mit dynamischen und nicht-dynamischen Lernmaterialien zu untersuchen. Die erste Studie enthüllte deutliche Unterschiede in Bezug auf die Art und Weise, in welcher Visualisierer und Verbalisierer statische Bild/Text-Kombinationen analysieren, um von diesen zu lernen. Konkret konzentrieren sich Visualisierer primär auf die Bilder, wohingegen Verbalisierer eher auf die Texte fokussieren. Lernende beider Typen weisen jedoch dabei eine aktive Art des Lernens innerhalb ihrer präferierten Stimuli auf. Entgegen der Erwartungen zeigten Visualisierer jedoch keine Überlegenheit im Umgang mit Bildern im Vergleich zu Verbalisierern, da sie keine schnellere und effektivere Identifikation relevanter Bereiche innerhalb der Bilder erzielten. Indirekt bestätigen jedoch die Ergebnisse, dass Verbalisierer weniger Kompetenzen im Umgang mit Bildinformationen aufwiesen, da sie schneller auf die nicht-informativen Bereiche der Bilder wechselten als die Visualisierer. Obwohl der Wissenstest in Bezug auf den Lernerfolg keine Unterschiede zwischen den beiden Gruppen zeigte, erreichten die Visualisierer bessere Resultate im Verständnistest. Die Ergebnisse der zweiten Studie bestätigten, dass beim Lernen mit einer multimedialen, nichtinteraktiven Lernumgebung eine auditive Erklärung zu besseren Lernergebnissen führt als ein schriftlich dargebotener Text. Zudem konnte ein Einfluss des visuellen kognitiven Stils auf das Lernen mit Texten aufgezeigt werden. Dieser stellt sich durch die Tatsache dar, dass Personen mit einem ausgeprägteren visuellen kognitiven Stil einen besseren Lernerfolg erzielen, wenn eine Kombination aus statischen Bildern und geschriebenem Text verwendet wird. Andererseits kann jedoch eine Kombination aus einem ausgeprägten visuellen kognitiven Stil, Animationen und auditiven Erklärungen wiederum zu einer Verschlechterung des Lernerfolgs führen. Die Studie konnte jedoch keine signifikanten Ergebnisse in Bezug auf den Einfluss verbaler kognitiver Stile auf das Lernen nachweisen. In der dritten Studie ging es um den Einfluss interaktiver Kontrollelemente (selbstgesteuert vs. systemgesteuert) beim Lernen mit auditiven Erklärungen oder schriftlichen Texten. Die Ergebnisse zeigen, dass auditive Erklärungen zu besseren Lernerfolgen führt, wenn systemgesteuert statt selbstgesteuert (also nicht-interaktiv) gelernt wird. Zudem zeigte in diesem Fall die Verwendung von Animationen bessere Resultate als die von statischen Bildern. In der Gruppe der „Visualisierer“ mit ausgeprägtem visuellen kognitiven Stil führte die Kombination von statischen Bildern, selbstgesteuertem Design und auditiv dargebotenen Informationen hingegen zu einer kognitiven Überlastung und einem Rückgang der Lernerfolge. Auch in dieser Studie gab es keine signifikanten Ergebnisse in Bezug auf den verbalen kognitiven Stil. Die Ergebnisse der drei Studien unterstützen die Annahme, dass der kognitive Stil eine wichtige Rolle beim Lernen spielt. Insbesondere der visuelle kognitive Stil scheint einen moderierenden Einfluss beim Lernen mit dynamischen und nicht-dynamischen Medien auszuüben. Dieser Einfluss hängt dabei vom Design (selbstgesteuert vs. systemgesteuert) und der Modalität der Erklärungen (auditiv vs. textuell) ab. Zudem konnte durch die Analyse der Blickmuster zwischen Visualisierern und Verbalisierern ein erweitertes Verständnis darüber gewonnen werden, wie die beiden Gruppen unterschiedlich mit Informationen aus multimedialen Materialien umgehen. Diese Doktorarbeit leistet damit einen Beitrag zur theoretischen Forschung im Bereich des multimedialen Lernens und kognitiver Stile sowie zu praktischen Konsequenzen des Designs von Lernmaterialien zu effektiver Bildung

Multimodal reading and second language learning

Most of the texts that second language learners engage with include both text (written and/or spoken) and images. The use of images accompanying texts is believed to support reading comprehension and facilitate learning. Despite their widespread use, very little is known about how the presentation of multiple input sources affects the attentional demands and the underlying cognitive processes involved. This paper provides a review of research on multimodal reading, with a focus on attentional processing. It first introduces the relevant theoretical frameworks and empirical evidence provided in support of the use of pictures in reading. It then reviews studies that have looked at the processing of text and pictures in first and second language contexts. Based on this review, main gaps in research and future research directions are identified. The discussion provided in this paper aims at advancing research on multimodal reading in a second language. Achieving a better understanding of the underlying cognitive processes in multimodal reading is crucial to inform pedagogical practices and to develop theoretical accounts of second language multimodal reading

An Eye Tracking Study to Investigate the Influence of Language and Text Direction on Multimedia

This study investigated how native language orientation influences spatial bias, first visual fixation on screen, first visual fixation on pictures, learning outcomes, and mental effort of learners. Previous studies supported the effect of native language writing or reading direction on spatial bias, examining written text and images created by the participants (Barrett et al., 2002; Boroditsky, 2001; Chatterjee, Southwood & Basiko, 1999; Spalek & Hammad, 2005). However, no study investigated writing direction in multimedia presentations using eye tracking. This study addresses this gap. A total of 84 participants completed the study forming four groups. The first group (NativeLeft_InstrEng) consisted of individuals whose native language is written from left to right and who have never experienced a right to left language. They received the material in English. The second group (NativeRight_InstrAra), whose native language is written from right to left, received the material in Arabic. The third group (NativeLeft_LrnRight_InstrEng) consists of individuals whose native language is written from left to right and who are learning or have learned a language written from right to left. They received the material in English. The fourth group (NativeRight_InstrEng), whose native language is written from right to left, received the material in English. Participants were asked to complete a survey that consisted of eight sections: demographic questions, self-estimate prior knowledge test, the instructional unit, mental effort rating, sentence forming questions, recalling questions, sequence question and finally, post-test questions. Eye tracking was used to detect first fixation on screen and pictures, and results were compared with participants’ written responses. Eye movements can be considered the blueprint for how students process the visual information (Underwood & Radach, 1998). Significant results for learning and spatial bias confirmed that spatial bias is associated with native language orientation such that the left-oriented learners were more likely to demonstrate left bias on the screen, while participants who were right-oriented demonstrated right bias. However, exposure to other languages, culture, or beliefs; or living for some time in a country which uses a language with a different orientation can influence learner’s spatial bias, as seen with group NativeRight_InstrEng. Finally, differences in visual fixations on screen and pictures were not significant perhaps due to the simplicity of pictures used in this study

Chapter 03: Multimedia Learning Theory and Instructional Message Design

Multimedia learning theory describes how the designers of instructional messages, systems and learning environments can optimize learning. The principles and heuristics of multimedia learning theory have application in traditional and online environments, with young and adult learners, in K-12, higher education, military, corporate, government, and informal learning environments. This diversity of application is based on the foundational premise that all learners can independently process auditory and visual information, have limited working memory resources, and require cognitive resources to process new information and to learn. This chapter describes the basic tenets of multimedia learning theory, best practices that can improve our message design and communication, and exciting future directions that we can take new research

Effects of Illustrations on Retention and Visual Attention Using Authentic Textbooks

Problem The purpose of this exploratory study was to validate a model of multimedia learning, incorporating learner-specific characteristics such as intelligence, memory, and measures of visual attention. Method The sample consisted of 62 Andrews University students (26 males, 36 females, and mean age 21.7). Data were gathered by means of standardized testing (RPM, WAIS- III) and eye-tracking. MLR was used to determine significant visual attention predictors for retention and SEM was used to test a hypothesized model of multimedia learning. Results Multiple Linear Regression was significant (p.05; GFI = .923; CFI = .986; RMSEA = .043; PGFI = .451) proving the hypothesized model’s excellent fit to the data. Conclusions This exploratory study indicates that there are different learning strategies in a college population that are not related to characteristics such as intelligence or working memory. These strategies are learned or acquired and have thus clear implications for practice

Chapter 3: Multimedia Learning Theory and Instructional Message Design

Multimedia learning theory describes how the designers of instructional messages, systems, and learning environments can optimize learning. The principles and heuristics of multimedia learning theory have application in traditional and online environments, with young and adult learners, in K-12, higher education, military, corporate, government, and informal learning environments. This diversity of application is based on the foundational premise that all learners can independently process auditory and visual information, have limited working memory resources, and require cognitive resources to process new information and to learn. This chapter describes the basic tenets of multimedia learning theory, best practices that can improve our message design and communication, and exciting future directions that we can take new research

Integration Scaffolding in Hypermedia Learning

This dissertation project used 80 undergraduate students to examine the effectiveness of three forms of facilitation in hypermedia learning with text and diagrams about the human circulatory system: 1) signaling key terms, 2) prompted referencing of diagrammatic representations, and 3) integration scaffolding which provided facilitation in locating corresponding components within diagrams. These three experimental manipulations were compared to a control condition in which learners used the same hypermedia learning environment, without any facilitative feature in coordinating between text and diagrams. Two measures captured differences in learning: 1) a multiple choice pretest and posttest of declarative and conceptual knowledge and 2) a diagram interpretation task requiring learners to use diagrams to explain their understanding of the circulatory system. Eye-tracking measures and concurrent think-aloud protocols were collected during the 20-minute learning sessions to provide process measures of students\u27 learning and a self-report cognitive load measure was administered immediately after the learning session. Results indicated that the integration scaffolding condition led to higher posttest scores on the multiple choice measure, but no significant differences were detected for the diagram interpretation task. Eye-tracking results demonstrated that the integration scaffolding condition had a higher number of and a higher total duration of fixations on relevant areas within the diagrams. The relevant areas represent diagrammatic representations corresponding to the textual referents within the accompanying text. Additionally, these learners spent a significantly larger proportion of their time inspecting diagrams looking at the relevant areas of the diagrams and a significantly larger proportion of these learners\u27 fixations were on relevant areas. Analyses of learners\u27 self-regulated learning processes, based on concurrent think-aloud protocols, indicated that the integration scaffolding condition also generated more correct summarizations than the remaining groups. The self-report cognitive load measure failed to reveal any differences among the learning conditions. Taken together, these results provide support for models of text-picture integration (Mayer, 2005; Schnotz, 2005) and, to some extent, Cognitive Load Theory. Further, the experiment suggests that directing learners\u27 attention to corresponding elements within text and diagrams can be an effective technique for facilitating the process of text-picture integration