Integrative Processing of Verbal and Graphical Information During Re-reading Predicts Learning from Illustrated Text: An Eye-Movement Study

Printed or digital textbooks contain texts accompanied by various kinds of visualisation. Successful comprehension of these materials requires integrating verbal and graphical information. This study investigates the time course of processing an illustrated text through eye-tracking methodology in the school context. The aims were to identify patterns of first- and second-pass reading and to examine whether the integrative processing of text and picture during the less automatic and more purposeful second-pass reading predicts learning, after controlling for reading comprehension, prior knowledge, and self-concept. Forty-three 7th graders read an illustrated science text while their eye-movements were recorded. A cluster analysis revealed two processing patterns during the first-pass reading, which differed for the time spent on the main concepts in the text and picture. During re-reading, two patterns of stronger and weaker integrative processing emerged. Integration of verbal and graphical information was revealed by the frequency of second-pass transitions from text to picture and from picture to text, and the duration of picture re-inspecting while re-reading text information (look-from text to picture) and re-reading text information while re-inspecting the visualised information (look-from picture to text). A series of hierarchical regression analyses indicated that only the patterns of integrative processing during the second-pass reading uniquely predict verbal and graphical recalls, and the transfer of knowledge. The study provides evidence that the delayed processing which integrates text and graphics contributes to text retention and the application of newly learned knowledge, over and above individual characteristics. The educational significance is outlined

The Role of Working Memory in Integrative Reading of Text and Picture: An Eye Tracking Study

Despite beliefs about the effectiveness visualization has on learning, researchers have found that adding pictures to text does not always lead to better learning outcomes. Although extensive empirical research has examined the relevant factors that can enhance the effects of images added to text, individual differences in underlying cognitive capacities are relatively unexplored in the literature. As such, the purpose of this dissertation study is to examine the role of executive control in the integrative reading processes of elementary students as well as the learning outcomes of students who read illustrated scientific texts through the observation of their eye movement patterns. To achieve this goal, this study examines three research areas: the first investigates the unique, direct contributions of fourth and fifth grade students’ integrative reading of text and pictures as evidenced by eye movement patterns and the link to their learning outcomes, while controlling for contributions of working memory capacity. The second examines the unique, direct contributions of fourth and fifth grade students’ working memory capacity to their comprehension of illustrated science texts. The final study examines unique, direct contributions of fourth and fifth grade students’ working memory capacity to their reading processes, including text processing, picture processing, and the integrative reading of text and pictures. This dissertation is expected to contribute to the extant theoretical and empirical literature. First, the results supplement current theories of multimedia learning by specifying the role of attention shifting and inhibitory control. In the current multimedia learning theories, the role of executive control is not clearly determined. Findings in this study further examine the current cognitive model of text-picture integration by recognizing the significant role of executive control capacity in learners. Second, the results will extend current knowledge about elementary school students’ cognitive processes when reading illustrated science texts through the use of an eye tracking technique. Finally, for the upper elementary students, who advance from the learning to read to reading to learn stage, identifying relevant cognitive factors in learning with visualizations will provide foundations for creating and delivering adequate interventions

Multimodal reading and second language learning

Most of the texts that second language learners engage with include both text (written and/or spoken) and images. The use of images accompanying texts is believed to support reading comprehension and facilitate learning. Despite their widespread use, very little is known about how the presentation of multiple input sources affects the attentional demands and the underlying cognitive processes involved. This paper provides a review of research on multimodal reading, with a focus on attentional processing. It first introduces the relevant theoretical frameworks and empirical evidence provided in support of the use of pictures in reading. It then reviews studies that have looked at the processing of text and pictures in first and second language contexts. Based on this review, main gaps in research and future research directions are identified. The discussion provided in this paper aims at advancing research on multimodal reading in a second language. Achieving a better understanding of the underlying cognitive processes in multimodal reading is crucial to inform pedagogical practices and to develop theoretical accounts of second language multimodal reading

Deaf and Hard of Hearing Readers and Science Comics: A Mixed Methods Investigation on Process

Deaf and hard of hearing (DHH) students bring diverse language and literacy backgrounds to the task of academic reading, which becomes increasingly complex and abstract in the upper grades. Teachers often differentiate their instruction by providing multimedia resources, of which students interact with verbal and pictorial information. A growing body of research supports multimedia learning; however, most of the studies have focused exclusively on learning outcomes, leaving teachers in the dark about the cognitive processes underlying these effects. This mixed methods study addresses this gap by using a nonfiction comic to investigate the reading processes of DHH 7th -12th grade students. Eye tracking and cued retrospective protocol were employed in a concurrent nested design to answer the question, how do DHH students read and learn from multimedia science texts? This study was guided by the cognitive theory of multimedia which states that reading comprehension is better supported when learning from words and pictures rather than words alone, especially when readers cognitively integrate the two representations to form a coherent mental model. Temporal and sequential eye tracking results revealed that readers’ transitions between related words and pictures were a statistically significant variable in explaining factual knowledge learning outcomes. These strategic shifts in attention were further explained by readers’ retrospective verbal reports of their thinking. Students’ descriptions of their vii reading processes were interpreted into the following themes: repairing, connecting representations, passive transitions, and connecting to self. The integration of quantitative and qualitative methods at the interpretation stage revealed that although the theme of repairing was equally distributed across all student reports, the theme of connecting representations was largely present in the reports from students who made high counts of integrative transitions. The major findings of this study align with the cognitive theory of multimedia, that students’ learning outcomes were significantly predicted by the deliberate strategies to cognitively integrate words and pictures to form and maintain a coherent mental model. The discussion includes ways in which teachers can capitalize on explicit modeling of these behaviors and employ students’ “think alouds” to better understand and support the development of effective multimedia reading processes

Reading Alphabetic and Nonalphabetic Writing Systems: A Case Study of Bilingual Teachers\u27 Reading Processes through Eye Movement Miscue Analysis

This case study investigates the reading processes of two bilingual teachers who speak English as a second language and use different first languages—Mandarin Chinese and Korean. The two participants read researcher-selected digital texts in English and in their respective first language, retold the texts, and answered comprehension questions about the texts. Their reading aloud and eye movements were recorded for miscue and eye movement analysis. Using Eye Movement Miscue Analysis, the findings showcase the distinctive characteristics of their first-language and second-language reading processes. The cross-linguistic comparison between bilingual reading processes further shows the bilingual participants\u27 similarities and differences in terms of the use of language systems, eye movements, language variations, and image use. This study supports the understanding of non-Roman alphabetical language speakers\u27 reading process, adds to our understanding of the bilingual reading process, and provides teaching and research implications for bilingual teachers and educators

Do easy-to-read adaptations really facilitate sentence processing for adults with a lower level of education? An experimental eye-tracking study

The Easy-to-Read guidelines recommend visual support and lexical simplification to facilitate text processing, but few studies have empirically verified the efficacy of these guidelines. This study examined the influence of these recommendations on sentence processing by examining eye movements at the text- and word-level in adult readers. We tested 30 non-university adults (low education level) and 30 university adults (high education level). The experimental task consisted of 60 sentences. Half were accompanied by an image and half were not, and half contained a low-frequency word and half a high-frequency word. Results showed that visual support and lexical simplification facilitated processing in both groups of adults, and non-university adults were significantly slower than university adults at sentence processing. However, lexical simplification resulted in faster processing in the non-university adults’ group. Conclusions focus on the mechanisms in which both adaptations benefit readers, and practical implications for reading comprehension

Eye(s) see what you do: The Role of Social Mechanisms in the Effectiveness of Eye Movement Modeling Examples as an Instructional Tool for Multimedia Learning

Learning with multimedia material (e.g. text and pictures) often leads to better learning outcomes than learning with text alone. However, it is challenging for learners to apply effective information processing strategies to benefit from multimedia materials for their learning success. Not all learners are capable of doing this without instructional support. Eye Movement Modeling Examples (EMME) support learners in internalising and applying effective multimedia processing strategies such as selection, organisation and integration. EMME consist of videos with recorded eye movements of an (expert) model illustrating the dynamic change of the eye movements of the model during the processing of the task material. Previous studies have shown a positive effect of EMME on the use of multimedia processing strategies as well as on learning outcomes. So far, however, the question of possible mechanisms underlying the effectiveness of EMME for multimedia learning has remained open. The present work therefore used two different approaches in three experiments to investigate the question of whether social mechanisms and factors play a role in the effectiveness of EMME. The first approach examined whether social mechanisms contribute to the efficacy of EMME beyond the effects of perceptual and (meta-) cognitive mechanisms. However, the results were not conclusive on this question. In a second approach, it was investigated whether social factors such as ( alleged) model-observer similarity and social comparison orientation can influence the effectiveness of EMME. Although the results for this approach provided first empirical evidence that social factors can influence the effectiveness of EMME, the overall picture was rather inconsistent. On the basis of the results of the three experiments it is therefore not possible to draw final conclusions about the role of social mechanisms and factors for the effectiveness of EMME. In summary, EMME can be an effective instructional support for multimedia learning. In addition, however, it was found that there are factors that can moderate the effectiveness of EMME.Lernen mit multimedialem Material (z. B. Text und Bild) führt häufig zu besseren Lernergebnissen als das Lernen mit Text allein. Es stellt Lernende jedoch vor die Herausforderung, geeignete Informationsverarbeitungsstrategien anzuwenden, um von multimedialen Materialien für ihren Lernerfolg zu profitieren. Hierzu sind nicht alle Lernenden ohne instruktionale Unterstützung in der Lage. Eye Movement Modeling Examples (EMME) unterstützen Lernende durch die Modellierung erfolgreicher multimedialer Verarbeitungsstrategien wie Selektion, Organisation und Integration dabei, diese zu internalisieren und anzuwenden. EMME bestehen aus Videos mit aufgezeichneten Blickbewegungen eines (Experten-) Modells, die die dynamische Veränderung des Blickverhaltens des Modells bei der Verarbeitung des Aufgabematerials zeigen. Bisherige Studien zeigen weitestgehend einen positiven Effekt von EMME sowohl auf die Nutzung multimedialer Verarbeitungsstrategien als auch auf die Lernergebnisse. Bislang blieb jedoch die Frage nach möglichen Mechanismen, die der Wirksamkeit von EMME für multimediales Lernen zugrunde liegen, offen. Die vorliegende Arbeit untersuchte daher mithilfe von zwei verschiedenen Ansätzen in drei Experimenten die Frage, ob soziale Mechanismen und Faktoren eine Rolle für die Wirksamkeit von EMME spielen. In einem ersten Ansatz wurde untersucht, ob soziale Mechanismen über die Wirkung von perzeptuellen und (meta-) kognitiven Mechanismen hinaus zur Wirksamkeit von EMME beitragen. Die Ergebnisse waren in Hinblick auf diese Frage jedoch nicht eindeutig. In einem zweiten Ansatz wurde untersucht, ob soziale Faktoren wie (vermeintliche) Modell-Beobachterähnlichkeit und soziale Vergleichsorientierung die Wirksamkeit von EMME beeinflussen können. Auch wenn die Ergebnisse für diesen Ansatz erste empirische Hinweise lieferten, dass soziale Faktoren die Effektivität von EMME beeinflussen können, zeigte sich insgesamt ein eher inkonsistentes Bild. Auf Basis der Ergebnisse der drei Experimente ist es daher nicht möglich, endgültige Schlussfolgerungen über die Rolle sozialer Mechanismen und Faktoren für die Wirksamkeit von EMME zu ziehen. Zusammenfassend zeigte sich, dass EMME eine wirksame instruktionale Unterstützung für multimediales Lernen sein können. Darüber hinaus zeigte sich jedoch auch, dass es Faktoren gibt, die die Effektivität von EMME moderieren können

Young L2 learners' online processing of information in a graded reader during reading-only and reading-while-listening conditions: A study of eye-movements

Combining reading with auditory input has been shown to be an effective way of supporting reading fluency and reading comprehension in a second language. Previous research has also shown that reading comprehension can be further supported by pictorial information. However, the studies conducted so far have mainly included adults or adolescents and have been based on post-reading tests that, although informative, do not contribute to our understanding of how learners' processing of the several sources of input in multimodal texts changes with the presence of auditory input and the effect that potential differences could have on comprehension. The present study used eyetracking to examine how young learners process the pictorial and textual information in a graded reader under reading only and reading-while-listening conditions. Results showed that readers spent more time processing the text in the reading only condition, while more time was spent processing the images in the reading-while-listening mode. Nevertheless, comprehension scores were similar for the readers in the two conditions. Additionally, our results suggested a significant (negative) relationship between the amount of time learners spent processing the text and comprehension scores in both modes