International audienceWe present a method devised to automatically analyze pupils' verbalizations during reading and to reveal some of the strategies they use. An experiment with 44 primary school pupils (3rd and 5th grade) reading a narrative text and verbalizing what they understood at predefined breaks showed, firstly, that machine results are correlated with experts'; secondly, that the recall of recent sentences is not uniform across verbalizations and subject to a grade effect; thirdly, that there is a grade effect in the recall of distal causal sentences

Bianco, Maryse

Dascalu, Mihai

Dessus, Philippe

Nardy, Aurélie

Toffa, Françoise

Trausan-Matu, Stefan

Hal - Université Grenoble Alpes

Automated analysis of pupils’ self-explanations of anarrative textPhilippe Dessus, Maryse Bianco, Aure´lie Nardy, Franc¸oise Toffa, MihaiDascalu, Stefan Trausan-MatuTo cite this version:Philippe Dessus, Maryse Bianco, Aure´lie Nardy, Franc¸oise Toffa, Mihai Dascalu, et al.. Au-tomated analysis of pupils’ self-explanations of a narrative text. de Vries, E.;K. Scheiter;.Meeting of the EARLI SIG 2 ”Comprehension of Text and Graphics”, Aug 2012, France. LSE,Pierre-Mende`s-France university, pp.52-54, 2012. <hal-00906940>HAL Id: hal-00906940https://hal.archives-ouvertes.fr/hal-00906940Submitted on 20 Nov 2013HAL is a multi-disciplinary open accessarchive for the deposit and dissemination of sci-entific research documents, whether they are pub-lished or not. The documents may come fromteaching and research institutions in France orabroad, or from public or private research centers.L’archive ouverte pluridisciplinaire HAL, estdestine´e au de´poˆt et a` la diffusion de documentsscientifiques de niveau recherche, publie´s ou non,e´manant des e´tablissements d’enseignement et derecherche franc¸ais ou e´trangers, des laboratoirespublics ou prive´s.Meeting of the EARLI SIG 2 “Comprehension of Text and Graphics”. Grenoble : Pierre-Mendès-France university, 2012, 28-31 August  Automated Analysis of Pupils’ Self-Explanations  of a Narrative Text Philippe Dessus*, Maryse Bianco*, Aurélie Nardy*, Françoise Toffa*, Mihai Dascalu**,  & Stefan Trausan-Matu** *LSE & IUFM, Grenoble-1 and Grenoble-2 Universities, France **“POLITEHNICA” University of Bucharest, Romania {Philippe.Dessus, Maryse.Bianco, Aurelie.Nardy}@upmf-grenoble.fr, Francoise.Toffa@gmail.com, {Mihai.Dascalu, trausan}@cs.pub.ro  Abstract. We present a method devised to automatically analyze pupils’ verbalizations during reading and to reveal some of the strategies they use. An experiment with 44 primary school pupils (3rd and 5th grade) reading a narrative text and verbalizing what they understood at predefined breaks showed, firstly, that machine results are correlated with experts’; secondly, that the recall of recent sentences is not uniform across verbalizations and subject to a grade effect; thirdly, that there is a grade effect in the recall of distal causal sentences.  Keywords: Self-Explanations; Reading Strategies; Latent Semantic Analysis; Comprehension  Introduction The use of strategies during reading is widely recognized as a crucial determinant of reading comprehension. Second degree and high school pupils who are good comprehenders are mostly strate-gic readers (Graesser, 2007). These strategies can be elicited through self-explanations and have been categorized by McNamara (2004) as follows: comprehension monitoring, paraphrasing, elaboration, prediction, and bridging. One important skill these strategies exploit is to be able to establish semantic and causal relationships between the read sentences (Wolfe, Magliano, & Larsen, 2005). Based on these findings, McNamara et al. (2007) developed iSTART, a cognitive tutor that automati-cally categorizes self-explanations, partly using Latent Semantic Analysis (Landauer & Dumais, 1997). Any thorough analysis of self-explanation reports is a very demanding and subjectivity-oriented activity, and the use of systems like iSTART to detect pupils’ reading strategies is more than challenging. Since a cognitive tutor guides the reader through pre-defined steps alternating between reading and verbalizations, we consider its use as a scenarization of the reading and comprehension process. This computer-based scenarization is made possible through the wide range of reading strate-gies and the feedback possibilities (Vitale & Romance, 2007). At least two kinds of reading scenarios are considered: pedagogical scenarios, whose aim is to orchestrate the reading activities, and assess-ment scenarios, targeted at supporting the way learners’ reading is assessed. The aim of this paper is to focus on the second kind of scenarios and to present preliminary results for the basis of a cognitive reading tutor. We introduce and test a way to automatically analyze pupils’ self-explanations of what they have read, through an LSA-based analysis of word usage. Since paraphrases are one of the most frequently used strategies (McNamara, 2004), we focus our study on how two main kinds of sentences are paraphrased: focal (the latest sentence before a verbalization) and causal sentences (identified by a causal analysis of the text), because it is worth distinguishing the mere paraphrase of the latest read sentence and more elaborated paraphrases, involving a deeper comprehension of the read text. This paper investigates novel research paths. Firstly, it focuses on elementary school pupils, a category of students seldom investigated. Secondly, an LSA space is chosen that fits best the pupil’s knowledge by using a corpus composed by Denhière et al. (2007), comprising 3.2 million words and validated by a test involving association norms. Thirdly, we propose a dynamic view of the self-explanation process, in analyzing the verbalizations at different break points throughout the story. Meeting of the EARLI SIG 2 “Comprehension of Text and Graphics”. Grenoble : Pierre-Mendès-France university, 2012, 28-31 August  Our research questions are firstly to compare human expert categorization of paraphrases to the semantic similarity between text sentences and self-explanations, obtained by means of LSA. Secondly, we expect a “recency effect”, stating that the information children self-explain most often pertains to very close sentences to the verbalization break. Thirdly, we will investigate the way pupils account for causal relations (either local or distal) in retelling causally-related text sentences. Method Participants 22 third and 22 fifth grade pupils from the same school and from a middle socio-economic background participated in our experiment. Materials and Procedure One narrative text was read and self-explained by the pupils: Matilda (453 words, 6 self-explanation breaks). The text was chosen to be within the reading level of participants, so that differences in verbalizations would indicate differences in reading strategies instead of comprehension difficulties. In order to perform a fine-grained analysis, the initial text was split in 45 segments (of about 1 sentence each). We performed a causal analysis so that both local (when the causal antecedent is close to the reference sentence) and distal antecedents (when the causal antecedent is somewhat farther, out of the reader’s working memory) of sentences were determined as in Millis et al. (2006). We finally performed a propositional analysis of the text, which allowed us to extract macro-propositions and to support the coding of what was remembered by the participants. Participants individually read the text out loud and stopped at predetermined breaks to self-explain the text segment just read, the whole activity being recorded. The task was explained to pupils as follows: “During your reading you will stop at each icon to tell out loud what you have understood, just at this time”. Their verbalizations were then transcribed and each self-explanation was semantically compared with LSA—all the text sentences before the self-explanation breaks. Two of the co-authors analyzed pupils’ verbalizations proposition by proposition and categorized them according to McNamara’s (2004) coding scheme. Disagreements were also discussed and resolved by consensus. Results First of all, we computed accuracy measures in order to compare human vs. LSA values of sentence relatedness and to check the validity of the computer-based measures. Pearson correlations between the number of paraphrases per verbalization (Vn) detected by the two raters and LSA similarities between each verbalization and the previous sentences were as follows: V1: r=.48; V2: r=.58; V3: r=.74; V4: r=.29; V5: r=.57; V6: r=.61, which shows that human judgments of paraphrases expressed by children on each paragraph are moderately to strongly related to LSA measures of similarities. We then investigated the extent to which each self-explanation was related to the last read sentence (focal). Figure 1a presents the cosine similarities, processed by LSA, between each verbalization and the focal sentence, by grade. We observe that the recency effect varies across verbalization plots, indicating that this effect is dependent of the content conveyed by the last sentences. Moreover, the focal sentence, in general, does not have a higher similarity with the related verbalization than the average of other previous sentences, except for V4: t(43)=7.5, p < .0005. Two-way ANOVAs showed a significant difference between grades for V6, F(1, 42)=7.01; p < .05 and a tendency for V2, F(1, 42)=3.22, p < .09. Although grade 3 pupils tended to recall the last sentence at these points more frequently, the semantic content of the last sentence seems to be the main determinant of focal recall. The third hypothesis predicted that the semantic content of local and distal sentences, as determined by the causal analysis, is more often verbalized than the rest of the previous text and the focal sentence. Moreover, the local-centered causal sentences were expected to be better recalled than the Meeting of the EARLI SIG 2 “Comprehension of Text and Graphics”. Grenoble : Pierre-Mendès-France university, 2012, 28-31 August  distal-centered ones (see Figure 1b, depicting the LSA-based similarities between Vn and their related causal sentences). Results first showed that local and distal causal sentences are, in all cases but two (local vs. V1 and V5), significantly more verbalized than the rest of the text. Moreover, the content of local causal sentences was significantly better recalled than focal sentences in V1 and V3 (resp. t(43)=3.11, p < .005; t(43)=9.45, p < .0005). Unexpectedly, the content of distal causal sentences was better recalled than local causal sentences for V1 : t(43)=6.09, p < .0005; V2 : t(43)=8.49, p < .0005. Two-way ANOVAs showed significant differences between grades for V1 (distal), F(1, 42)=4.43, p < .05; and a tendency for V6 (distal), F(1, 42)=3.90, p<.06 and for V3 (local), F(1, 42)=2.91; p < .1. Overall, participants’ strategies focused on causality, rather than recency.    Figure 1 a) Mean LSA-based values for similarity of focal sentences by grade; b) Mean LSA-based values for similarity of causal sentences, by grade. Lines: local causality; bars: distal causality. Discussion This study presented a first attempt to set up the foundations of a cognitive reading tutor aiming at analyzing pupils verbalizations to get some traces of their strategies. Results showed that LSA-based analyses of verbalizations correlate moderately to high with those of human experts. Additionally, and as also shown by Trabasso and van den Broek (1985), participants tended to recall sentences they read according to causality-driven, rather than recency-driven strategies, which reveal to some extent their comprehension strategies. Eventually, there was a grade effect on the way distal and local causal sentences are recalled. Further research will aim to refine and to improve the validity of automated analyses, as well as to combine the verbalization-based with complexity-based ones, in order to calibrate a pupil’s reading performance. References Denhière, G., Lemaire, B., Bellissens, C., & Jhean-Larose, S. (2007). A semantic space for modeling children's semantic memory. In T. K. Landauer, D. McNamara, S. Dennis & W. Kintsch (Eds.), Handbook of Latent Semantic Analysis (pp. 143–165). Mahwah: Erlbaum. Graesser, A. C. (2007). An introduction to strategic reading comprehension. In D. S. McNamara (Ed.), Reading comprehension strategies (pp. 3–26). Mahwah: Erlbaum. Landauer, T. K., & Dumais, S. T. (1997). A solution to Plato's problem. Psychol. Rev., 104(2), 211–240. McNamara, D. (2004). SERT: Self-Explanation Reading Training. Discourse Processes, 38, 1–30. McNamara, D., Boonthum, C., & Levinstein, I. (2007). Evaluating self-explanations in iSTART: Comparing word-based and LSA algorithms. In T. K. Landauer, D. McNamara, S. Dennis & W. Kintsch (Eds.), Handbook of Latent Semantic Analysis (pp. 227–241). Mahwah: Erlbaum. Millis, K., Magliano, J., & Todaro, S. (2006). Measuring discourse-level processes with verbal protocols and Latent Semantic Analysis. Scientific Studies of Reading, 10(3), 225–240. Trabasso, T., & van den Broek, P. (1985). Causal thinking and the representation of narrative events. Journal of Memory and Language, 24, 612–630.  Meeting of the EARLI SIG 2 “Comprehension of Text and Graphics”. Grenoble : Pierre-Mendès-France university, 2012, 28-31 August  Vitale, M. R., & Romance, N. R. (2007). A knowledge-based framework for unifying content-area reading comprehension and reading comprehension strategies. In D. S. McNamara (Ed.), Reading comprehension strategies (pp. 73–104). Mahwah: Erlbaum. Wolfe, M. B. W., Magliano, J. P., & Larsen, B. (2005). Causal and semantic relatedness in discourse understanding and representation. Discourse Processes, 39(2-3), 165–187. 

Automated analysis of pupils' self-explanations of a narrative text

Abstract

Similar works

Full text

Available Versions

Hal - Université Grenoble Alpes