Nyström, Marcus

Ögren, Magnus

Lund University Publications

LUND UNIVERSITYPO Box 117221 00 Lund+46 46-222 00 00How illustrations influence performance and eye movement behaviour when solvingproblems in vector calculusNyström, Marcus; Ögren, MagnusPublished in:7:e Pedagogiska Inspirationskonferensen 20122012Link to publicationCitation for published version (APA):Nyström, M., & Ögren, M. (2012). How illustrations influence performance and eye movement behaviour whensolving problems in vector calculus. In 7:e Pedagogiska Inspirationskonferensen 2012 (Pedagogiskainspirationskonferensen LTH).Total number of authors:2General rightsUnless other specific re-use rights are stated the following general rights apply:Copyright and moral rights for the publications made accessible in the public portal are retained by the authorsand/or other copyright owners and it is a condition of accessing publications that users recognise and abide by thelegal requirements associated with these rights. • Users may download and print one copy of any publication from the public portal for the purpose of private studyor research. • You may not further distribute the material or use it for any profit-making activity or commercial gain • You may freely distribute the URL identifying the publication in the public portalRead more about Creative commons licenses: https://creativecommons.org/licenses/Take down policyIf you believe that this document breaches copyright please contact us providing details, and we will removeaccess to the work immediately and investigate your claim.LTHs 7:e Pedagogiska Inspirationskonferens, 30 augusti 2012How illustrations influence performance and eyemovement behaviour when solving problems invector calculusMarcus Nyström and Magnus ÖgrenAbstract—Mathematical formulas in vector calculus often havedirect visual representations, which in form of illustrations areused extensively during teaching and when assessing students’levels of understanding. However, there is very little, if any,empirical evidence of how the illustrations are utilized duringproblem solving and whether they are beneficial to comprehen-sion. In this paper we collect eye movements and performancescores (true or false answers) from students while solving eightproblems in vector calculus; 20 students solve illustrated prob-lems whereas 16 students solve the same problems, but withoutthe illustrations. Results show no overall performance benefitfor illustrated problems even though they are clearly visuallyattended. Surprisingly, we found a significant effect of whetherthe answer to the problem was true of false; students were morelikely to answer that the question was true given an illustratedproblem. We interpret this finding as if the illustrations persuadethe students that the answer is true, irrespective of whether ornot it in fact is. These results may question the tacit consensusamong teachers of vector calculus that illustrations are generallybeneficial for comprehending a problem.Index Terms—Problem solving, vector calculus, eye tracking,illustrations.I. INTRODUCTIONPRINTED textbooks are today becoming replaced bycomputerized media that can include both text, images,sound, and animations. In the wake of this transition, thereis an increasing amount of research investigating how suchmultimedia presentation affect students’ learning and compre-hension. An influential framework on learning is the multi-media principle, which states that students learn better frompictures and words than from picture alone [1]. While thisprinciple has been extensively used in educational psychologyusing standard texts and pictures, there is no evidence ofwhether these principles generalize to situations where textis presented along with equations and illustrations of complexmathematical form.Researchers in educational psychology and multimedialearning have recently started to use eye tracking to gaininsight into how students distribute their visual attention wheninspecting learning materials or solving problems [2]. Eyemovements provide information of where people look, for howlong they look there, and how different parts of the problemare visually integrated. Prototypical eye movements such asfixations, where the eye is relatively still, and saccades, whichMarcus Nyström is with the Humanities Laboratory at Lund University.Magnus Ögren is with the Department of Mathematics, Technical Universityof Denmark.are rapid eye movements that move the eye between fixation,are known to correlate with ongoing cognitive processes inthe brain [3], [4]. Therefore, eye movements provide detailedinformation about how the students process a problem andtheir ways toward finding a solution to it.As an example of how eye tracking has been used toreveal problem solving strategies, experts were in [5] foundto look at an imaginary radius of a circle required to solvea geometrical problem, whereas novices did not. In anotherstudy, participants looked at locations where a mechanicalproblem was previously shown in order to solve a new, relatedproblem [6]. This was taken as evidence that eye movementsto empty spaces were used to recall what was previouslyshown and use information from working memory to solvea new problem. In mathematical problem solving, previouswork has shown that eye movements can reveal expert andnovice strategies; experts read the question carefully beforeattending to the rest of the problem and, when inspecting anequation with four possible interpretations, performed pairwisecomparisons of the equation and the correct alternative [7].In this study, we investigate the effect of illustrations onproblem comprehension in vector calculus and use eye track-ing to measure how students visually process the problems.II. METHODSEye movements, speech, and true or false answers werecollected from 36 students (Mage = 21.5, SDage = 3.0).The students were randomly assigned to one of two group:one (N = 20) solved eight illustrated problem and the other(N = 16) solved the same problems but without illustrations.They were students from Lund Institute of Technology (LTH)two weeks into a course in vector calculus. Each problemcomprised an input text describing the problem (top/centreleft), a question about the problem (bottom left) and, for oneof the groups, a related illustration (centre right). The problemswere shown in random order to the students. Figure 1 showsone of the problems.When arriving to the experiment, the students were intro-duced to the equipment, and shown one example problemnot included in the experiment. They were asked to solvethe problems while concurrently verbalizing their thoughts.Eye movements were recorded at 250 Hz with the RED250system and iView X (v. 2.7.13) from SensoMotoric Instru-ments (Berlin, Germany). A problem could be inspected fora maximum of two minutes, but the students were asked toLTHs 7:e Pedagogiska Inspirationskonferens, 30 augusti 2012Fig. 1. One of the eight problem superimposed with eye movement datafrom one student. In the non-illustrated condition, the students solved thesame problem but without have access to the illustration. Lines and circlesrepresent saccades and fixations and illustrate where the student looked whilesolving the problems. The diameter of a circle is proportional to the fixationduration.respond with a mouse click as soon as they knew the answer.The response triggered the appearance of a new stimuluswhere the students answered whether they thought the answerwas true or false.III. RESULTSFigure 2 shows the proportion of correct answers for il-lustrated and non-illustrated problems. There seems to be nogeneral benefit of including an illustration, but the proportionof correct answers appear to be influenced by whether theanswer to a question is true (T) or false (F), in particular whenan illustration is present. Running a linear mixed effect modelin R [8] with proportion of correct answers as an outcome vari-able, illustration (with/without) and correct answer (true/false)as predictors, and student (1–36) and problem (1–8) as randomfactors reveals that there is a significant effect only of whetherthe answer is true or false (p < 0.05). More interestingly,there is an interaction between the predictors (p < 0.01). Thisinteraction is visualized in Figure 3, and confirms that theproportion of correct answer increases when the illustration ispresent and the answer is true.Given the small overall impact the illustrations had onperformance, it is tempting to believe that the students used theinformation from the illustrations sparingly. However, whilethe input and the question attracted the majority of students’visual attention (on average 22 and 21%), the illustrations werealways inspected (11%). Figure 4 shows where the studentswere fixating over the time course of problem solving forone representative problem. Initially, the input text was read,and after about 50 fixation (about 15 seconds) the majority ofstudents had continued to the question. When an illustrationis present (Figure 4(b)), it attracts a fairly constant proportionof fixations over time, and is at every time instant inspectedby about 5–20% of the viewers.IV. DISCUSSION AND CONCLUSIONSWe investigated how illustrations influence comprehensionand visual behaviour when students solve problem in vectorcalculus. Even though we found no comprehension benefitof utilizing illustrations, the illustrated problems had a larger1 20.40.450.50.550.60.650.70.75Condition (with illustration = 1)Proportion of correct answers  Answer true = 0Answer true = 1Fig. 3. A plot visualizing the interaction between the presentation conditionand whether the answer is true or false. The plot shows that an illustratedproblem is beneficial only when the answer to the question is true.0 50 100 150 200 250 300 350 400 45000.20.40.60.81Fixation numberProportion of fixationsclosed_loop_1−0_nograph.jpg  (a) Non-illustrated problem0 50 100 150 200 250 300 350 400 45000.20.40.60.81Fixation numberProportion of fixationsclosed_loop_1−0.jpg  InputQuestionPictureElsewhere(b) Illustrated problemFig. 4. Proportion of fixations in AOIs over time for problem P1. Thestudents start reading the input text, and then proceeds to the question. Thepicture is inspected a fairly constant proportion over time.LTHs 7:e Pedagogiska Inspirationskonferens, 30 augusti 20121/F 2/T 3/T 4/F 5/F 6/T 7/F 8/F00.20.40.60.81Problem number/Answer true (T) or false (F)Proportion of correct answers  With illustrationWithout illustrationFig. 2. The proportion of correct answers per problem.number of correct answers only when the answer was true.We interpret this finding as if the illustrations persuasivelyinfluence the students to believe that a statement is trueregardless of whether or not it in fact is. Similar effects havebeen reported from previous research in brain imaging wherearticles accompanied by brain images received higher ratingsof scientific reasoning than the same articles without images[9]. The reason for the higher credibility of brain images wasexplained by that they provide a concrete representation ofthe abstract cognitive processes required to interpret findingsfrom cognitive neuroscience. It can be hypothesized that theillustrations of complex concepts in vector calculus servethe same purpose by offloading the cognitive system byexternalizing mental representations of vector fields to physicalimages.The eye movement data revealed how the illustrationscaptured students’ attention over the time course of problemsolving. The illustrations were indeed visually attended, ap-parently enough to influence the students’ answers. Futureresearch is however required to answer more in detail therelationship between how students look at text, equations, andillustrations and how problems are identified and solved.ACKNOWLEDGMENTSThe authors want to thank Sveriges läromedelsförfattaresförbund (SLFF) for sponsoring this research. Halszka Jarodzkais acknowledged for valuable comments and ideas to theexperimental design.REFERENCES[1] R. Mayer, “Multimedia learning,” Psychology of Learning and Motiva-tion, vol. 41, pp. 85–139, 2002.[2] T. Van Gog and K. Scheiter, “Eye tracking as a tool to study and enhancemultimedia learning,” Learning and Instruction, vol. 20, no. 2, pp. 95–99,2010.[3] K. Rayner, “Eye movements in reading and information processing: 20years of research,” Psychological Bulletin, vol. 124, no. 3, pp. 372–422,1998.[4] K. Holmqvist, M. Nyström, R. Andersson, R. Dewhurst, H. Jarodzka, andJ. van de Weijer, Eye tracking: A comprehensive guide to methods andmeasures. Oxford: Oxford University Press, 2011.[5] J. Epelboim and P. Suppes, “A model of eye movements and visualworking memory during problem solving in geometry,” Vision Research,vol. 41, no. 12, pp. 1561–1574, 2001.[6] D. Yoon and N. H. Narayanan, “Mental imagery in problem solving:An eye tracking study,” in Proceedings of the 2004 Symposium on Eye-Tracking Research & Applications. New York: ACM, 2004, pp. 77–84.[7] C. Andrà, F. Arzarello, F. Ferrara, K. Holmqvist, P. Lindstrom, O. Robutti,and C. Sabena, “How students read mathematical representations: Aneye tracking study,” in Proceedings of the 33rd Conference of theInternational Group for the Psychology of Mathematics Education, vol. 2,2009, pp. 49–56.[8] D. Bates and M. Maechler, lme4: Linear mixed-effects models usingS4 classes, 2010, r package version 0.999375-42. [Online]. Available:http://CRAN.R-project.org/package=lme4[9] D. P. McCabe and A. D. Caster, “Seeing is believing: The effect of brainimages on judgments of scientic reasoning,” Cognition, vol. 107, pp. 343–352, 2008.Marcus Nyström develops eye-tracking methods and analysis tools for eyemovement data at the Humanities Laboratory at Lund University. He hasworked in a wide spectrum of basic and applied eye movement research fromperceptual video compression to educational psychology. He has co-authoredthe book Eye Tracking: A comprehensive guide to methods and measures(2011), Oxford University Press.Magnus Ögren is teaching at the Technical University of Denmark where heconducts research in applied mathematics and computational physics.

English

How illustrations influence performance and eye movement behaviour when solving problems in vector calculus

Mathematical formulas in vector calculus often have direct visual representations, which in form of illustrations are used extensively during teaching and when assessing students’ levels of understanding. However, there is very little, if any, empirical evidence of how the illustrations are utilized during problem solving and whether they are beneficial to comprehension. In this paper we collect eye movements and performance scores (true or false answers) from students while solving eight problems in vector calculus; 20 students solve illustrated problems whereas 16 students solve the same problems, but without the illustrations. Results show no overall performance benefit for illustrated problems even though they are clearly visually attended. Surprisingly, we found a significant effect of whether the answer to the problem was true of false; students were more likely to answer that the question was true given an illustrated problem. We interpret this finding as if the illustrations persuade the students that the answer is true, irrespective of whether or not it in fact is. These results may question the tacit consensus among teachers of vector calculus that illustrations are generally beneficial for comprehending a problem

Online Research Database In Technology

                       General rights Copyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights.  • Users may download and print one copy of any publication from the public portal for the purpose of private study or research. • You may not further distribute the material or use it for any profit-making activity or commercial gain • You may freely distribute the URL identifying the publication in the public portal   If you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediately and investigate your claim.     Downloaded from orbit.dtu.dk on: Dec 20, 2017How illustrations influence performance and eye movement behaviour when solvingproblems in vector calculus.Ögren, Magnus; Nyström, MarcusPublished in:Proceedings: LTHs 7:e Pedagogiska InspirationskonferensPublication date:2012Link back to DTU OrbitCitation (APA):Ögren, M., & Nyström, M. (2012). How illustrations influence performance and eye movement behaviour whensolving problems in vector calculus. In Proceedings: LTHs 7:e Pedagogiska InspirationskonferensLTHs 7:e Pedagogiska Inspirationskonferens, 30 augusti 2012How illustrations influence performance and eyemovement behaviour when solving problems invector calculusMarcus Nystro¨m and Magnus O¨grenAbstract—Mathematical formulas in vector calculus often havedirect visual representations, which in form of illustrations areused extensively during teaching and when assessing students’levels of understanding. However, there is very little, if any,empirical evidence of how the illustrations are utilized duringproblem solving and whether they are beneficial to comprehen-sion. In this paper we collect eye movements and performancescores (true or false answers) from students while solving eightproblems in vector calculus; 20 students solve illustrated prob-lems whereas 16 students solve the same problems, but withoutthe illustrations. Results show no overall performance benefitfor illustrated problems even though they are clearly visuallyattended. Surprisingly, we found a significant effect of whetherthe answer to the problem was true of false; students were morelikely to answer that the question was true given an illustratedproblem. We interpret this finding as if the illustrations persuadethe students that the answer is true, irrespective of whether ornot it in fact is. These results may question the tacit consensusamong teachers of vector calculus that illustrations are generallybeneficial for comprehending a problem.Index Terms—Problem solving, vector calculus, eye tracking,illustrations.I. INTRODUCTIONPRINTED textbooks are today becoming replaced bycomputerized media that can include both text, images,sound, and animations. In the wake of this transition, thereis an increasing amount of research investigating how suchmultimedia presentation affect students’ learning and compre-hension. An influential framework on learning is the multi-media principle, which states that students learn better frompictures and words than from picture alone [1]. While thisprinciple has been extensively used in educational psychologyusing standard texts and pictures, there is no evidence ofwhether these principles generalize to situations where textis presented along with equations and illustrations of complexmathematical form.Researchers in educational psychology and multimedialearning have recently started to use eye tracking to gaininsight into how students distribute their visual attention wheninspecting learning materials or solving problems [2]. Eyemovements provide information of where people look, for howlong they look there, and how different parts of the problemare visually integrated. Prototypical eye movements such asfixations, where the eye is relatively still, and saccades, whichMarcus Nystro¨m is with the Humanities Laboratory at Lund University.Magnus O¨gren is with the Department of Mathematics, Technical Universityof Denmark.are rapid eye movements that move the eye between fixation,are known to correlate with ongoing cognitive processes inthe brain [3], [4]. Therefore, eye movements provide detailedinformation about how the students process a problem andtheir ways toward finding a solution to it.As an example of how eye tracking has been used toreveal problem solving strategies, experts were in [5] foundto look at an imaginary radius of a circle required to solvea geometrical problem, whereas novices did not. In anotherstudy, participants looked at locations where a mechanicalproblem was previously shown in order to solve a new, relatedproblem [6]. This was taken as evidence that eye movementsto empty spaces were used to recall what was previouslyshown and use information from working memory to solvea new problem. In mathematical problem solving, previouswork has shown that eye movements can reveal expert andnovice strategies; experts read the question carefully beforeattending to the rest of the problem and, when inspecting anequation with four possible interpretations, performed pairwisecomparisons of the equation and the correct alternative [7].In this study, we investigate the effect of illustrations onproblem comprehension in vector calculus and use eye track-ing to measure how students visually process the problems.II. METHODSEye movements, speech, and true or false answers werecollected from 36 students (Mage = 21.5, SDage = 3.0).The students were randomly assigned to one of two group:one (N = 20) solved eight illustrated problem and the other(N = 16) solved the same problems but without illustrations.They were students from Lund Institute of Technology (LTH)two weeks into a course in vector calculus. Each problemcomprised an input text describing the problem (top/centreleft), a question about the problem (bottom left) and, for oneof the groups, a related illustration (centre right). The problemswere shown in random order to the students. Figure 1 showsone of the problems.When arriving to the experiment, the students were intro-duced to the equipment, and shown one example problemnot included in the experiment. They were asked to solvethe problems while concurrently verbalizing their thoughts.Eye movements were recorded at 250 Hz with the RED250system and iView X (v. 2.7.13) from SensoMotoric Instru-ments (Berlin, Germany). A problem could be inspected fora maximum of two minutes, but the students were asked toLTHs 7:e Pedagogiska Inspirationskonferens, 30 augusti 2012Fig. 1. One of the eight problem superimposed with eye movement datafrom one student. In the non-illustrated condition, the students solved thesame problem but without have access to the illustration. Lines and circlesrepresent saccades and fixations and illustrate where the student looked whilesolving the problems. The diameter of a circle is proportional to the fixationduration.respond with a mouse click as soon as they knew the answer.The response triggered the appearance of a new stimuluswhere the students answered whether they thought the answerwas true or false.III. RESULTSFigure 2 shows the proportion of correct answers for il-lustrated and non-illustrated problems. There seems to be nogeneral benefit of including an illustration, but the proportionof correct answers appear to be influenced by whether theanswer to a question is true (T) or false (F), in particular whenan illustration is present. Running a linear mixed effect modelin R [8] with proportion of correct answers as an outcome vari-able, illustration (with/without) and correct answer (true/false)as predictors, and student (1–36) and problem (1–8) as randomfactors reveals that there is a significant effect only of whetherthe answer is true or false (p < 0.05). More interestingly,there is an interaction between the predictors (p < 0.01). Thisinteraction is visualized in Figure 3, and confirms that theproportion of correct answer increases when the illustration ispresent and the answer is true.Given the small overall impact the illustrations had onperformance, it is tempting to believe that the students used theinformation from the illustrations sparingly. However, whilethe input and the question attracted the majority of students’visual attention (on average 22 and 21%), the illustrations werealways inspected (11%). Figure 4 shows where the studentswere fixating over the time course of problem solving forone representative problem. Initially, the input text was read,and after about 50 fixation (about 15 seconds) the majority ofstudents had continued to the question. When an illustrationis present (Figure 4(b)), it attracts a fairly constant proportionof fixations over time, and is at every time instant inspectedby about 5–20% of the viewers.IV. DISCUSSION AND CONCLUSIONSWe investigated how illustrations influence comprehensionand visual behaviour when students solve problem in vectorcalculus. Even though we found no comprehension benefitof utilizing illustrations, the illustrated problems had a larger1 20.40.450.50.550.60.650.70.75Condition (with illustration = 1)Proportion of correct answers  Answer true = 0Answer true = 1Fig. 3. A plot visualizing the interaction between the presentation conditionand whether the answer is true or false. The plot shows that an illustratedproblem is beneficial only when the answer to the question is true.0 50 100 150 200 250 300 350 400 45000.20.40.60.81Fixation numberProportion of fixationsclosed_loop_1−0_nograph.jpg  (a) Non-illustrated problem0 50 100 150 200 250 300 350 400 45000.20.40.60.81Fixation numberProportion of fixationsclosed_loop_1−0.jpg  InputQuestionPictureElsewhere(b) Illustrated problemFig. 4. Proportion of fixations in AOIs over time for problem P1. Thestudents start reading the input text, and then proceeds to the question. Thepicture is inspected a fairly constant proportion over time.LTHs 7:e Pedagogiska Inspirationskonferens, 30 augusti 20121/F 2/T 3/T 4/F 5/F 6/T 7/F 8/F00.20.40.60.81Problem number/Answer true (T) or false (F)Proportion of correct answers  With illustrationWithout illustrationFig. 2. The proportion of correct answers per problem.number of correct answers only when the answer was true.We interpret this finding as if the illustrations persuasivelyinfluence the students to believe that a statement is trueregardless of whether or not it in fact is. Similar effects havebeen reported from previous research in brain imaging wherearticles accompanied by brain images received higher ratingsof scientific reasoning than the same articles without images[9]. The reason for the higher credibility of brain images wasexplained by that they provide a concrete representation ofthe abstract cognitive processes required to interpret findingsfrom cognitive neuroscience. It can be hypothesized that theillustrations of complex concepts in vector calculus servethe same purpose by offloading the cognitive system byexternalizing mental representations of vector fields to physicalimages.The eye movement data revealed how the illustrationscaptured students’ attention over the time course of problemsolving. The illustrations were indeed visually attended, ap-parently enough to influence the students’ answers. Futureresearch is however required to answer more in detail therelationship between how students look at text, equations, andillustrations and how problems are identified and solved.ACKNOWLEDGMENTSThe authors want to thank Sveriges la¨romedelsfo¨rfattaresfo¨rbund (SLFF) for sponsoring this research. Halszka Jarodzkais acknowledged for valuable comments and ideas to theexperimental design.REFERENCES[1] R. Mayer, “Multimedia learning,” Psychology of Learning and Motiva-tion, vol. 41, pp. 85–139, 2002.[2] T. Van Gog and K. Scheiter, “Eye tracking as a tool to study and enhancemultimedia learning,” Learning and Instruction, vol. 20, no. 2, pp. 95–99,2010.[3] K. Rayner, “Eye movements in reading and information processing: 20years of research,” Psychological Bulletin, vol. 124, no. 3, pp. 372–422,1998.[4] K. Holmqvist, M. Nystro¨m, R. Andersson, R. Dewhurst, H. Jarodzka, andJ. van de Weijer, Eye tracking: A comprehensive guide to methods andmeasures. Oxford: Oxford University Press, 2011.[5] J. Epelboim and P. Suppes, “A model of eye movements and visualworking memory during problem solving in geometry,” Vision Research,vol. 41, no. 12, pp. 1561–1574, 2001.[6] D. Yoon and N. H. Narayanan, “Mental imagery in problem solving:An eye tracking study,” in Proceedings of the 2004 Symposium on Eye-Tracking Research & Applications. New York: ACM, 2004, pp. 77–84.[7] C. Andra`, F. Arzarello, F. Ferrara, K. Holmqvist, P. Lindstrom, O. Robutti,and C. Sabena, “How students read mathematical representations: Aneye tracking study,” in Proceedings of the 33rd Conference of theInternational Group for the Psychology of Mathematics Education, vol. 2,2009, pp. 49–56.[8] D. Bates and M. Maechler, lme4: Linear mixed-effects models usingS4 classes, 2010, r package version 0.999375-42. [Online]. Available:http://CRAN.R-project.org/package=lme4[9] D. P. McCabe and A. D. Caster, “Seeing is believing: The effect of brainimages on judgments of scientic reasoning,” Cognition, vol. 107, pp. 343–352, 2008.Marcus Nystro¨m develops eye-tracking methods and analysis tools for eyemovement data at the Humanities Laboratory at Lund University. He hasworked in a wide spectrum of basic and applied eye movement research fromperceptual video compression to educational psychology. He has co-authoredthe book Eye Tracking: A comprehensive guide to methods and measures(2011), Oxford University Press.Magnus O¨gren is teaching at the Technical University of Denmark where heconducts research in applied mathematics and computational physics.

How illustrations influence performance and eye movement behaviour when solving problems in vector calculus.

http://orbit.dtu.dk/ws/files/18212144/23_Nystroem_OEgren_2012.pdf

How illustrations influence performance and eye movement behaviour when solving problems in vector calculus.

Abstract

Similar works

Full text

Available Versions

Lund University Publications

Online Research Database In Technology