Memory and metacognition in classroom learning : the role of item order in learning with particular reference to the interleaving effect

Education needs to be effective, but previous research suggests that teachers and learners alike are not always aware of which practices lead to lasting, transferable learning and which do not. In particular, research into evidence-based teaching strategies such as the spacing effect, interleaving and retrieval practice have shown a striking mismatch between what classroom choices are supported by the evidence and metacognitive beliefs on the part of learners. In part, this is because these strategies make the process of learning more challenging and error-prone; what Bjork and Bjork (1991) refer to as desirable difficulties tend to lead to poorer performance in the short-term but better learning in the long-term. As such, they are often mistakenly rejected by learners who cannot easily perceived their benefits. This thesis focuses on desirable difficulties that relate to the timing and order in which classroom examples are presented, and in particular on interleaving – the process of mixing or alternating the order of examples during learning. Previous research has established the strength and boundary conditions of other desirable difficulties such as the spacing effect (Cepeda et al., 2006) but a clear picture of these issues was lacking when it came to interleaving. A systematic review was therefore conducted to gauge the strength of the evidence on interleaving, and its potential for application to the secondary classroom. It found that interleaving (as compared to blocking) is associated with high effect sizes which persist across experimental designs and do not appear to be biased by the work of specific labs. However, there was also a gap in the literature when it came to classroom-based field research on the technique, and very little work had been done which related directly to higher-order skills – a key element of many exam-based courses. The next stage of this thesis was therefore to extend investigation of interleaving to classroom situations, focusing particularly on psychology teaching at school level. In a pilot study, high school students engaged in an introductory week for a psychology course experienced spaced and interleaved learning tasks, allowing a computer-based methodology to be tested but revealing no effect of interleaving in the context of brief presentations of factual information. A follow-up which used similar methodology applied to learning the skills of application and evaluation found an advantage of interleaving over blocking. The latter study also found a trend in favour of self-explanation – another desirable difficulty – that did not reach significance. As desirable difficulties are often counterintuitive, this thesis also aimed to investigate whether teachers would endorse these techniques, and what might discourage them from doing so. A wide-ranging survey on learning and memory suggested that teachers’ beliefs about memory are generally more accurate than prior findings among the general public, but are out of line with the scientific consensus when it comes to desirable difficulties such as spacing and retrieval practice. A follow-up study focused on three techniques in particular – interleaving, spacing, and retrieval practice (all desirable difficulties). New student teachers and in-service teachers were shown a set of vignettes, each of which presented a classroom situation relating to one of these techniques and required a response on a 7-point scale to indicate their belief in which of two alternatives (for example, interleaving vs. blocking) would lead to better outcomes. This study found that a minority of teachers favoured the techniques overall, though spacing was more widely endorsed (49% overall) than retrieval practice (30%), and interleaving was endorsed least of all (4%). No relationship was found between years of experience and accuracy across the sample of in-service teachers, and this group were less accurate than the student teachers, supporting the idea that experience does not help when it comes to adopting teaching techniques which are based on counterintuitive features of human memory. Finally, the thesis addresses the implications of these findings for both teaching and professional learning. It considers the role of both interleaving and other related techniques, as well as looking at ways of inculcating research evidence into the profession. It is noted that flawed beliefs about learning and memory often link to teacher identity, and that this is a barrier when it comes to teachers’ choosing to engage with evidence (or not). Some synthesis from the ideas can be achieved by considering the role of desirable difficulties as professional learning tools, and a series of recommendations are set out. The methodology used in the research – systematic reviewing and computer-based field experiments – is also evaluated, and directions for future work identified.Education needs to be effective, but previous research suggests that teachers and learners alike are not always aware of which practices lead to lasting, transferable learning and which do not. In particular, research into evidence-based teaching strategies such as the spacing effect, interleaving and retrieval practice have shown a striking mismatch between what classroom choices are supported by the evidence and metacognitive beliefs on the part of learners. In part, this is because these strategies make the process of learning more challenging and error-prone; what Bjork and Bjork (1991) refer to as desirable difficulties tend to lead to poorer performance in the short-term but better learning in the long-term. As such, they are often mistakenly rejected by learners who cannot easily perceived their benefits. This thesis focuses on desirable difficulties that relate to the timing and order in which classroom examples are presented, and in particular on interleaving – the process of mixing or alternating the order of examples during learning. Previous research has established the strength and boundary conditions of other desirable difficulties such as the spacing effect (Cepeda et al., 2006) but a clear picture of these issues was lacking when it came to interleaving. A systematic review was therefore conducted to gauge the strength of the evidence on interleaving, and its potential for application to the secondary classroom. It found that interleaving (as compared to blocking) is associated with high effect sizes which persist across experimental designs and do not appear to be biased by the work of specific labs. However, there was also a gap in the literature when it came to classroom-based field research on the technique, and very little work had been done which related directly to higher-order skills – a key element of many exam-based courses. The next stage of this thesis was therefore to extend investigation of interleaving to classroom situations, focusing particularly on psychology teaching at school level. In a pilot study, high school students engaged in an introductory week for a psychology course experienced spaced and interleaved learning tasks, allowing a computer-based methodology to be tested but revealing no effect of interleaving in the context of brief presentations of factual information. A follow-up which used similar methodology applied to learning the skills of application and evaluation found an advantage of interleaving over blocking. The latter study also found a trend in favour of self-explanation – another desirable difficulty – that did not reach significance. As desirable difficulties are often counterintuitive, this thesis also aimed to investigate whether teachers would endorse these techniques, and what might discourage them from doing so. A wide-ranging survey on learning and memory suggested that teachers’ beliefs about memory are generally more accurate than prior findings among the general public, but are out of line with the scientific consensus when it comes to desirable difficulties such as spacing and retrieval practice. A follow-up study focused on three techniques in particular – interleaving, spacing, and retrieval practice (all desirable difficulties). New student teachers and in-service teachers were shown a set of vignettes, each of which presented a classroom situation relating to one of these techniques and required a response on a 7-point scale to indicate their belief in which of two alternatives (for example, interleaving vs. blocking) would lead to better outcomes. This study found that a minority of teachers favoured the techniques overall, though spacing was more widely endorsed (49% overall) than retrieval practice (30%), and interleaving was endorsed least of all (4%). No relationship was found between years of experience and accuracy across the sample of in-service teachers, and this group were less accurate than the student teachers, supporting the idea that experience does not help when it comes to adopting teaching techniques which are based on counterintuitive features of human memory. Finally, the thesis addresses the implications of these findings for both teaching and professional learning. It considers the role of both interleaving and other related techniques, as well as looking at ways of inculcating research evidence into the profession. It is noted that flawed beliefs about learning and memory often link to teacher identity, and that this is a barrier when it comes to teachers’ choosing to engage with evidence (or not). Some synthesis from the ideas can be achieved by considering the role of desirable difficulties as professional learning tools, and a series of recommendations are set out. The methodology used in the research – systematic reviewing and computer-based field experiments – is also evaluated, and directions for future work identified

Reconstructing Data Provenance from Log Files

Data provenance describes the derivation history of data, capturing details such as the entities involved and the relationships between entities. Knowledge of data provenance can be used to address issues, such as data quality assurance, data audit and system security. However, current computer systems are usually not equipped with means to acquire data provenance. Modifying underlying systems or introducing new monitoring software for provenance logging may be too invasive for production systems. As a result, data provenance may not always be available. This thesis investigates the completeness and correctness of data provenance reconstructed from log files with respect to the actual derivation history. To accomplish this, we designed and tested a solution that first extracts and models information from log files into provenance relations then reconstructs the data provenance from those relations. The reconstructed output is then evaluated against the ground truth provenance. The thesis also details the methodology used for constructing a dataset for provenance reconstruction research. Experimental results revealed data provenance that completely captures the ground truth can be reconstructed from system-layer log files. However, the outputs are susceptible to errors generated during event logging and errors induced by program dependencies. Results also show that usage of log files of different granularities collected from the system can help resolve logging errors described. Experiments with removing suspected program dependencies using approaches such as blacklisting and clustering have shown that the number of errors can be reduced by a factor of one hundred. Conclusions drawn from this research contribute towards the work on using reconstruction as an alternative approach for acquiring data provenance from computer systems