The problem of assessing problem solving: can comparative judgement help?

School mathematics examination papers are typically dominated by short, structured items that fail to assess sustained reasoning or problem solving. A contributory factor to this situation is the need for student work to be marked reliably by a large number of markers of varied experience and competence. We report a study that tested an alternative approach to assessment, called comparative judgement, which may represent a superior method for assessing open-ended questions that encourage a range of unpredictable responses. An innovative problem solving examination paper was specially designed by examiners, evaluated by mathematics teachers, and administered to 750 secondary school students of varied mathematical achievement. The students’ work was then assessed by mathematics education experts using comparative judgement as well as a specially designed, resourceintensive marking procedure. We report two main findings from the research. First, the examination paper writers, when freed from the traditional constraint of producing a mark scheme, designed questions that were less structured and more problem-based than is typical in current school mathematics examination papers. Second, the comparative judgement approach to assessing the student work proved successful by our measures of inter-rater reliability and validity. These findings open new avenues for how school mathematics, and indeed other areas of the curriculum, might be assessed in the future

The use of extrafloral nectar in pest management: overcoming context dependence

Extrafloral nectar (EFN) provides plants with indirect defence against herbivores by attracting predatory insects, predominantly ants. Decades of research have supported the role of EFN as an effective plant defence, dating back to Thomas Belt\u27s description of ants on acacia in 1874. Despite this extensive body of literature, knowledge of the ecological role of EFN has rarely been applied in the field of pest management. We review the existing literature on the use of EFN in agriculture and consider the obstacles that have hindered this transition. Chief among these obstacles is the influence of ecological context on the outcome of EFN-mediated interactions. As such, we consider the options for various agricultural systems in the light of the growth habit of EFN-producing species, focusing first on orchard species and then on herbaceous crops. In each case, we highlight the benefits and difficulties of utilizing EFN as a pest management tool and of measuring its efficacy. Synthesis and applications. We argue that it is time for a shift in extrafloral nectar (EFN) research towards applied settings and seek to address the question: How can a context-dependent and often inducible plant trait be utilized as a reliable tool in agricultural pest management? Breeding crops for increased EFN production, and intercropping with EFN-producing plants, can enhance assemblages of beneficial insects in many agricultural settings. Orchard systems, in particular, provide an ecological context in which the attraction of ants can contribute to cost-effective and sustainable pest management programmes over a broad geographic range

Developing Work-Ready Software Engineers using Real-world Team-based Projects as a Catalyst for Learning

The National Software Academy in Wales is a recentlyestablished collaboration between Cardiff University, WelshGovernment, the Alacrity Foundation and the technologyindustry. The aim is to address the gap in the skills market forwork ready software engineers. The Academy runs an industryfocused BSc degree in Applied Software Engineering that seeslocal and national industry partners actively participating in therunning of the program and gives students the opportunity towork on real-world team-based projects throughout their degreecourse. This paper reports on the first year of the program: thesuccess and positive engagement from industry partners, theintegrated style of delivery across program modules, and thechallenges faced going into year 2

Teaching the substitutive conception of the equals sign

A cumulative body of research has shown that children typically shift from an operational to a relational conception of the equals sign as they move through schooling. Jones (2008) argued that a truly relational conception of the equals sign comprises a substitutive component and a sameness component. Here we present two studies that build on this argument. The first investigated how the equals sign is typically presented to primary children in England, and we report that in the main an operational conception seems to be promoted. The second study measured the impact of a specially designed intervention on early secondary children's conceptions of the equals sign. Pre- and post-test data revealed that the intervention promoted substitutive and sameness components of symbolic equivalence. We consider the theoretical and pedagogical implications of the results

Measuring conceptual understanding: the case of fractions

Developing measures of the quality of understanding of a given mathematical concept has traditionally been a difficult and resource-intensive process. We tested an alternative approach, called Comparative Judgement (CJ), that is based not on psychometric instruments or clinical interviews but collective expertise. Eight mathematics education experts used CJ to assess 25 student responses to a test designed to probe conceptual understanding of fractions. Analysis revealed the CJ assessment process yielded high internal consistency, inter-rater reliability and validity. We discuss the implications of the results for using CJ to measure mathematical understanding in a variety of domains and contexts

Fifty years of A-level mathematics: have standards changed?

Advanced-level (A-level) mathematics is a high-profile qualification taken by many school leavers in England, Wales, Northern Ireland and around the world as preparation for university study. Concern has been expressed in these countries that standards in A-level mathematics have declined over time, and that school leavers enter university or the workplace lacking the required mathematical knowledge and skills. The situation in England, Wales and Northern Ireland reflects more general international concerns about decreasing educational standards. However, evidence to support this concern has been of limited scope, rarely subjected to peer-review, and of questionable validity. Our study overcame the limitations of previous research into standards over time by applying a comparative judgement technique that enabled the direct comparison of mathematical performance across different examinations. Furthermore, unlike previous research, all examination questions were re-typeset and candidate responses rewritten to reduce bias arising from surface cues. Using this technique, mathematics experts judged A-level scripts from the 1960s, 1990s and the current decade. We report that the experts believed current A-level mathematics standards to have declined since the 1960s, although there was no evidence that they believed standards have declined since the 1990s. We contrast our findings with those from previous comparison studies and consider implications for future research into standards over time

Substitution and sameness: two components of a relational conception of the equals sign

A sophisticated and flexible understanding of the equals sign is important for arithmetic competence and for learning further mathematics, particularly algebra. Research has identified two common conceptions held by children: the equals sign as an operator, and the equals sign as signalling the same value on both sides of the equation. We argue here that as well as these two conceptions, the notion of substitution is also an important part of a sophisticated understanding of mathematical equivalence. We provide evidence from a cross-cultural study in which English and Chinese children were asked to rate the “cleverness” of operational, sameness and substitutive definitions of the equals sign. A Principle Components Analysis revealed the substitutive items were distinct from the sameness items. Furthermore, Chinese children rated the substitutive items as ‘very clever’, whereas the English children rated them as ‘not so clever’, suggesting that the notion of substitution develops differently across the two countries. Implications for developmental models of children’s understanding of equivalence are discussed

Measuring conceptual understanding using comparative judgement

The importance of improving students’ understanding of core concepts in mathematics is well established. However, assessing the impact of different teaching interventions designed to improve students’ conceptual understanding requires the validation of adequate measures. Here we propose a novel method of measuring conceptual understanding based on comparative judgement (CJ). Contrary to traditional instruments, the CJ approach allows test questions for any topic to be developed rapidly. In addition, CJ does not require a detailed rubric to represent conceptual understanding of a topic, as it is instead based on the collective knowledge of experts. In the current studies, we compared CJ to already established instruments to measure three topics in mathematics: understanding the use of p-­‐values in statistics, understanding derivatives in calculus, and understanding the use of letters in algebra. The results showed that CJ was valid as compared to established instruments, and achieved high reliability. We conclude that CJ is a quick and efficient alternative method of measuring conceptual understanding in mathematics and could therefore be particularly useful in intervention studies

How fast is your detector? The effect of temporal response on image quality

With increasing interest in high-speed imaging, there should be an increased interest in the response times of our scanning transmission electron microscope detectors. Previous works have highlighted and contrasted the performance of various detectors for quantitative compositional or structural studies, but here, we shift the focus to detector temporal response, and the effect this has on captured images. The rise and decay times of eight detectors' single-electron response are reported, as well as measurements of their flatness, roundness, smoothness, and ellipticity. We develop and apply a methodology for incorporating the temporal detector response into simulations, showing that a loss of resolution is apparent in both the images and their Fourier transforms. We conclude that the solid-state detector outperforms the photomultiplier tube-based detectors in all areas bar a slightly less elliptical central hole and is likely the best detector to use for the majority of applications. However, using the tools introduced here, we encourage users to effectively evaluate which detector is most suitable for their experimental needs