3. Toward a Cognitive Theory for the Measu rement of Achievement

INTRODUCTION Given the demands for higher levels of learning in our schools and the press for education in the skilled trades, the professions, and the sciences, we must develop more powerful and specific methods for assessing achievement. We need forms of assessment that educators can use to improve educational practice and to diagnose individual progress by monitoring the outcomes of learning and training. Compared to the well-developed technology for aptitude measurement and selection testing, however, the measurement of achievement and diagnosis of learning problems is underdeveloped. This is because the correlational models that support prediction are insufficient for the task of prescribing remediation or other instructional interventions. Tests can predict fa ilure without a theory of what causes success, but intervening to prevent failure and enhance competence requires deeper understanding. The study of the nature of learning is therefore integral to the assessment of achievement. We must use what we know about the cognitive properties of acquired proficiency and about the structures and processes that develop as a student becomes competent in a domain . We know that learning is not simply a matter of the accretion of subject-matter concepts and procedures; it consists rather of organizing and restructuring of this information to enable skillful procedures and processes of problem representation and solution. Somehow, tests must be sensitive to how well this structuring has proceeded in the student being tested. The usual forms of achievement tests are not effective diagnostic aids. In order for tests to become usefully prescriptive, they must identify performance components that facilitate or interfere with current proficiency and the attainment of eventual higher levels of achievement. Curriculum analysis of the content and skill to be learned in a subject matter does not automatically provide information about how students attain competence about the difficulties they meet in attaining it. An array of subject-matter subtests differing in difficulty is not enough for useful diagnosis. Rather, qualitative indicators of specific properties of performance that influence learning and characterize levels of competence need to be identified. In order to ascertain the critical differences between successful and unsuccessful student performance, we need to appraise the knowledge structures and cognitive processes that reveal degrees of competence in a field of study. We need a fuller understanding of what to test and how test items relate to target knowledge. In contrast, most of current testing technology is post hoc and has focused on what to do after test items are constructed. Analysis of item difficulty, development of discrimination indices, scaling and norming procedures, and analysis of test dimensions and factorial composition take place after the item is written. A theory of acquisition and performance is needed before and during item design

Structured computer-based training in the interpretation of neuroradiological images

Computer-based systems may be able to address a recognised need throughout the medical profession for a more structured approach to training. We describe a combined training system for neuroradiology, the MR Tutor that differs from previous approaches to computer-assisted training in radiology in that it provides case-based tuition whereby the system and user communicate in terms of a well-founded Image Description Language. The system implements a novel method of visualisation and interaction with a library of fully described cases utilising statistical models of similarity, typicality and disease categorisation of cases. We describe the rationale, knowledge representation and design of the system, and provide a formative evaluation of its usability and effectiveness

Effectiveness of QR Code Technology for the Child User’s Informal Learning

Quick Response (QR) code technology plays an important role in scaffolding the child user’s active learning in informal environments. This study examines the impact of mobile phones and QR codes on two informal learning outcomes: increased interest and greater knowledge understanding. Ninety-one children and their families participated in the study as part of the iQ Zoo Project. Qualitative findings suggest that most children’s interest in learning about animals was either maintained or increased as a result of the experience. Quantitative results reveal that QR Code Technology was effective in promoting knowledge gains, especially on subjects that are challenging for the informal learner

Expertise and the interpretation of computerized physiological data: implications for the design of computerized monitoring in neonatal intensive care

This paper presents the outcomes from a cognitive engineering project addressing the design problems of computerized monitoring in neonatal intensive care. Cognitive engineering is viewed, in this project, as a symbiosis between cognitive science and design practice. A range of methodologies has been used: interviews with neonatal staff, ward observations and experimental techniques. The results of these investigations are reported, focusing specifically on the differences between junior and senior physicians in their interpretation of monitored physiological data. It was found that the senior doctors made better use of the different knowledge sources available than the junior doctors. The senior doctors were able to identify more relevant physiological patterns and generated more and better inferences than did their junior colleagues. Expertise differences are discussed in the context of previous psychological research in medical expertise. Finally, the paper discusses the potential utility of these outcomes to inform the design of computerized decision support in neonatal intensive care

Expertise in medicine: using the expert performance approach to improve simulation training

Context We critically review how medical education can benefit from systematic use of the expert performance approach as a framework for measuring and enhancing clinical practice. We discuss how the expert performance approach can be used to better understand the mechanisms underpinning superior performance among health care providers and how the framework can be applied to create simulated learning environments that present increased opportunities to engage in deliberate practice. Expert Performance Approach The expert performance approach is a systematic, evidence-based framework for measuring and analysing superior performance. It has been applied in a variety of domains, but has so far been relatively neglected in medicine and health care. Here we outline the framework and demonstrate how it can be effectively applied to medical education. Deliberate Practice Deliberate practice is defined as a structured and reflective activity, which is designed to develop a critical aspect of performance. Deliberate practice provides an opportunity for error detection and correction, repetition, access to feedback and requires maximal effort, complete concentration and full attention. We provide guidance on how to structure simulated learning environments to encourage the accumulation of deliberate practice. Conclusions We highlight the role of simulation-based training in conjunction with deliberate practice activities such as reflection, rehearsal, trial-and-error learning and feedback in improving the quality of patient care. We argue that the development of expertise in health care is directly related to the systematic identification and improvement of quantifiable performance metrics. In order to optimise the training of expert health care providers, advances in simulation technology need to be coupled with effective instructional systems design, with the latter being strongly guided by empirical research from the learning and cognitive sciences

The Dreyfus model of clinical problem-solving skills acquisition: a critical perspective

Context: The Dreyfus model describes how individuals progress through various levels in their acquisition of skills and subsumes ideas with regard to how individuals learn. Such a model is being accepted almost without debate from physicians to explain the ‘acquisition’ of clinical skills. Objectives: This paper reviews such a model, discusses several controversial points, clarifies what kind of knowledge the model is about, and examines its coherence in terms of problem-solving skills. Dreyfus’ main idea that intuition is a major aspect of expertise is also discussed in some detail. Relevant scientific evidence from cognitive science, psychology, and neuroscience is reviewed to accomplish these aims. Conclusions: Although the Dreyfus model may partially explain the ‘acquisition’ of some skills, it is debatable if it can explain the acquisition of clinical skills. The complex nature of clinical problem-solving skills and the rich interplay between the implicit and explicit forms of knowledge must be taken into consideration when we want to explain ‘acquisition’ of clinical skills. The idea that experts work from intuition, not from reason, should be evaluated carefully