The Road Ahead for State Assessments

The adoption of the Common Core State Standards offers an opportunity to make significant improvements to the large-scale statewide student assessments that exist today, and the two US DOE-funded assessment consortia -- the Partnership for the Assessment of Readiness for College and Careers (PARCC) and the SMARTER Balanced Assessment Consortium (SBAC) -- are making big strides forward. But to take full advantage of this opportunity the states must focus squarely on making assessments both fair and accurate.A new report commissioned by the Rennie Center for Education Research & Policy and Policy Analysis for California Education (PACE), The Road Ahead for State Assessments, offers a blueprint for strengthening assessment policy, pointing out how new technologies are opening up new possibilities for fairer, more accurate evaluations of what students know and are able to do. Not all of the promises can yet be delivered, but the report provides a clear set of assessment-policy recommendations. The Road Ahead for State Assessments includes three papers on assessment policy.The first, by Mark Reckase of Michigan State University, provides an overview of computer adaptive assessment. Computer adaptive assessment is an established technology that offers detailed information on where students are on a learning continuum rather than a summary judgment about whether or not they have reached an arbitrary standard of "proficiency" or "readiness." Computer adaptivity will support the fair and accurate assessment of English learners (ELs) and lead to a serious engagement with the multiple dimensions of "readiness" for college and careers.The second and third papers give specific attention to two areas in which we know that current assessments are inadequate: assessments in science and assessments for English learners.In science, paper-and-pencil, multiple choice tests provide only weak and superficial information about students' knowledge and skills -- most specifically about their abilities to think scientifically and actually do science. In their paper, Chris Dede and Jody Clarke-Midura of Harvard University illustrate the potential for richer, more authentic assessments of students' scientific understanding with a case study of a virtual performance assessment now under development at Harvard. With regard to English learners, administering tests in English to students who are learning the language, or to speakers of non-standard dialects, inevitably confounds students' content knowledge with their fluency in Standard English, to the detriment of many students. In his paper, Robert Linquanti of WestEd reviews key problems in the assessment of ELs, and identifies the essential features of an assessment system equipped to provide fair and accurate measures of their academic performance.The report's contributors offer deeply informed recommendations for assessment policy, but three are especially urgent.Build a system that ensures continued development and increased reliance on computer adaptive testing. Computer adaptive assessment provides the essential foundation for a system that can produce fair and accurate measurement of English learners' knowledge and of all students' knowledge and skills in science and other subjects. Developing computer adaptive assessments is a necessary intermediate step toward a system that makes assessment more authentic by tightly linking its tasks and instructional activities and ultimately embedding assessment in instruction. It is vital for both consortia to keep these goals in mind, even in light of current technological and resource constraints.Integrate the development of new assessments with assessments of English language proficiency (ELP). The next generation of ELP assessments should take into consideration an English learners' specific level of proficiency in English. They will need to be based on ELP standards that sufficiently specify the target academic language competencies that English learners need to progress in and gain mastery of the Common Core Standards. One of the report's authors, Robert Linquanti, states: "Acknowledging and overcoming the challenges involved in fairly and accurately assessing ELs is integral and not peripheral to the task of developing an assessment system that serves all students well. Treating the assessment of ELs as a separate problem -- or, worse yet, as one that can be left for later -- calls into question the basic legitimacy of assessment systems that drive high-stakes decisions about students, teachers, and schools." Include virtual performance assessments as part of comprehensive state assessment systems. Virtual performance assessments have considerable promise for measuring students' inquiry and problem-solving skills in science and in other subject areas, because authentic assessment can be closely tied to or even embedded in instruction. The simulation of authentic practices in settings similar to the real world opens the way to assessment of students' deeper learning and their mastery of 21st century skills across the curriculum. We are just setting out on the road toward assessments that ensure fair and accurate measurement of performance for all students, and support for sustained improvements in teaching and learning. Developing assessments that realize these goals will take time, resources and long-term policy commitment. PARCC and SBAC are taking the essential first steps down a long road, and new technologies have begun to illuminate what's possible. This report seeks to keep policymakers' attention focused on the road ahead, to ensure that the choices they make now move us further toward the goal of college and career success for all students. This publication was released at an event on May 16, 2011

Context-aware Assessment Using QR-codes

In this paper we present the implementation of a general mechanism to deliver tests based on mobile devices and matrix codes. The system is an extension of Siette, and has not been specifically developed for any subject matter. To evaluate the performance of the system and show some of its capabilities, we have developed a test for a second-year college course on Botany at the School of Forestry Engineering. Students were equipped with iPads and took an outdoor test on plant species identification. All students were able to take and complete the test in a reasonable time. Opinions expressed anonymously by the students in a survey about the usability of the system and the usefulness of the test were very favorable. We think that the application presented in this paper can broaden the applicability of automatic assessment techniques.The presentation of this work has been co-founded by the Universidad de Málaga. Campus de Excelencia Internacional Andalucía Tech

Influence of COVID-19 confinement in students’ performance in higher education

Documento procedente de EdArXiv Preprints , se deposita en Biblos-e Archivo versión 2 de 7 Mayo de 2020This study explores the effects of COVID-19 confinement in the students’ performance in higher education. Using a field experiment of 458 students from three different subjects in Universidad Autónoma de Madrid (Spain), we study the differences in assessments by dividing students into two groups. The first group (control) corresponds to academic years 2017/2018 and 2018/2019. The second group (experimental) corresponds to students from 2019/2020, which is the group of students that interrupted their face-to-face activities because of the confinement. The results show that there is a significant positive effect of the COVID-19 confinement on students’ performance. This effect is also significative in activities that did not change their format when performed after the confinement. We find that this effect is significative both in subjects that increased the number of assessment activities and subjects that did not change the workload of students. Additionally, an analysis of students’ learning strategies before confinement shows that students did not study on a continuous basis. Based on these results, we conclude that COVID-19 confinement changed students’ learning strategies to a more continuous habit, improving their efficiency. For these reasons, better scores in students’ assessment are expected due to COVID-19 confinement that can be explained by an improvement in their learning performanceThis work has been financed by the project Erasmus+ 2017-1-ES01-KA203-038266 Project of the European Union: “Advanced Design of e-Learning Applications Personalizing Teaching to Improve Virtual Education

Influence of COVID-19 confinement on students' performance in higher education

This study analyzes the effects of COVID-19 confinement on the autonomous learning performance of students in higher education. Using a field experiment with 458 students from three different subjects at Universidad Autónoma de Madrid (Spain), we study the differences in assessments by dividing students into two groups. The first group (control) corresponds to academic years 2017/2018 and 2018/2019. The second group (experimental) corresponds to students from 2019/2020, which is the group of students that had their face-to-face activities interrupted because of the confinement. The results show that there is a significant positive effect of the COVID-19 confinement on students' performance. This effect is also significant in activities that did not change their format when performed after the confinement. We find that this effect is significant both in subjects that increased the number of assessment activities and subjects that did not change the student workload. Additionally, an analysis of students' learning strategies before confinement shows that students did not study on a continuous basis. Based on these results, we conclude that COVID-19 confinement changed students' learning strategies to a more continuous habit, improving their efficiency. For these reasons, better scores in students' assessment are expected due to COVID-19 confinement that can be explained by an improvement in their learning performanceThis research was funded by ADeAPTIVE (Advanced Design of e-Learning Applications Personalizing Teaching to Improve Virtual Education) project with the support of the Erasmus + programme of the European Union (grant number 2017-1-ES01-KA203-038266). This study was also funded by ACCIO´, Spain (Pla d’Actuacio´ de Centres Tecnològics 2019) under the project Augmented Workplace. This study was also funded by the Fondo Supera COVID-19 (Project: Development of tools for the assessment in higher education in the COVID-19 confinemen

Students’ and lecturers’ perceptions of computerised adaptive testing as the future of assessing students

The COVID-19 pandemic has been a catalyst for the increased adoption and acceptance of technology in teaching and learning, and assessment. Using a quantitative research design, the study surveyed 640 lecturers and students in higher education institutions in South Africa, using an online survey platform to attain lecturers’ and students’ views of adopting computerised adaptive testing (CAT) in their respective modules, and their perceptions of such a testing methodology. The study found that lecturers and students were comfortable engaging in online learning, with a large percentage being the most comfortable with assessing and completing exams, tests, and activities online. Positive perceptions of adopting CAT as an assessment tool for their qualifications were expressed, with the majority recommending their HEI to implement CAT. A statistical difference was found between race and personal perceptions of CAT. It was further established that the higher the level of knowledge and understanding of CAT exist, the higher the academic perceptions levels of CAT are

Efficiently Measuring the Cognitive Ability of LLMs: An Adaptive Testing Perspective

Large language models (LLMs), like ChatGPT, have shown some human-like cognitive abilities. For comparing these abilities of different models, several benchmarks (i.e. sets of standard test questions) from different fields (e.g., Literature, Biology and Psychology) are often adopted and the test results under traditional metrics such as accuracy, recall and F1, are reported. However, such way for evaluating LLMs can be inefficient and inaccurate from the cognitive science perspective. Inspired by Computerized Adaptive Testing (CAT) used in psychometrics, we propose an adaptive testing framework for LLM evaluation. Rather than using a standard test set and simply reporting accuracy, this approach dynamically adjusts the characteristics of the test questions, such as difficulty, based on the model's performance. This allows for a more accurate estimation of the model's abilities, using fewer questions. More importantly, it allows LLMs to be compared with humans easily, which is essential for NLP models that aim for human-level ability. Our diagnostic reports have found that ChatGPT often behaves like a ``careless student'', prone to slip and occasionally guessing the questions. We conduct a fine-grained diagnosis and rank the latest 6 instruction-tuned LLMs from three aspects of Subject Knowledge, Mathematical Reasoning, and Programming, where GPT4 can outperform other models significantly and reach the cognitive ability of middle-level students. Different tests for different models using efficient adaptive testing -- we believe this has the potential to become a new norm in evaluating large language models

Towards a Research Agenda on Computer-Based Assessment - Challenges and Needs for European Educational Measurement

In 2006 the European Parliament and the Council of Europe have passed recommendations on key competences for lifelong learning and the use of a common reference tool to observe and promote progress in terms of the achievement of goals formulated in ¿Lisbon strategy¿ in March 2000 (revised in 2006, see http://ec.europa.eu/growthandjobs/) and its follow-up declarations. For those areas which are not already covered by existing measurements (foreign languages and learning-to-learn skills), indicators for the identification of such skills are now needed, as well as effective instruments for carrying out large-scale assessments in Europe. In this context it is hoped that electronic testing could improve the effectiveness of the needed assessments, i.e. to improve identification of skills, by reducing costs of the whole operation (financial efforts, human resources etc.). The European Commission is asked to assist Member States to define the organisational and resource implications for them of the construction and administration of tests, including looking into the possibility of adopting e-testing as the means to administer the tests. In addition to traditional testing approaches carried out in a paper-pencil mode, there are a variety of aspects needed to be taken into account when computer-based testing is deployed, such as software quality, secure delivery, if Internet-based: reliable network capacities, support, maintenance, software costs for development and test delivery, including licences. Future European surveys are going to introduce new ways of assessing student achievements. Tests can be calibrated to the specific competence level of each student and become more stimulating, going much further than it can be achieved with traditional multiple choice questions. Simulations provide better means of contextualising skills to real life situations and providing a more complete picture of the actual competence to be assessed. However, a variety of challenges require more research into the barriers posed by the use of technologies, e.g. in terms of computer, performance and security. The ¿Quality of Scientific Information¿ Action (QSI) and the Centre for Research on Lifelong Learning (CRELL) are carrying out a research project on quality criteria of Open Source skills assessment tools. 2 workshops were carried out in previous years bringing together European key experts from assessment research and practice in order to identify and discuss quality criteria relevant for carrying out large-scale assessments at a European level. This report reflects the contributions made on experiences and key challenges for European skills assessment.JRC.G.9-Econometrics and statistical support to antifrau

An academic perspective of assessment questions bank

There are several electronic assessment systems being used in institutions of higher education (HE), especially in Open and Distance Learning (ODL) institutions. Some of these institutions built their assessment system into their institution’s Virtual Learning Environments (VLE). Most of these assessment systems are for general purposes where assessment questions are in the form of simple multiple choice question (MCQ) or short-answer questions. In practice, these types of assessment questions do not match many of the current learning requirements and learning outcomes. The concept of an assessment question bank that can be used by academics to share assessment content within or across an institution is not new, but the advancement of technology and technical developments now have made such a repository realizable than ever before. A question bank is now a specialized repository that can be accessed via a web interface for platform independence. The use of technology in developing the question bank provides much relief for the chores associated with preparing assessments, which in turn enhances the quality of the questions and improves the quality of the assessments. This paper presents the experience of Open University Malaysia (OUM) in developing its own Question Bank (QBank). This QBank system is designed to help the Subject Matter Experts (SMEs) who need to develop, classify and store their assessment such as MCQ and essay-type exam questions. This software is integrated with the OUM’s Virtual Learning Environments (myVLE) in order to allow easier and wider access to the SMEs and faculty. (Abstract by author

Technology and Testing

From early answer sheets filled in with number 2 pencils, to tests administered by mainframe computers, to assessments wholly constructed by computers, it is clear that technology is changing the field of educational and psychological measurement. The numerous and rapid advances have immediate impact on test creators, assessment professionals, and those who implement and analyze assessments. This comprehensive new volume brings together leading experts on the issues posed by technological applications in testing, with chapters on game-based assessment, testing with simulations, video assessment, computerized test development, large-scale test delivery, model choice, validity, and error issues. Including an overview of existing literature and ground-breaking research, each chapter considers the technological, practical, and ethical considerations of this rapidly-changing area. Ideal for researchers and professionals in testing and assessment, Technology and Testing provides a critical and in-depth look at one of the most pressing topics in educational testing today