cdcatR: An R package for cognitive diagnostic computerized adaptive testing

Cognitive diagnosis models (CDMs) are confirmatory latent class models that provide fine-grained information about skills and cognitive processes. These models have gained attention in the last few years because of their usefulness in educational and psychological settings. Recently, numerous developments have been made to allow for the implementation of cognitive diagnosis computerized adaptive testing (CD-CAT). Despite methodological advances, CD-CAT applications are still scarce. To facilitate research and the emergence of empirical applications in this area, we have developed the cdcatR package for R software. The purpose of this document is to illustrate the different functions included in this package. The package includes functionalities for data generation, model selection based on relative fit information, implementation of several item selection rules (including item exposure control), and CD-CAT performance evaluation in terms of classification accuracy, item exposure, and test length. In conclusion, an R package is made available to researchers and practitioners that allows for an easy implementation of CD-CAT in both simulation and applied studies. Ultimately, this is expected to facilitate the development of empirical applications in this areaThis research was funded by Ministerio de Ciencia e Innovación, grant number PSI2017- 85022-P, and Cátedra de Modelos y Aplicaciones Psicométricas (Instituto de Ingeniería del Conocimiento and Autonomous University of Madrid

Technology and Testing

From early answer sheets filled in with number 2 pencils, to tests administered by mainframe computers, to assessments wholly constructed by computers, it is clear that technology is changing the field of educational and psychological measurement. The numerous and rapid advances have immediate impact on test creators, assessment professionals, and those who implement and analyze assessments. This comprehensive new volume brings together leading experts on the issues posed by technological applications in testing, with chapters on game-based assessment, testing with simulations, video assessment, computerized test development, large-scale test delivery, model choice, validity, and error issues. Including an overview of existing literature and ground-breaking research, each chapter considers the technological, practical, and ethical considerations of this rapidly-changing area. Ideal for researchers and professionals in testing and assessment, Technology and Testing provides a critical and in-depth look at one of the most pressing topics in educational testing today

Integrating Timing Considerations to Improve Testing Practices

Integrating Timing Considerations to Improve Testing Practices synthesizes a wealth of theory and research on time issues in assessment into actionable advice for test development, administration, and scoring. One of the major advantages of computer-based testing is the capability to passively record test-taking metadata—including how examinees use time and how time affects testing outcomes. This has opened many questions for testing administrators. Is there a trade-off between speed and accuracy in test taking? What considerations should influence equitable decisions about extended-time accommodations? How can test administrators use timing data to balance the costs and resulting validity of tests administered at commercial testing centers? In this comprehensive volume, experts in the field discuss the impact of timing considerations, constraints, and policies on valid score interpretations; administrative accommodations, test construction, and examinees’ experiences and behaviors; and how to implement the findings into practice. These 12 chapters provide invaluable resources for testing professionals to better understand the inextricable links between effective time allocation and the purposes of high-stakes testing

Discovering Dyslexia: A Phenomenological Study of Dyslexia Experiences Across Age Groups

This phenomenological study focuses on this research question: What types of experiences do students identified early with dyslexia have in comparison to those identified at a later stage in life? Five children and three adults ranging in age from 8-years-old to middle-age were asked to participate in this study. Each participant was interviewed about their diagnosis of dyslexia and what it is like to cope with the disorder in school and everyday life. Participants\u27 accounts were reviewed and organized according to 11 major themes. These themes were compared and contrasted across age groups to look for similarities and differences in experiences. The results of the study indicated that dyslexia diagnosis and treatment has improved in recent years. However, progress can be made in educating others about dyslexia. Further research should be conducted to extend the generalizability of this study to a larger population with a more diverse demographic

Neuropsychological Domains: Comparability in Construct Equivalence Across Test Batteries

The National Center for Education Statistics (NCES, 2016) reported an increase in the number of non-native English-speaking students in U.S. public schools as well as a frequent coexistent correlation with low-SES and poverty, but not for all racial or ethnic minority groups. Because it is well known that SES and language difference play an important role in academic achievement, it is imperative that school psychologists attend to these variables when considering the validity of obtained test scores and their support for subsequent diagnostic conclusions, especially when current rates of ELLs in special education suggest that evaluations are not necessarily providing unbiased results (NCES 2013). This trend remains troublesome despite advances in psychometrics and test development based on theoretical models of intelligence (i.e., CHC, Luria). However, use of tests from varied theoretical camps provides an additional challenge, as not all batteries measure constructs in similar ways (i.e., construct equivalence). As a result, this study evaluated the comparability of construct equivalence on neuropsychological measures across batteries and tests, the extent to which typical neuropsychological domains vary according to how much “language” is used in the measurement of each domain, and the equivalence of scores when domains are assessed in high SES monolingual and bilingual populations in a sample of 252 school-age individuals who underwent evaluations in a private clinic. Results indicated that there is variation in how domains are constructed on certain batteries, confirming that for some tests there is not construct equivalence; high SES bilinguals and monolinguals seem to perform just as well on language tests; and that linguistic demand impacts bilinguals’ performance. Post-hoc analyses indicated that the presence of a diagnosis sometimes indicated poorer performance on domain tasks. Implications include the need to consider the impacts of language, disability, and SES when evaluating bilingual students, as well as test selection during evaluation planning. Further research is needed to address the differences in performance for high and low SES bilinguals and address the possible presence of a “bilingual advantage.

Towards a Research Agenda on Computer-Based Assessment - Challenges and Needs for European Educational Measurement

In 2006 the European Parliament and the Council of Europe have passed recommendations on key competences for lifelong learning and the use of a common reference tool to observe and promote progress in terms of the achievement of goals formulated in ¿Lisbon strategy¿ in March 2000 (revised in 2006, see http://ec.europa.eu/growthandjobs/) and its follow-up declarations. For those areas which are not already covered by existing measurements (foreign languages and learning-to-learn skills), indicators for the identification of such skills are now needed, as well as effective instruments for carrying out large-scale assessments in Europe. In this context it is hoped that electronic testing could improve the effectiveness of the needed assessments, i.e. to improve identification of skills, by reducing costs of the whole operation (financial efforts, human resources etc.). The European Commission is asked to assist Member States to define the organisational and resource implications for them of the construction and administration of tests, including looking into the possibility of adopting e-testing as the means to administer the tests. In addition to traditional testing approaches carried out in a paper-pencil mode, there are a variety of aspects needed to be taken into account when computer-based testing is deployed, such as software quality, secure delivery, if Internet-based: reliable network capacities, support, maintenance, software costs for development and test delivery, including licences. Future European surveys are going to introduce new ways of assessing student achievements. Tests can be calibrated to the specific competence level of each student and become more stimulating, going much further than it can be achieved with traditional multiple choice questions. Simulations provide better means of contextualising skills to real life situations and providing a more complete picture of the actual competence to be assessed. However, a variety of challenges require more research into the barriers posed by the use of technologies, e.g. in terms of computer, performance and security. The ¿Quality of Scientific Information¿ Action (QSI) and the Centre for Research on Lifelong Learning (CRELL) are carrying out a research project on quality criteria of Open Source skills assessment tools. 2 workshops were carried out in previous years bringing together European key experts from assessment research and practice in order to identify and discuss quality criteria relevant for carrying out large-scale assessments at a European level. This report reflects the contributions made on experiences and key challenges for European skills assessment.JRC.G.9-Econometrics and statistical support to antifrau

Integrating Timing Considerations to Improve Testing Practices

Integrating Timing Considerations to Improve Testing Practices synthesizes a wealth of theory and research on time issues in assessment into actionable advice for test development, administration, and scoring. One of the major advantages of computer-based testing is the capability to passively record test-taking metadata—including how examinees use time and how time affects testing outcomes. This has opened many questions for testing administrators. Is there a trade-off between speed and accuracy in test taking? What considerations should influence equitable decisions about extended-time accommodations? How can test administrators use timing data to balance the costs and resulting validity of tests administered at commercial testing centers? In this comprehensive volume, experts in the field discuss the impact of timing considerations, constraints, and policies on valid score interpretations; administrative accommodations, test construction, and examinees’ experiences and behaviors; and how to implement the findings into practice. These 12 chapters provide invaluable resources for testing professionals to better understand the inextricable links between effective time allocation and the purposes of high-stakes testing

When Easy Becomes Boring and Difficult Becomes Frustrating: Disentangling the Effects of Item Difficulty Level and Person Proficiency on Learning and Motivation.

The research on electronic learning environments has evolved towards creating adaptive learning environments. In this study, the focus is on adaptive curriculum sequencing, in particular, the efficacy of an adaptive curriculum sequencing algorithm based on matching the item difficulty level to the learner’s proficiency level. We therefore explored the effect of the relative difficulty level on learning outcome and motivation. Results indicate that, for learning environments consisting of questions focusing on just one dimension and with knowledge of correct response, it does not matter whether we present easy, moderate or difficult items or whether we present the items with a random mix of difficulty levels, regarding both learning and motivation

Systematic review of research on artificial intelligence applications in higher education – where are the educators?

According to various international reports, Artificial Intelligence in Education (AIEd) is one of the currently emerging fields in educational technology. Whilst it has been around for about 30 years, it is still unclear for educators how to make pedagogical advantage of it on a broader scale, and how it can actually impact meaningfully on teaching and learning in higher education. This paper seeks to provide an overview of research on AI applications in higher education through a systematic review. Out of 2656 initially identified publications for the period between 2007 and 2018, 146 articles were included for final synthesis, according to explicit inclusion and exclusion criteria. The descriptive results show that most of the disciplines involved in AIEd papers come from Computer Science and STEM, and that quantitative methods were the most frequently used in empirical studies. The synthesis of results presents four areas of AIEd applications in academic support services, and institutional and administrative services: 1. profiling and prediction, 2. assessment and evaluation, 3. adaptive systems and personalisation, and 4. intelligent tutoring systems. The conclusions reflect on the almost lack of critical reflection of challenges and risks of AIEd, the weak connection to theoretical pedagogical perspectives, and the need for further exploration of ethical and educational approaches in the application of AIEd in higher education

Dissertations and theses on technology and L2 LEARNING (2000-2015)

This list has been updated with an eye towards providing working links to actual dissertation texts. Permanent links available free of charge through university databases have been privileged; for-profit enterprises that sell dissertation manuscripts have been listed when universities do not provide for the electronic dissemination of these documents. Manuscripts that are not available by either of these means are nevertheless listed as we hope to provide the most complete coverage of dissertations in the field of CALL that is possible. You will notice that the reference list style is somewhat different than what is suggested by APA. We decided upon a format that includes the name of the institution where the dissertation was completed such that a quick scan of the list reveals where people are studying CALL related themes—and which universities are field leaders in terms of freely distributing dissertations. If you notice that dissertations completed at your institution have not been included, please send us the references, so we can update this list yearly