Technology and Testing

From early answer sheets filled in with number 2 pencils, to tests administered by mainframe computers, to assessments wholly constructed by computers, it is clear that technology is changing the field of educational and psychological measurement. The numerous and rapid advances have immediate impact on test creators, assessment professionals, and those who implement and analyze assessments. This comprehensive new volume brings together leading experts on the issues posed by technological applications in testing, with chapters on game-based assessment, testing with simulations, video assessment, computerized test development, large-scale test delivery, model choice, validity, and error issues. Including an overview of existing literature and ground-breaking research, each chapter considers the technological, practical, and ethical considerations of this rapidly-changing area. Ideal for researchers and professionals in testing and assessment, Technology and Testing provides a critical and in-depth look at one of the most pressing topics in educational testing today

3. Toward a Cognitive Theory for the Measu rement of Achievement

INTRODUCTION Given the demands for higher levels of learning in our schools and the press for education in the skilled trades, the professions, and the sciences, we must develop more powerful and specific methods for assessing achievement. We need forms of assessment that educators can use to improve educational practice and to diagnose individual progress by monitoring the outcomes of learning and training. Compared to the well-developed technology for aptitude measurement and selection testing, however, the measurement of achievement and diagnosis of learning problems is underdeveloped. This is because the correlational models that support prediction are insufficient for the task of prescribing remediation or other instructional interventions. Tests can predict fa ilure without a theory of what causes success, but intervening to prevent failure and enhance competence requires deeper understanding. The study of the nature of learning is therefore integral to the assessment of achievement. We must use what we know about the cognitive properties of acquired proficiency and about the structures and processes that develop as a student becomes competent in a domain . We know that learning is not simply a matter of the accretion of subject-matter concepts and procedures; it consists rather of organizing and restructuring of this information to enable skillful procedures and processes of problem representation and solution. Somehow, tests must be sensitive to how well this structuring has proceeded in the student being tested. The usual forms of achievement tests are not effective diagnostic aids. In order for tests to become usefully prescriptive, they must identify performance components that facilitate or interfere with current proficiency and the attainment of eventual higher levels of achievement. Curriculum analysis of the content and skill to be learned in a subject matter does not automatically provide information about how students attain competence about the difficulties they meet in attaining it. An array of subject-matter subtests differing in difficulty is not enough for useful diagnosis. Rather, qualitative indicators of specific properties of performance that influence learning and characterize levels of competence need to be identified. In order to ascertain the critical differences between successful and unsuccessful student performance, we need to appraise the knowledge structures and cognitive processes that reveal degrees of competence in a field of study. We need a fuller understanding of what to test and how test items relate to target knowledge. In contrast, most of current testing technology is post hoc and has focused on what to do after test items are constructed. Analysis of item difficulty, development of discrimination indices, scaling and norming procedures, and analysis of test dimensions and factorial composition take place after the item is written. A theory of acquisition and performance is needed before and during item design

Psychometrics in Practice at RCEC

A broad range of topics is dealt with in this volume: from combining the psychometric generalizability and item response theories to the ideas for an integrated formative use of data-driven decision making, assessment for learning and diagnostic testing. A number of chapters pay attention to computerized (adaptive) and classification testing. Other chapters treat the quality of testing in a general sense, but for topics like maintaining standards or the testing of writing ability, the quality of testing is dealt with more specifically.\ud All authors are connected to RCEC as researchers. They present one of their current research topics and provide some insight into the focus of RCEC. The selection of the topics and the editing intends that the book should be of special interest to educational researchers, psychometricians and practitioners in educational assessment

Exploring the Role of Culture in Online Searching Behavior from a Cognitive Perspective

This research investigates differences in Web search behavior from a “cultural cognition” perspective, which was introduced by Nisbett and Norenzayan (2002). To explore the impact of culture and thinking styles on searching behavior is the objective. The overarching hypothesis is that there are cultural differences in cognitive styles and that these differences may lead into different searching behaviors. The findings help search engine designers provide an adaptive navigation support to users from different cultural backgrounds and cognitive styles. The fundamental research question underpinning this research becomes what are the relationships between users’ cognitive styles and cultural background, and their Web search behavior? The results of this doctoral study help with understanding how cognitive processes are modulated by cultures so as to give rise to cultural specific thinking and Web navigation styles. The assumption here is that differences in cognitive style will drive variations in Web search behavior based on national cultural orientation. To examine the research hypotheses, hundred and eleven research subjects are selected from three cultural groups: Americans, Iranians, and Chinese. The two independent factors that are studied in this experimental research are: cultural background and cognitive style. After answering a demographic questionnaire, the participants are given two types of task scenarios to perform on Google while two tracking applications (TechSmith Morae and MyGaze) record their mouse and eye movements. The users’ search behavior is studied through several measures of eye gaze, click behavior, and viewing pattern. By employing the Repeated Measures Analysis of Variance (RM ANOVA) test, the relationships between the independent variables and the search measurements are studied and analyzed using SAS statistical analysis application. This research was conducted in a controlled usability lab, and the participants were provided with a $10 incentive upon finishing the experiment. Even though the results from the statistical analysis did not show significant evidence to support variation in cognitive styles between the participating cultural groups, country of origin appeared to be a strong predictor of differences in the measurements of search behavior. Considering the research hypotheses and the results, we found that the Americans’ search behaviors tended to be more analytical and less explorative, whereas the Iranians and Chinese tended to engage in more exploratory behaviors

EQUIVALENCY EVIDENCE OF THE ENGLISH COMPETENCY TEST ACROSS DIFFERENT MODES: A RASCH ANALYSIS

The outbreak of the COVID-19 pandemic has transformed the educational landscape in a way unseen before. Educational institutions are navigating between offline and online learning worldwide. Computer-based testing is rapidly taking over paper-and-pencil testing as the dominant mode of assessment. In some settings, computer-based and paper-and-pencil assessments can also be offered side-by-side, in which case test developers should ensure the evidence of equivalence between both versions. This study aims to establish the equivalency evidence of different delivery modes of the English Competency Test, an English language assessment for civil service officers developed and used by the Human Resources Development Education and Training Center, a civil service training institution under the Ministry of Finance of the Republic of Indonesia. Psychometric analyses were carried out with the Rasch model to measure the unidimensionality, reliability, separation, and standard error of measurement estimates. The findings demonstrate that the paper-and-pencil and computer-based versions of the language assessment exhibit comparatively equivalent psychometric properties. The computer-based version of the English Competency Test is proven to offer a reliable and comparable alternative to the paper-and-pencil version

Towards a Research Agenda on Computer-Based Assessment - Challenges and Needs for European Educational Measurement

In 2006 the European Parliament and the Council of Europe have passed recommendations on key competences for lifelong learning and the use of a common reference tool to observe and promote progress in terms of the achievement of goals formulated in ¿Lisbon strategy¿ in March 2000 (revised in 2006, see http://ec.europa.eu/growthandjobs/) and its follow-up declarations. For those areas which are not already covered by existing measurements (foreign languages and learning-to-learn skills), indicators for the identification of such skills are now needed, as well as effective instruments for carrying out large-scale assessments in Europe. In this context it is hoped that electronic testing could improve the effectiveness of the needed assessments, i.e. to improve identification of skills, by reducing costs of the whole operation (financial efforts, human resources etc.). The European Commission is asked to assist Member States to define the organisational and resource implications for them of the construction and administration of tests, including looking into the possibility of adopting e-testing as the means to administer the tests. In addition to traditional testing approaches carried out in a paper-pencil mode, there are a variety of aspects needed to be taken into account when computer-based testing is deployed, such as software quality, secure delivery, if Internet-based: reliable network capacities, support, maintenance, software costs for development and test delivery, including licences. Future European surveys are going to introduce new ways of assessing student achievements. Tests can be calibrated to the specific competence level of each student and become more stimulating, going much further than it can be achieved with traditional multiple choice questions. Simulations provide better means of contextualising skills to real life situations and providing a more complete picture of the actual competence to be assessed. However, a variety of challenges require more research into the barriers posed by the use of technologies, e.g. in terms of computer, performance and security. The ¿Quality of Scientific Information¿ Action (QSI) and the Centre for Research on Lifelong Learning (CRELL) are carrying out a research project on quality criteria of Open Source skills assessment tools. 2 workshops were carried out in previous years bringing together European key experts from assessment research and practice in order to identify and discuss quality criteria relevant for carrying out large-scale assessments at a European level. This report reflects the contributions made on experiences and key challenges for European skills assessment.JRC.G.9-Econometrics and statistical support to antifrau

Exploring the factors that influence attitudes and achievement when students take computerized tests

Currently, a problem exists in K-12 education related to the use of technology for the assessment of student learning. Specifically, due to the lack of access to and infrequent use of computers for middle school students, the rise in the use of high stakes computer-based tests may negatively impact student test scores in poor, urban schools. The conceptual framework of this study was informed by Albert Bandura\u27s theory of self-efficacy, the work of The National Center for Fair and Open Testing regarding ending the misuses and flaws of standardized testing, and James Popham\u27s research on quality assessment. The central research question explored the influence of socioeconomic status, computer access/use, attitudes towards computers, and student achievement levels on computerized tests. This research study was a case study involving 2 charter schools in Michigan. The researcher assumed the role of a non-participant observer and was the primary source for data collection and analysis. The participants for this study were students in Grades 6, 7, and 8 at one suburban and one urban charter school. Multiple sources of evidence were collected, including observations, surveys, and documents. Data analysis was conducted at two levels: category construction was used to examine data for each single case, and a cross-case analysis was used to examine the data for patterns and themes, using the research questions as a guide. A key finding was that home computer access coupled with sole use had a positive influence on student achievement, a positive influence on self-perceptions of computer ability, and significantly influenced the amount of computer usage. Implications for positive social change in education were that practitioners would become aware of the negative effects of computerized testing and implement strategies to mitigate the negative effects

Exploring Predictors of Reading Comprehension for Struggling Adult Readers: A Quantile Regression Approach

There is a paucity of research examining the skills that contribute to reading comprehension for adults who struggle with reading, which includes one in six adults in the United States (Organisation for Economic Co-operation and Development, 2013). The current studies addressed some of the gaps in this literature. Study 1 explored the Simple View of Reading (SVR), which postulates that reading comprehension is predicted by two component skills: decoding and linguistic comprehension (Gough & Tunmer, 1986). Participants included 392 struggling adult readers who were native speakers of English. The dimensionality of the SVR components was examined using confirmatory factor analysis. For the decoding component, a parsimonious latent representation inclusive of phonic decoding and word recognition provided good fit to the data. With respect to linguistic comprehension, the subcomponents of oral vocabulary and listening comprehension emerged as separable yet highly related constructs. A structural equation model showed that decoding and listening comprehension made significant unique contributions to reading comprehension, whereas oral vocabulary did not emerge as a significant unique predictor. Additionally, quantile regression analyses indicated that decoding, listening comprehension, and oral vocabulary exhibited significant unique effects on reading comprehension at low, average, and high levels of reading comprehension performance (.10, .50, and .90 quantiles), with decoding making the largest unique contributions. Study 2 examined the influence of decoding, oral vocabulary, fluency, listening comprehension, background knowledge, and inferencing across different reading comprehension tests. Participants included 168 struggling adult readers who were native speakers of English. The explanatory effects of the predictors were estimated for three reading comprehension tests: WJ Passage Comprehension (WJ-PC), RISE Reading Comprehension (RISE-RC), and RAPID Reading Comprehension (RAPID-RC). Ordinary least squares regression analyses indicated that all predictors except for listening comprehension uniquely explained variance in WJ-PC scores, whereas significant unique predictors were limited to decoding and listening comprehension for RAPID-RC and only decoding for RISE-RC. Quantile regression analyses indicated that the effects of oral vocabulary and background knowledge differed across levels of WJ-PC performance, the effects of decoding and listening comprehension differed across levels of RAPID-RC performance, and the effect of decoding was stable across levels of RISE-RC performance

Pitch ability as an aptitude for tone learning

Tone languages such as Mandarin use voice pitch to signal lexical contrasts, presenting a challenge for second/foreign language (L2) learners whose native languages do not use pitch in this manner. The present study examined components of an aptitude for mastering L2 lexical tone. Native English speakers with no previous tone language experience completed a Mandarin word learning task, as well as tests of pitch ability, musicality, L2 aptitude, and general cognitive ability. Pitch ability measures improved predictions of learning performance beyond musicality, L2 aptitude, and general cognitive ability and also predicted transfer of learning to new talkers. In sum, although certain nontonal measures help predict successful tone learning, the central components of tonal aptitude are pitch-specific perceptual measures