Integrating curriculum-based dynamic assessment in computerized adaptive testing: Development and predictive validity of the EDPL-BAI battery on reading competence

In recent decades, there have been significant changes in the conceptualization of reading as well as in the perception of how this activity should be assessed. Interest in the analysis of reading processes has led to the emergence of new explanatory models based primarily on the contributions of cognitive psychology. In parallel, there have been notable advances in measurement procedures, especially in models based on Item Response Theory (IRT), as well as in the capacity and performance of specific software programs that allow data to be managed and analyzed. These changes have contributed significantly to the rise of testing procedures such as computerized adaptive tests (CATs), whose fundamental characteristic is that the sequence of items presented in the tests is adapted to the level of competence that the subject manifests. Likewise, the incorporation of elements of dynamic assessment (DA) as the prompts are gradually offered allows for obtaining information about the type and degree of support required to optimize the subject’s performance. In this sense, the confluence of contributions from DA and CATs offers a new possibility for approaching the assessment of learning processes. In this article, we present a longitudinal research developed in two phases, through which a computerized dynamic adaptive assessment battery of reading processes (EDPL-BAI) was configured. The research frame involved 1,831 students (46% girls) from 13 public schools in three regions of Chile. The purpose of this study was to analyze the differential contribution on reading competence of dynamic scores obtained in a subsample composed of 324 (47% girls) students from third to sixth grade after the implementation of a set of adaptive dynamic tests of morpho-syntactic processes. The results achieved in the structural equation modeling indicate a good global fit. Individual relationships show a significant contribution of calibrated score that reflects estimated knowledge level on reading competence, as well as dynamic scores based on the assigned value of graduated prompts required by the students. These results showed significant predictive values on reading competence and incremental validity in relation to predictions made by static criterion tests

Accounting students' IT applicaton skills over a 10-year period

This paper reports on the changing nature of a range of information technology (IT) application skills that students declare on entering an accounting degree over the period from 1996 to 2006. Accounting educators need to be aware of the IT skills students bring with them to university because of the implications this has for learning and teaching within the discipline and the importance of both general and specific IT skills within the practice and craft of accounting. Additionally, IT skills constitute a significant element within the portfolio of employability skills that are increasingly demanded by employers and emphasized within the overall Higher Education (HE) agenda. The analysis of students' reported IT application skills on entry to university, across a range of the most relevant areas of IT use in accounting, suggest that their skills have continued to improve over time. However, there are significant differential patterns of change through the years and within cohorts. The paper addresses the generalizability of these findings and discusses the implications of these factors for accounting educators, including the importance of recognising the differences that are potentially masked by the general increase in skills; the need for further research into the changing nature, and implications, of the gender gap in entrants' IT application skills; and the low levels of entrants' spreadsheet and database skills that are a cause for concern

Development and Evaluation of the Nebraska Assessment of Computing Knowledge

One way to increase the quality of computing education research is to increase the quality of the measurement tools that are available to researchers, especially measures of students’ knowledge and skills. This paper represents a step toward increasing the number of available thoroughly-evaluated tests that can be used in computing education research by evaluating the psychometric properties of a multiple-choice test designed to differentiate undergraduate students in terms of their mastery of foundational computing concepts. Classical test theory and item response theory analyses are reported and indicate that the test is a reliable, psychometrically-sound instrument suitable for research with undergraduate students. Limitations and the importance of using standardized measures of learning in education research are discussed

Researching participants taking IELTS Academic Writing Task 2 (AWT2) in paper mode and in computer mode in terms of score equivalence, cognitive validity and other factors

Computer-based (CB) assessment is becoming more common in most university disciplines, and international language testing bodies now routinely use computers for many areas of English language assessment. Given that, in the near future, IELTS also will need to move towards offering CB options alongside traditional paper-based (PB) modes, the research reported here prepares for that possibility, building on research carried out some years ago which investigated the statistical comparability of the IELTS writing test between the two delivery modes, and offering a fresh look at the relevant issues. By means of questionnaire and interviews, the current study investigates the extent to which 153 test-takers’ cognitive processes, while completing IELTS Academic Writing in PB mode and in CB mode, compare with the real-world cognitive processes of students completing academic writing at university. A major contribution of our study is its use – for the first time in the academic literature – of data from research into cognitive processes within real-world academic settings as a comparison with cognitive processing during academic writing under test conditions. The most important conclusion from the study is that according to the 5-facet MFRM analysis, there were no significant differences in the scores awarded by two independent raters for candidates’ performances on the tests taken under two conditions, one paper-and-pencil and the other computer. Regarding analytic scores criteria, the differences in three areas (i.e. Task Achievement, Coherence and Cohesion, and Grammatical Range and Accuracy) were not significant, but the difference reported in Lexical Resources was significant, if slight. In summary, the difference of scores between the two modes is at an acceptable level. With respect to the cognitive processes students employ in performing under the two conditions of the test, results of the Cognitive Process Questionnaire (CPQ) survey indicate a similar pattern between the cognitive processes involved in writing on a computer and writing with paper-and-pencil. There were no noticeable major differences in the general tendency of the mean of each questionnaire item reported on the two test modes. In summary, the cognitive processes were employed in a similar fashion under the two delivery conditions. Based on the interview data (n=30), it appears that the participants reported using most of the processes in a similar way between the two modes. Nevertheless, a few potential differences indicated by the interview data might be worth further investigation in future studies. The Computer Familiarity Questionnaire survey shows that these students in general are familiar with computer usage and their overall reactions towards working with a computer are positive. Multiple regression analysis, used to find out if computer familiarity had any effect on students’ performances on the two modes, suggested that test-takers who do not have a suitable familiarity profile might perform slightly worse than those who do, in computer mode. In summary, the research offered in this report offers a unique comparison with realworld academic writing, and presents a significant contribution to the research base which IELTS and comparable international testing bodies will need to consider, if they are to introduce CB test versions in future

The Impact of Curriculum-Based Examinations on Learning in Canadian Secondary Schools

Externally set curriculum-based examinations at the end of high school apparently have pervasive backwash effects on middle school students, their parents, teachers and school administrators. Holding the social class background of students constant, students from Canadian provinces with examination systems were substantially (23 percent of a standard deviation) better prepared in mathematics and 18 percent of a standard deviation better prepared in science than students from provinces lacking such exams. The effect of an exam system on mathematics achievement of 13 year olds is larger in a standard deviation metric than the decline in math SAT scores between 1969 and 1980 that has been such a focus of public concern. Other natural experiments yield similar findings. When adjustments are made for ethnicity, gender and social class of SAT test takers, New York State ranks higher on the SAT than any of the other 38 states where the test is taken by large numbers of students. The mathematics and science achievement of Swedish high school seniors declined in the years following the elimination of high/medium stakes curriculum-based exams. The analysis also found that examination systems had pervasive effects on school administrators, teachers and parents. In the provinces with external exams, schools were more likely to: -- employ specialist teachers of mathematics and science -- employ teachers who had studied the subject in college, -- have high quality science laboratories -- schedule extra hours of math and science instruction -- assign more homework in math, in science and in other subjects -- have students do or watch experiments in science class and -- schedule frequent tests in math and science class. At home students watch less TV, spend more time reading for fun, and are more likely to report their parents want them to do well in math and science. In addition, parents are more likely to talk to their child about what they are learning at school

Evaluation of Evidence-Based Practices in Online Learning: A Meta-Analysis and Review of Online Learning Studies

A systematic search of the research literature from 1996 through July 2008 identified more than a thousand empirical studies of online learning. Analysts screened these studies to find those that (a) contrasted an online to a face-to-face condition, (b) measured student learning outcomes, (c) used a rigorous research design, and (d) provided adequate information to calculate an effect size. As a result of this screening, 51 independent effects were identified that could be subjected to meta-analysis. The meta-analysis found that, on average, students in online learning conditions performed better than those receiving face-to-face instruction. The difference between student outcomes for online and face-to-face classes—measured as the difference between treatment and control means, divided by the pooled standard deviation—was larger in those studies contrasting conditions that blended elements of online and face-to-face instruction with conditions taught entirely face-to-face. Analysts noted that these blended conditions often included additional learning time and instructional elements not received by students in control conditions. This finding suggests that the positive effects associated with blended learning should not be attributed to the media, per se. An unexpected finding was the small number of rigorous published studies contrasting online and face-to-face learning conditions for K–12 students. In light of this small corpus, caution is required in generalizing to the K–12 population because the results are derived for the most part from studies in other settings (e.g., medical training, higher education)

Psychometrics in Practice at RCEC

A broad range of topics is dealt with in this volume: from combining the psychometric generalizability and item response theories to the ideas for an integrated formative use of data-driven decision making, assessment for learning and diagnostic testing. A number of chapters pay attention to computerized (adaptive) and classification testing. Other chapters treat the quality of testing in a general sense, but for topics like maintaining standards or the testing of writing ability, the quality of testing is dealt with more specifically.\ud All authors are connected to RCEC as researchers. They present one of their current research topics and provide some insight into the focus of RCEC. The selection of the topics and the editing intends that the book should be of special interest to educational researchers, psychometricians and practitioners in educational assessment

Sample sizes for the SF-6D preference based measure of health from the SF-36: a practical guide

Background Health Related Quality of Life (HRQoL) measures are becoming more frequently used in clinical trials and health services research, both as primary and secondary endpoints. Investigators are now asking statisticians for advice on how to plan and analyse studies using HRQoL measures, which includes questions on sample size. Sample size requirements are critically dependent on the aims of the study, the outcome measure and its summary measure, the effect size and the method of calculating the test statistic. The SF-6D is a new single summary preference-based measure of health derived from the SF-36 suitable for use clinical trials and in the economic evaluation of health technologies. Objectives To describe and compare two methods of calculating sample sizes when using the SF-6D in comparative clinical trials and to give pragmatic guidance to researchers on what method to use. Methods We describe two main methods of sample size estimation. The parametric (t-test) method assumes the SF-6D data is continuous and normally distributed and that the effect size is the difference between two means. The non-parametric (Mann-Whitney MW) method assumes the data are continuous and not normally distributed and the effect size is defined in terms of the probability that an observation drawn at random from population Y would exceed an observation drawn at random from population X. We used bootstrap computer simulation to compare the power of the two methods for detecting a shift in location. Results This paper describes the SF-6D and retrospectively calculated parametric and nonparametric effect sizes for the SF-6D from a variety of studies that had previously used the SF-36. Computer simulation suggested that if the distribution of the SF-6D is reasonably symmetric then the t-test appears to be more powerful than the MW test at detecting differences in means. Therefore if the distribution of the SF-6D is symmetric or expected to be reasonably symmetric then parametric methods should be used for sample size calculations and analysis. If the distribution of the SF-6D is skewed then the MW test appears to be more powerful at detecting a location shift (difference in means) than the t-test. However, the differences in power (between the t and MW tests) are small and decrease as the sample size increases. Conclusions We have provided a clear description of the distribution of the SF-6D and believe that the mean is an appropriate summary measure for the SF-6D when it is to be used in clinical trials and the economic evaluation of new health technologies. Therefore pragmatically we would recommend that parametric methods be used for sample size calculation and analysis when using the SF-6D.sample size; health-related quality of life; SF-36; preference-based measures of health; bootstrap simulation

Feedback type as a moderator of the relationship between achievement goals and feedback reactions

The aim of the current study is to shed new light on the inconsistent relationship between performance-approach (PAp) goals and feedback reactions by examining feedback type as a moderator. Results of a field experiment (N = 939) using a web-based work simulation task showed that the effect of achievement-approach goals was moderated by feedback type. Relative to individuals pursuing mastery-approach goals, individuals pursuing PAp goals responded more negatively to comparative feedback but not to task-referenced feedback. In line with the hypothesized mediated moderation model, the interaction between achievement goals and feedback type also indirectly affected task performance through feedback reactions. Providing employees with feedback is a key psychological principle used in a wide range of human resource and performance management instruments (e.g., developmental assessment centres, multi-source/360 degrees feedback, training, selection, performance appraisal, management education, computer-adaptive testing, and coaching). The current study suggests that organizations need to strike a balance between encouraging learning and encouraging performance, as too much emphasis on comparative performance (both in goal inducement and in feedback style) may be detrimental to employees' reactions and rate of performance improvement