The Role of Consequences in validity Theory

Peer Reviewedhttp://deepblue.lib.umich.edu/bitstream/2027.42/72502/1/j.1745-3992.1998.tb00826.x.pd

Test development and use in five Iberian Latin American countries

The abundance of scholarship on test development and use generally is higher in English-speaking than in Iberian Latin American countries. The purpose of this article is to help overcome this imbalance by describing and identifying similarities and differences in test development and use in two Iberian (Portugal and Spain) and three of the largest Latin American (Argentina, Brazil, and Venezuela) countries. The stages of test development in each country, roles of professional associations, presence of standards for test use, professionals’ educational training, commonly used tests, together with prominent challenges to continued progress are discussed. Test development and use in these five countries are transitioning from a dependence on the use of translated tests to greater reliance on adapted and finally nationally constructed tests. Continued growth requires adherence to international standards guiding test development and use. Stronger alliance among professional associations in the Iberian Latin American countries could serve as a catalyst to promote test development in these regions.A abundância de estudos sobre o desenvolvimento do teste e usar geralmente é maior em Inglês de língua do que nos países latino-americanos ibéricos. O objetivo deste artigo é ajudar a superar este desequilíbrio, descrever e identificar semelhanças e diferenças no desenvolvimento de testes e uso em dois Ibérica (Portugal e Espanha) e três dos maiores países da América Latina (Argentina, Brasil e Venezuela). Os estágios de desenvolvimento do teste em cada país, os papéis das associações profissionais, presença de padrões para uso de teste, a formação dos profissionais da educação, os testes comumente utilizados, juntamente com desafios importantes ao progresso continuado são discutidos. Desenvolvimento de testes e uso nestes cinco países estão em transição de uma dependência do uso de testes traduzidos para uma maior dependência de testes adaptados e finalmente construídos nacionalmente. O crescimento contínuo exige a adesão a padrões internacionais orientadores desenvolvimento de testes e uso. Aliança mais forte entre as associações profissionais dos países latino-americanos ibéricos poderia servir como um catalisador para promover o desenvolvimento do teste nessas regiões

Using Differential Item Functioning to evaluate potential bias in a high stakes postgraduate knowledge based assessment

BACKGROUND: Fairness is a critical component of defensible assessment. Candidates should perform according to ability without influence from background characteristics such as ethnicity or sex. However, performance differs by candidate background in many assessment environments. Many potential causes of such differences exist, and examinations must be routinely analysed to ensure they do not present inappropriate progression barriers for any candidate group. By analysing the individual questions of an examination through techniques such as Differential Item Functioning (DIF), we can test whether a subset of unfair questions explains group-level differences. Such items can then be revised or removed. METHODS: We used DIF to investigate fairness for 13,694 candidates sitting a major international summative postgraduate examination in internal medicine. We compared (a) ethnically white UK graduates against ethnically non-white UK graduates and (b) male UK graduates against female UK graduates. DIF was used to test 2773 questions across 14 sittings. RESULTS: Across 2773 questions eight (0.29%) showed notable DIF after correcting for multiple comparisons: seven medium effects and one large effect. Blinded analysis of these questions by a panel of clinician assessors identified no plausible explanations for the differences. These questions were removed from the question bank and we present them here to share knowledge of questions with DIF. These questions did not significantly impact the overall performance of the cohort. Group-level differences in performance between the groups we studied in this examination cannot be explained by a subset of unfair questions. CONCLUSIONS: DIF helps explore fairness in assessment at the question level. This is especially important in high-stakes assessment where a small number of unfair questions may adversely impact the passing rates of some groups. However, very few questions exhibited notable DIF so differences in passing rates for the groups we studied cannot be explained by unfairness at the question level

Practitioners' views and barriers to implementation of the Keeping Birth Normal tool: A pilot study

Background: Poor implementation of evidence in practice has been reported as a reason behind the continued rise in unnecessary interventions in labour and birth. A validated tool can enable the systematic measurement of care to target interventions to support implementation of evidence. The Keeping Birth Normal tool has been developed to measure and support implementation of evidence to reduce unnecessary interventions in labour and birth. Aims: This pilot sought the views of midwives about the usefulness and relevance of the Keeping Birth Normal tool in measuring and supporting practice; it also identified barriers to implementation. Methods: Five midwives supported by five preceptors tested the tool on a delivery suite and birth centre in a local NHS Trust. Mixed methods were employed. Participants completed a questionnaire about the relevance and usefulness of the tool. Semi-structured interviews explored participants' experience of using the tool in practice. Findings: The domains and items in the tool were viewed as highly relevant to reducing unnecessary interventions. Not all midwives were open to their practice being observed, but those who were reported benefits from critical reflection and role-modelling to support implementation. An important barrier is a lack of expertise among preceptors to support the implementation of skills to reduce unnecessary interventions. This includes skills in the use of rating scales and critical reflection. Where expertise is available, there is a lack of protected time for such structured supportive activity. Norms in birth environments that do not promote normal birth are another important barrier. Conclusions: Midwives found the items in the tool relevant to evidence-informed skills to reduce unnecessary interventions and useful for measuring and supporting implementation. To validate and generalise these findings, further evidence about the quality of items needs to be gathered. Successful implementation of the tool requires preceptors skilled in care that reduces unnecessary interventions, using rating scales, role-modelling and critical reflection. Such structured preceptorship requires protected time and can only thrive in a culture that promotes normal birth

Linking tests of English for academic purposes to the CEFR: the score user’s perspective

The Common European Framework of Reference for Languages (CEFR) is widely used in setting language proficiency requirements, including for international students seeking access to university courses taught in English. When different language examinations have been related to the CEFR, the process is claimed to help score users, such as university admissions staff, to compare and evaluate these examinations as tools for selecting qualified applicants. This study analyses the linking claims made for four internationally recognised tests of English widely used in university admissions. It uses the Council of Europe’s (2009) suggested stages of specification, standard setting, and empirical validation to frame an evaluation of the extent to which, in this context, the CEFR has fulfilled its potential to “facilitate comparisons between different systems of qualifications.” Findings show that testing agencies make little use of CEFR categories to explain test content; represent the relationships between their tests and the framework in different terms; and arrive at conflicting conclusions about the correspondences between test scores and CEFR levels. This raises questions about the capacity of the CEFR to communicate competing views of a test construct within a coherent overarching structure