Towards Consistency-Based Reliability Assessment

International audienceMOTIVATION : Merging information provided by several sources is an important issue and merging techniques have been extensively studied. When the reliability of the sources is not known, one can apply merging techniques such as majority or arbitration merging or distancebasedmerging for solving conflicts between information. At the opposite, if the reliability of the sources is known, either represented in a quantitative or in a qualitative way, then it can be used to manage contradictions: information provided by a source is generally weakened or ignored if it contradicts information provided by a more reliable source [1, 4, 6]. Assessing the reliability of information sources is thus crucial. The present paper addresses this key question. We adopt a qualitative point of view for reliability representation by assuming that the relative reliability of information sources is represented by a total preorder. This works considers that we have no information about the sources and in particular, we do not know if they are correct (i.e they provide true information) or not. We focus on a preliminary stage of observation and assessment of sources. We claim that during that stage the key issue is a consistency analysis of information provided by sources, whether it is the consistency of single reports or consistency w.r.t trusted knowledge or the consistency of different reports together. We adopt an axiomatic approach: first we give some postulates which characterize what this reliability preorder should be, then we define a generic operator for building this preorder in agreement with the postulates

How competent are healthcare professionals in working according to a bio-psycho-social model in healthcare? : the current status and validation of a scale

Background : Over the past decades, there has been a paradigm shift from a purely biomedical towards a bio-psycho-social (BPS) conception of disability and illness, which has led to a change in contemporary healthcare. However, there seems to be a gap between the rhetoric and reality of working within a BPS model. It is not clear whether healthcare professionals show the necessary skills and competencies to act according to the BPS model. Objective : The aim of this study was (1) to develop a scale to monitor the BPS competencies of healthcare professionals, (2) to define its factor-structure, (3) to check internal consistency, (4) test-retest reliability and (5) feasibility. Design and Setting : Item derivation for the BPS scale was based on qualitative research with seven multidisciplinary focus groups (n = 58) of both patients and professionals. In a cross-sectional study design, 368 healthcare professionals completed the BPS scale through a digital platform. An exploratory factor analysis was performed to determine underlying dimensions. Statistical coherence was expressed in item-total correlations and in Cronbach's alpha coefficient. An intra-class-correlation coefficient was used to rate the test-retest reliability. Results : The qualitative study revealed 45 items. The exploratory factor analysis showed five underlying dimensions labelled as: (1) networking, (2) using the expertise of the client, (3) assessment and reporting, (4) professional knowledge and skills and (5) using the environment. The results show a good to strong homogeneity (item-total ranged from 0.59 to 0.79) and a strong internal consistency (Cronbach's alpha ranged from 0.75 to 0.82). ICC ranged between 0.82 and 0.93. Conclusion : The BPS scale appeared to be a valid and reliable measure to rate the BPS competencies of the healthcare professionals and offers opportunities for an improvement in the healthcare delivery. Further research is necessary to test the construct validity and to detect whether the scale is responsive and able to detect changes over time

Estimating intra-rater reliability on an oral english proficiency test from a Bilingual Education Program

Este estudio tiene como objetivo presentar los resultados de una investigación la cual pretendía estimar el nivel de confiabilidad intra-evaluador en un examen de suficiencia oral en inglés, y determinar los diferentes factores internos y externos que afectan la consistencia del evaluador. Los participantes involucrados en el desarrollo de este estudio fueron dos profesores encargados de evaluar la sección de habla de un examen de suficiencia administrado en la Licenciatura en Bilingüismo con énfasis en inglés. Se calculó un coeficiente de correlación con el fin de establecer la consistencia de los evaluadores mientras que un protocolo verbal retrospectivo fue llevado a cabo para recopilar información acerca de los factores que influyen en la confiabilidad del evaluador. Los resultados sugieren que hay un alto nivel de confiabilidad intra-evaluador en el examen de suficiencia en cuanto el coeficiente de correlación arrojó valores superiores a .80. No obstante, aspectos relacionados con la falta de adhesión a los criterios de la rúbrica, la relación evaluador-estudiante, las condiciones físicas, y la presión y responsabilidad del evaluador para dar una nota precisa fueron identificados como factores que afectan la consistencia del evaluador. Finalmente, se proporcionaron algunas implicaciones procedentes de esta investigación

Clinical audit project in undergraduate medical education curriculum: An assessment validation study

Objectives: To evaluate the merit of the Clinical Audit Project (CAP) in an assessment program for undergraduate medical education using a systematic assessment validation framework. Methods: A cross-sectional assessment validation study at one medical school in Western Australia, with retrospective qualitative analysis of the design, development, implementation and outcomes of the CAP, and quantitative analysis of assessment data from four cohorts of medical students (2011-2014). Results: The CAP is fit for purpose with clear external and internal alignment to expected medical graduate outcomes. Substantive validity in students’ and examiners’ response processes is ensured through relevant methodological and cognitive processes. Multiple validity features are built-in to the design, planning and implementation process of the CAP. There is evidence of high internal consistency reliability of CAP scores (Cronbach’s alpha \u3e 0.8) and inter-examiner consistency reliability (intra-class correlation\u3e0.7). Aggregation of CAP scores is psychometrically sound, with high internal consistency indicating one common underlying construct. Significant but moderate correlations between CAP scores and scores from other assessment modalities indicate validity of extrapolation and alignment between the CAP and the overall target outcomes of medical graduates. Standard setting, score equating and fair decision rules justify consequential validity of CAP scores interpretation and use. Conclusions: This study provides evidence demonstrating that the CAP is a meaningful and valid component in the assessment program. This systematic framework of validation can be adopted for all levels of assessment in medical education, from individual assessment modality, to the validation of an assessment program as a whole

Public perceptions of unreliability in examination results in England: a new perspective

Presented at the 36th International Association for Educational Assessment (IAEA) Annual Conference in Bangkok, Thailand, 22-27 August 2010. This report is about a quantitative study on public perceptions of unreliability in examination results, using an online questionnaire survey

Measures of Health-Related Quality of Life Outcomes in Pediatric Neurosurgery: Literature Review

Background Improving value in healthcare means optimizing outcomes and minimizing costs. The emerging pay-for-performance era requires understanding of the effect of healthcare services on health-related quality of life (HRQoL). Pediatric and surgical subspecialties have yet to fully integrate HRQoL measures into practice. The present study reviewed and characterized the HRQoL outcome measures across various pediatric neurosurgical diagnoses. Methods A literature review was performed by searching PubMed and Google Scholar with search terms such as “health-related quality of life” and “pediatric neurosurgery” and then including the specific pathologies for which a HRQoL instrument was found (e.g., “health-related quality of life” plus “epilepsy”). Each measurement was evaluated by content and purpose, relative strengths and weaknesses, and validity. Results We reviewed 68 reports. Epilepsy, brain tumor, cerebral palsy, spina bifida, hydrocephalus, and scoliosis were diagnoses found in reported studies that had used disease-specific HRQoL instruments. Information using general HRQoL instruments was also reported. Internal, test–retest, and/or interrater reliability varied across the instruments, as did face, content, concurrent, and/or construct validity. Few instruments were tested enough for robust reliability and validity. Significant variability was found in the usage of these instruments in clinical studies within pediatric neurosurgery. Conclusions The HRQoL instruments used in pediatric neurosurgery are currently without standardized guidelines and thus exhibit high variability in use. Clinicians should support the development and application of these methods to optimize these instruments, promote standardization of research, improve performance measures to reflect clinically modifiable and meaningful outcomes, and, ultimately, lead the national discussion in healthcare quality and patient-centered care

The implicit relational assessment procedure: emerging reliability and validity data

The Implicit Relational Assessment Procedure (IRAP) is a measure of ‘implicit cognition' developed on the basis of a contemporary behavioural analysis of language and cognition. The IRAP has now been applied to a range of foci over five years of published research. A frequently-cited caveat in publications to date is the need for further research to gauge the reliability and validity of the IRAP as an implicit measure. This review paper will provide a critical synthesis of available evidence for reliability and validity. The review applies a multifaceted test-theory approach to validity, and reliability is assessed through meta-analysis of published data. The discussion critically considers reviewed IRAP evidence with reference to the extant literature on alternative implicit measures, limitations of studies to date, and consideration of broader conceptual issues