Standard setting: Comparison of two methods

BACKGROUND: The outcome of assessments is determined by the standard-setting method used. There is a wide range of standard – setting methods and the two used most extensively in undergraduate medical education in the UK are the norm-reference and the criterion-reference methods. The aims of the study were to compare these two standard-setting methods for a multiple-choice question examination and to estimate the test-retest and inter-rater reliability of the modified Angoff method. METHODS: The norm – reference method of standard -setting (mean minus 1 SD) was applied to the 'raw' scores of 78 4th-year medical students on a multiple-choice examination (MCQ). Two panels of raters also set the standard using the modified Angoff method for the same multiple-choice question paper on two occasions (6 months apart). We compared the pass/fail rates derived from the norm reference and the Angoff methods and also assessed the test-retest and inter-rater reliability of the modified Angoff method. RESULTS: The pass rate with the norm-reference method was 85% (66/78) and that by the Angoff method was 100% (78 out of 78). The percentage agreement between Angoff method and norm-reference was 78% (95% CI 69% – 87%). The modified Angoff method had an inter-rater reliability of 0.81 – 0.82 and a test-retest reliability of 0.59–0.74. CONCLUSION: There were significant differences in the outcomes of these two standard-setting methods, as shown by the difference in the proportion of candidates that passed and failed the assessment. The modified Angoff method was found to have good inter-rater reliability and moderate test-retest reliability

Applying a Thurstonian, two-stage method in the standardized assessment of writing

National standardized assessment programs have increasingly included extended written performances, amplifying the need for reliable, valid and efficient methods of assessment. This article examines a two-stage method using comparative judgments and calibrated exemplars as a complement and alternative to existing methods of assessing writing. Written performances were taken from Australia’s NAPLAN assessment, which included both narrative and persuasive performances from students aged 8 to 15. In Stage 1, assessors performed comparative judgments on 160 performances to form a scale of 36 calibrated exemplars. These comparative judgments showed a very high level of reliability and concurrent validity. In Stage 2, assessors scored 2380 new performances by matching them to the most similar calibrated exemplar. These matching judgments showed a generally high level of reliability and concurrent validity and were reasonably efficient after a familiarization period. Further research is suggested to enhance Stage 2 by simplifying the exemplar scale and scaffolding it with detailed descriptors. Overall, the findings support the use of the method in standardized writing assessment and its application to various learning areas

Applying a Thurstonian, two-stage method in the standardized assessment of writing

National standardized assessment programs have increasingly included extended written performances, amplifying the need for reliable, valid, and efficient methods of assessment. This article examines a two-stage method using comparative judgments and calibrated exemplars as a complement and alternative to existing methods of assessing writing. Written performances were taken from Australia’s National Assessment Program—Literacy and Numeracy, which included both narrative and persuasive performances from students aged 8 to 15. In Stage 1, assessors performed comparative judgments on 160 performances to form a scale of 36 calibrated exemplars. These comparative judgments showed a very high level of reliability and concurrent validity. In Stage 2, assessors scored 2,380 new performances by matching them to the most similar calibrated exemplar. These matching judgments showed a generally high level of reliability and concurrent validity and were reasonably efficient after a familiarization period. Further research is suggested to enhance Stage 2 by simplifying the exemplar scale and scaffolding it with detailed descriptors. Overall, the findings support the use of the method in standardized writing assessment and its application to various learning areas

Charge collection and pore filling in solid-state dye-sensitized solar cells.

The solar to electrical power conversion efficiency for dye-sensitized solar cells (DSCs) incorporating a solid-state organic hole-transporter can be over 5%. However, this is for devices significantly thinner than the optical depth of the active composites and by comparison to the liquid electrolyte based DSCs, which exhibit efficiencies in excess of 10%, more than doubling of this efficiency is clearly attainable if all the steps in the photovoltaic process can be optimized. Two issues are currently being addressed by the field. The first aims at enhancing the electron diffusion length by either reducing the charge recombination or enhancing the charge transport rates. This should enable a larger fraction of photogenerated charges to be collected. The second, though less actively investigated, aims to improve the physical composite formation, which in this instance is the infiltration of mesoporous TiO(2) with the organic hole-transporter 2,2',7,7'-tetrakis(N,N-di-p-methoxypheny-amine)-9,9'-spirobifluorene (spiro-MeOTAD). Here, we perform a broad experimental study to elucidate the limiting factors to the solar cell performance. We first investigate the charge transport and recombination in the solid-state dye-sensitized solar cell under realistic working conditions via small perturbation photovoltage and photocurrent decay measurements. From these measurements we deduce that the electron diffusion length near short-circuit is as long as 20 µm. However, at applied biases approaching open-circuit potential under realistic solar conditions, the diffusion length becomes comparable with the film thickness, ∼2 µm, illustrating that real losses to open-circuit voltage, fill factor and hence efficiency are occurring due to ineffective charge collection. The long diffusion length near short-circuit, on the other hand, illustrates that another process, separate from ineffective charge collection, is rendering the solar cell less than ideal. We investigate the process of TiO(2) mesopore infiltration with spiro-MeOTAD by examining the cross-sectional images of and performing photo-induced absorption spectroscopy on devices with a range of thickness, infiltrated with spiro-MeOTAD with a range of concentrations. We present our interpretation of the mechanism for material infiltration, and by improving the casting conditions demonstrate efficient charge collection through devices of over 7 µm in thickness. This investigation represents an improvement in our understanding of the limiting factors to the dye-sensitized solar cell. However, much work, focused on composite formation and improved kinetic competition, is required to realize the true potential of this concept