Workplace-based assessment: effects of rater expertise

Traditional psychometric approaches towards assessment tend to focus exclusively on quantitative properties of assessment outcomes. This may limit more meaningful educational approaches towards workplace-based assessment (WBA). Cognition-based models of WBA argue that assessment outcomes are determined by cognitive processes by raters which are very similar to reasoning, judgment and decision making in professional domains such as medicine. The present study explores cognitive processes that underlie judgment and decision making by raters when observing performance in the clinical workplace. It specifically focuses on how differences in rating experience influence information processing by raters. Verbal protocol analysis was used to investigate how experienced and non-experienced raters select and use observational data to arrive at judgments and decisions about trainees’ performance in the clinical workplace. Differences between experienced and non-experienced raters were assessed with respect to time spent on information analysis and representation of trainee performance; performance scores; and information processing––using qualitative-based quantitative analysis of verbal data. Results showed expert-novice differences in time needed for representation of trainee performance, depending on complexity of the rating task. Experts paid more attention to situation-specific cues in the assessment context and they generated (significantly) more interpretations and fewer literal descriptions of observed behaviors. There were no significant differences in rating scores. Overall, our findings seemed to be consistent with other findings on expertise research, supporting theories underlying cognition-based models of assessment in the clinical workplace. Implications for WBA are discussed

Supporting Teacher Decision-Making Through Appropriate Feedback

The quality of teaching is significantly enhanced through feedback to teachers about their teaching. Whereas systems to show student learning exist, those showing the emotional state of the classroom do not. We argue that such systems could greatly improve teaching, and in consequence, pupil attainment. Data for such systems could come from biometric sensors or from cameras and microphone monitoring the gaze of students, their restlessness, and the level of noise in the classroom. Such systems would be valuable in particular for teachers with less experience

Early Career Teachers’ ability to focus on typical students errors in relation to the complexity of a mathematical topic

The paper presents results from a computer-based assessment in which 171 early career mathematics teachers from Germany were asked to anticipate typical student errors on a given mathematical topic and identify them under time constraints. Fast and accurate perception and knowledge-based judgments are widely accepted characteristics of teacher competence. The item-wise length of anticipation time, the complexity of mathematical topics and the frequency of right or wrong given answers were used as indicators for teacher competence. The data revealed that anticipation time and the complexity of mathematical topics were related with each other. The groups of test persons with correct and incorrect answers behaved contrarily to the length of the anticipation time. Whereas test persons with correct answers needed more time to anticipate typical errors with an increasing complexity of the errors, the test persons giving false answers responded very quickly, even with an increasing error complexity. This finding confirms results of the expertise research, which emphasize that expert teachers focus more intensively if this is required by a complex task