The reliability and validity of a portfolio designed as a programmatic assessment of performance in an integrated clinical placement

Background Little is known about the technical adequacy of portfolios in reporting multiple complex academic and performance-based assessments. We explored, first, the influencing factors on the precision of scoring within a programmatic assessment of student learning outcomes within an integrated clinical placement. Second, the degree to which validity evidence supported interpretation of student scores. Methods Within generalisability theory, we estimated the contribution that each wanted factor (i.e. student capability) and unwanted factors (e.g. the impact of assessors) made to the variation in portfolio task scores. Relative and absolute standard errors of measurement provided a confidence interval around a pre-determined pass/fail standard for all six tasks. Validity evidence was sought through demonstrating the internal consistency of the portfolio and exploring the relationship of student scores with clinical experience. Results The mean portfolio mark for 257 students, across 372 raters, based on six tasks, was 75.56 (SD, 6.68). For a single student on one assessment task, 11% of the variance in scores was due to true differences in student capability. The most significant interaction was context specificity (49%), the tendency for one student to engage with one task and not engage with another task. Rater subjectivity was 29%. An absolute standard error of measurement of 4.74%, gave a 95% CI of +/- 9.30%, and a 68% CI of +/- 4.74% around a pass/fail score of 57%. Construct validity was supported by demonstration of an assessment framework, the internal consistency of the portfolio tasks, and higher scores for students who did the clinical placement later in the academic year. Conclusion A portfolio designed as a programmatic assessment of an integrated clinical placement has sufficient evidence of validity to support a specific interpretation of student scores around passing a clinical placement. It has modest precision in assessing students’ achievement of a competency standard. There were identifiable areas for reducing measurement error and providing more certainty around decision-making. Reducing the measurement error would require engaging with the student body on the value of the tasks, more focussed academic and clinical supervisor training, and revisiting the rubric of the assessment in the light of feedback

Psychiatrists and GPs : diagnostic decision making, personality profiles and attitudes toward depression and anxiety

Objectives: The objective of this article is to explore diagnostic decision making around psychological symptoms presenting to general practitioners (GPs) and psychiatrists, identify attitudinal and personality factors of possible relevance in these decisions, and compare GPs and psychiatrists to help identify potential educational targets. Methods: GPs and psychiatrists attended separate peer-facilitated workshops in which two case presentations were discussed. Decision making was explored by structured questions embedded in the workshop, with responses recorded by electronic keypad technology. Participants completed demographic questionnaires and measures of personality and attitudes to depression. Results: GPs and psychiatrists accorded emphasis to different elements of the history, and assigned different diagnoses based on the same set of symptoms. Both groups relied on non-pharmacological management for milder psychological symptoms; GPs were less likely to make a diagnosis of bipolar disorder. Traits of Extraversion and Agreeableness were associated with greater ease in treating depression. Conclusions: Differences in diagnostic decision making likely reflect the different contexts of specialist and generalist practice. Educational targets may include information about key symptoms to assist in diagnostic precision, but further information is needed to determine the best match between diagnostic processes, context and outcome. An awareness of the role of personality factors may help when designing education and support programs

Diagnostic processes in mental health : GPs and psychiatrists reading from the same book but on a different page

Objective: To explore the clinical reasoning processes underpinning diagnostic and management decision-making in treating patients presenting with psychological distress in general practice. Method: Practising GPs were invited to attend small-group workshops in which two case histories were presented. Discussion was GP-facilitated and recorded for thematic analysis. GPs provided demographic data, completed personality and attitudinal questionnaires, and answered a series of multiple-choice questions embedded in the cases. Results: GPs recognize the possibility of psychiatric disorders early in the clinical reasoning process, but are cautious about applying definitive diagnoses. GPs perceive that patients may be resistant to a psychiatric diagnosis and instead emphasize the need to build rapport and explore and exclude physical comorbidities. GPs see patients with a broad spectrum of distress, illness and impairment, in whom the initial presentation of psychological symptoms is often poorly differentiated and somatically focused, requiring elucidation over time. GPs therefore adopt a longitudinal strategy for diagnosis rather than investing heavily in cross-sectional assessment. Conclusion: GPs appear cognizant of possible psychiatric disorders and management strategies, but employ diagnostic strategies and decision-making processes that, in addition to experience and expertise, likely reflect key differences between the primary care and specialist practice settings