Evaluation of physicians' professional performance: An iterative development and validation study of multisource feedback instruments

Contains fulltext : 107798.pdf (publisher's version ) (Open Access)BACKGROUND: There is a global need to assess physicians' professional performance in actual clinical practice. Valid and reliable instruments are necessary to support these efforts. This study focuses on the reliability and validity, the influences of some sociodemographic biasing factors, associations between self and other evaluations, and the number of evaluations needed for reliable assessment of a physician based on the three instruments used for the multisource assessment of physicians' professional performance in the Netherlands. METHODS: This observational validation study of three instruments underlying multisource feedback (MSF) was set in 26 non-academic hospitals in the Netherlands. In total, 146 hospital-based physicians took part in the study. Each physician's professional performance was assessed by peers (physician colleagues), co-workers (including nurses, secretary assistants and other healthcare professionals) and patients. Physicians also completed a self-evaluation. Ratings of 864 peers, 894 co-workers and 1960 patients on MSF were available. We used principal components analysis and methods of classical test theory to evaluate the factor structure, reliability and validity of instruments. We used Pearson's correlation coefficient and linear mixed models to address other objectives. RESULTS: The peer, co-worker and patient instruments respectively had six factors, three factors and one factor with high internal consistencies (Cronbach's alpha 0.95 - 0.96). It appeared that only 2 percent of variance in the mean ratings could be attributed to biasing factors. Self-ratings were not correlated with peer, co-worker or patient ratings. However, ratings of peers, co-workers and patients were correlated. Five peer evaluations, five co-worker evaluations and 11 patient evaluations are required to achieve reliable results (reliability coefficient >/= 0.70). CONCLUSIONS: The study demonstrated that the three MSF instruments produced reliable and valid data for evaluating physicians' professional performance in the Netherlands. Scores from peers, co-workers and patients were not correlated with self-evaluations. Future research should examine improvement of performance when using MSF

Reliability, validity and discriminability of patient reported outcomes for non-specific low back pain in a nationwide physical therapy registry: A retrospective observational cohort study.

BackgroundA national clinical registry was established in the Netherlands containing data directly sampled from electronic health record systems of physical therapists (PTs). This registry aims to evaluate the potential of patient reported outcome measures (PROMs) to develop quality indicators (QIs) in physical therapy care.PurposeTo test to what extent the collected PROM data are reliable, valid and discriminatory between practices in measuring outcomes of patients with non-specific low back pain (NSLBP).MethodsIn this retrospective cohort study 865 PT practices with 6,560 PTs voluntarily collected PROM data of patients with NSLBP, using the Quebec Back Pain Disability Scale (QBPDS), the Numeric Pain Rating Scale (NPRS) and the Patient Specific Functioning Scale (PSFS). Reliability was determined by analysing the completeness of the dataset, the comparability by using national reference data, and through checking selection bias in the included patients. Validity was tested using the known-groups contrast between patients with (sub)acute vs. chronic NSLBP. To determine discriminative ability of outcomes between PT practices, case-mix corrected hierarchical multilevel analyses were performed.ResultsReliability was sufficient by confirming fifteen of the sixteen hypotheses: 59% of all patients opted in for data analysis, 42% of these included patients showed repeated measurement, comparing with reference data and potential selection bias showed LimitationsAlthough we have shown the reliability, validity and discriminative ability of the dataset in the quest to develop QIs, we are aware that reducing missing values in patient records and the selective participation of PTs that belong to the innovators needs attention in the next stages of implementation to avoid bias in the results.ConclusionPROMs of patients with NSLBP collected in the national clinical registry of KNGF are reliable, valid and able to discriminate between primary care PT practices

Factors predicting doctors' reporting of performance change in response to multisource feedback.

Contains fulltext : 107885.pdf (publisher's version ) (Open Access)ABSTRACT: BACKGROUND: Multi-source feedback (MSF) offers doctors feedback on their performance from peers (medical colleagues), coworkers and patients. Researchers increasingly point to the fact that only a small majority of doctors (60-70 percent) benefit from MSF. Building on medical education and social psychology literature, the authors identified several factors that may influence change in response to MSF. Subsequently, they quantitatively studied the factors that advance the use of MSF for practice change. METHODS: This observational study was set in 26 non-academic hospitals in the Netherlands. In total, 458 specialists participated in the MSF program. Besides the collation of questionnaires, the Dutch MSF program is composed of a reflective portfolio and a facilitative interview aimed at increasing the acceptance and use of MSF. All specialists who finished a MSF procedure between May 2008 and September 2010 were invited to complete an evaluation form. The dependent variable was self-reported change. Three categories of independent variables (personal characteristics, experiences with the assessments and mean MSF ratings) were included in the analysis. Multivariate regression analysis techniques were used to identify the relation between the independent variables and specialists' reported change in actual practice. RESULTS: In total, 238 medical specialists (response rate 52 percent) returned an evaluation form and participated in the study. A small majority (55 percent) of specialists reported to have changed their professional performance in one or more aspects in response to MSF. Regression analyses revealed that two variables had the most effect on reported change. Perceived quality of mentoring positively influenced reported change (regression coefficient beta = 0.527, p < 0.05) as did negative scores offered by colleagues. (regression coefficient beta = -0.157, p < 0.05). The explained variance of these two variables combined was 34 percent. CONCLUSIONS: Perceived quality of mentoring and MSF ratings from colleagues seem to be the main motivators for the self-reported change in response to MSF by specialists. These insights could leverage in increasing the use of MSF for practice change by investing in the quality of mentors