Evaluation of physicians' professional performance: An iterative development and validation study of multisource feedback instruments

Contains fulltext : 107798.pdf (publisher's version ) (Open Access)BACKGROUND: There is a global need to assess physicians' professional performance in actual clinical practice. Valid and reliable instruments are necessary to support these efforts. This study focuses on the reliability and validity, the influences of some sociodemographic biasing factors, associations between self and other evaluations, and the number of evaluations needed for reliable assessment of a physician based on the three instruments used for the multisource assessment of physicians' professional performance in the Netherlands. METHODS: This observational validation study of three instruments underlying multisource feedback (MSF) was set in 26 non-academic hospitals in the Netherlands. In total, 146 hospital-based physicians took part in the study. Each physician's professional performance was assessed by peers (physician colleagues), co-workers (including nurses, secretary assistants and other healthcare professionals) and patients. Physicians also completed a self-evaluation. Ratings of 864 peers, 894 co-workers and 1960 patients on MSF were available. We used principal components analysis and methods of classical test theory to evaluate the factor structure, reliability and validity of instruments. We used Pearson's correlation coefficient and linear mixed models to address other objectives. RESULTS: The peer, co-worker and patient instruments respectively had six factors, three factors and one factor with high internal consistencies (Cronbach's alpha 0.95 - 0.96). It appeared that only 2 percent of variance in the mean ratings could be attributed to biasing factors. Self-ratings were not correlated with peer, co-worker or patient ratings. However, ratings of peers, co-workers and patients were correlated. Five peer evaluations, five co-worker evaluations and 11 patient evaluations are required to achieve reliable results (reliability coefficient >/= 0.70). CONCLUSIONS: The study demonstrated that the three MSF instruments produced reliable and valid data for evaluating physicians' professional performance in the Netherlands. Scores from peers, co-workers and patients were not correlated with self-evaluations. Future research should examine improvement of performance when using MSF

The implementation of quality management systems in hospitals: a comparison between three countries

BACKGROUND: Is the implementation of Quality Management (QM) in health care proceeding satisfactorily and can national health care policies influence the implementation process? Policymakers and researchers in a country need to know the answer to this question. Cross country comparisons can reveal whether sufficient progress is being made and how this can be stimulated. The objective of the study was to investigate agreement and disparities in the implementation of QMS between The Netherlands, Hungary and Finland with respect to the evaluation model used and the national policy strategy of the three countries. METHODS: The study has a cross sectional design, based on measurements in 2000. Empirical data about QM-activities in hospitals were gathered by a self-administered questionnaire. The questionnaires were answered by the directors of the hospitals or the quality coordinators. The analyses are based on data from 101 hospitals in the Netherlands, 116 hospitals in Hungary and 59 hospitals in Finland. Outcome measures are the developmental stage of the Quality Management System (QMS), the development within five focal areas, and distinct QM-activities which were listed in the questionnaire. RESULTS: A mean of 22 QM-activities per hospital was found in the Netherlands and Finland versus 20 QM-activities in Hungarian hospitals. Only a small number of hospitals has already implemented a QMS (4% in The Netherlands,0% in Hungary and 3% in Finland). More hospitals in the Netherlands are concentrating on quality documents, whereas Finnish hospitals are concentrating on training in QM and guidelines. Cyclic quality improvement activities have been developed in the three countries, but in most hospitals the results were not used for improvements. All three countries pay hardly any attention to patient participation. CONCLUSION: The study demonstrates that the implementation of QM-activities can be measured at national level and that differences between countries can be assessed. The hypothesis that governmental legislation or financial reimbursement can stimulate the implementation of QM-activities, more than voluntary recommendations, could not be confirmed. However, the results show that specific obligations can stimulate the implementation of QM-activities more than general, framework legislation

Good Clinical Teachers Likely to be Specialist Role Models: Results from a Multicenter Cross-Sectional Survey

Medical educational reform includes enhancing role modelling of clinical teachers. This requires faculty being aware of their role model status and performance. We developed the System for Evaluation of Teaching Qualities (SETQ) to generate individualized feedback on previously defined teaching qualities and role model status for faculty in (non) academic hospitals.(i) To examine whether teaching qualities of faculty were associated with their being seen as a specialist role model by residents, and (ii) to investigate whether those associations differed across residency years and specialties.Cross-sectional questionnaire survey amongst 549 Residents of 36 teaching programs in 15 hospitals in the Netherlands. The main outcome measure was faculty being seen as specialist role models by residents. Statistical analyses included (i) Pearson's correlation coefficients and (ii) multivariable logistic generalized estimating equations to assess the (adjusted) associations between each of five teaching qualities and 'being seen as a role model'.407 residents completed a total of 4123 evaluations of 662 faculty. All teaching qualities were positively correlated with 'being seen as a role model' with correlation coefficients ranging from 0.49 for 'evaluation of residents' to 0.64 for 'learning climate' (P<0.001). Faculty most likely to be seen as good role models were those rated highly on 'feedback' (odds ratio 2.91, 95% CI: 2.41-3.51), 'a professional attitude towards residents' (OR 2.70, 95% CI: 2.34-3.10) and 'creating a positive learning climate' (OR 2.45, 95% CI: 1.97-3.04). Results did not seem to vary much across residency years. The relative strength of associations between teaching qualities and being seen as a role model were more distinct when comparing specialties.Good clinical educators are more likely to be seen as specialist role models for most residents

Involvement of patients or their representatives in quality management functions in EU hospitals:implementation and impact on patient-centred care strategies

OBJECTIVE: The objective of this study was to describe the involvement of patients or their representatives in quality management (QM) functions and to assess associations between levels of involvement and the implementation of patient-centred care strategies. DESIGN: A cross-sectional, multilevel STUDY DESIGN: that surveyed quality managers and department heads and data from an organizational audit. SETTING: Randomly selected hospitals (n = 74) from seven European countries (The Czech Republic, France, Germany, Poland, Portugal, Spain and Turkey). PARTICIPANTS: Hospital quality managers (n = 74) and heads of clinical departments (n = 262) in charge of four patient pathways (acute myocardial infarction, stroke, hip fracture and deliveries) participated in the data collection between May 2011 and February 2012. MAIN OUTCOME MEASURES: Four items reflecting essential patient-centred care strategies based on an on-site hospital visit: (1) formal survey seeking views of patients and carers, (2) written policies on patients' rights, (3) patient information literature including guidelines and (4) fact sheets for post-discharge care. The main predictors were patient involvement in QM at the (i) hospital level and (ii) pathway level. RESULTS: Current levels of involving patients and their representatives in QM functions in European hospitals are low at hospital level (mean score 1.6 on a scale of 0 to 5, SD 0.7), but even lower at departmental level (mean 0.6, SD 0.7). We did not detect associations between levels of involving patients and their representatives in QM functions and the implementation of patient-centred care strategies; however, the smallest hospitals were more likely to have implemented patient-centred care strategies. CONCLUSIONS: There is insufficient evidence that involving patients and their representatives in QM leads to establishing or implementing strategies and procedures that facilitate patient-centred care; however, lack of evidence should not be interpreted as evidence of no effect

Is the maturity of hospitals' quality improvement systems associated with measures of quality and patient safety?

UNLABELLED: ABSTRACT: BACKGROUND: Previous research addressed the development of a classification scheme for quality improvement systems in European hospitals. In this study we explore associations between the 'maturity' of the hospitals' quality improvement system and clinical outcomes. METHODS: The maturity classification scheme was developed based on survey results from 389 hospitals in eight European countries. We matched the hospitals from the Spanish sample (113 hospitals) with those hospitals participating in a nation-wide, voluntary hospital performance initiative. We then compared sample distributions and explored associations between the 'maturity' of the hospitals' quality improvement system and a range of composite outcomes measures, such as adjusted hospital-wide mortality, -readmission, -complication and -length of stay indices. Statistical analysis includes bivariate correlations for parametrically and non-parametrically distributed data, multiple robust regression models and bootstrapping techniques to obtain confidence-intervals for the correlation and regression estimates. RESULTS: Overall, 43 hospitals were included. Compared to the original sample of 113, this sample was characterized by a higher representation of university hospitals. Maturity of the quality improvement system was similar, although the matched sample showed less variability. Analysis of associations between the quality improvement system and hospital-wide outcomes suggests significant correlations for the indicator adjusted hospital complications, borderline significance for adjusted hospital readmissions and non-significance for the adjusted hospital mortality and length of stay indicators. These results are confirmed by the bootstrap estimates of the robust regression model after adjusting for hospital characteristics. CONCLUSIONS: We assessed associations between hospitals' quality improvement systems and clinical outcomes. From this data it seems that having a more developed quality improvement system is associated with lower rates of adjusted hospital complications. A number of methodological and logistic hurdles remain to link hospital quality improvement systems to outcomes. Further research should aim at identifying the latent dimensions of quality improvement systems that predict quality and safety outcomes. Such research would add pertinent knowledge regarding the implementation of organizational strategies related with quality of care outcomes

Development and validation of an index to assess hospital quality management systems

The study, "Deepening our Understanding of Quality Improvement in Europe (DUQuE)" has received funding from the European Community's Seventh Framework Programme (FP7/2007-2013) under grant agreement n° 241822. Funding to pay the Open Access publication charges for this article was provided by European Community's Seventh Framework Programme (FP7/2007-2013) under grant agreement no. 241822.Objective: The aim of this study was to develop and validate an index to assess the implementation of quality management systems (QMSs) in European countries. Design: Questionnaire development was facilitated through expert opinion, literature review and earlier empirical research. A cross-sectional online survey utilizing the questionnaire was undertaken between May 2011 and February 2012. We used psychometric methods to explore the factor structure, reliability and validity of the instrument. Setting and participants. As part of the Deepening our Understanding of Quality improvement in Europe (DUQuE) project, we invited a random sample of 188 hospitals in 7 countries. The quality managers of these hospitals were the main respondents. Main Outcome Measure. The extent of implementation of QMSs. Results: Factor analysis yielded nine scales, which were combined to build the Quality Management Systems Index. Cronbach's reliability coefficients were satisfactory (ranging from 0.72 to 0.82) for eight scales and low for one scale (0.48). Corrected item-total correlations provided adequate evidence of factor homogeneity. Inter-scale correlations showed that every factor was related, but also distinct, and added to the index. Construct validity testing showed that the index was related to recent measures of quality. Participating hospitals attained a mean value of 19.7 (standard deviation of 4.7) on the index that theoretically ranged from 0 to 27. Conclusion: Assessing QMSs across Europe has the potential to help policy-makers and other stakeholders to compare hospitals and focus on the most important areas for improvement

Involvement of patients or their representatives in quality management functions in EU hospitals : Implementation and impact on patient-centred care strategies

The study, "Deepening our Understanding of Quality Improvement in Europe (DUQuE)" has received funding from the European Community's Seventh Framework Programme (FP7/2007-2013) under grant agreement no 241822. Funding to pay the Open Access publication charges for this article was provided by European Community's Seventh Framework Programme (FP7/2007-2013) under grant agreement no 241822.Objective: The objective of this study was to describe the involvement of patients or their representatives in quality management (QM) functions and to assess associations between levels of involvement and the implementation of patient-centred care strategies. Design: A cross-sectional, multilevel study design that surveyed quality managers and department heads and data from an organizational audit. Setting: Randomly selected hospitals (n = 74) from seven European countries (The Czech Republic, France, Germany, Poland, Portugal, Spain and Turkey). Participants: Hospital qualitymanagers (n = 74) and heads of clinical departments (n = 262) in charge of four patient pathways (acute myocardial infarction, stroke, hip fracture and deliveries) participated in the data collection between May 2011 and February 2012. Main Outcome Measures: Four items reflecting essential patient-centred care strategies based on an on-site hospital visit: (1) formal survey seeking views of patients and carers, (2) written policies on patients' rights, (3) patient information literature including guidelines and (4) fact sheets for post-discharge care. The main predictors were patient involvement in QM at the (i) hospital level and (ii) pathway level. Results: Current levels of involving patients and their representatives in QM functions in European hospitals are low at hospital level (mean score 1.6 on a scale of 0 to 5, SD 0.7), but even lower at departmental level (mean 0.6, SD 0.7). We did not detect associations between levels of involving patients and their representatives in QM functions and the implementation of patient-centred care strategies; however, the smallest hospitals were more likely to have implemented patient-centred care strategies. Conclusions: There is insufficient evidence that involving patients and their representatives in QM leads to establishing or implementing strategies and procedures that facilitate patient-centred care; however, lack of evidence should not be interpreted as evidence of no effect