Using and Reporting the Delphi Method for Selecting Healthcare Quality Indicators: A Systematic Review

OBJECTIVE: Delphi technique is a structured process commonly used to developed healthcare quality indicators, but there is a little recommendation for researchers who wish to use it. This study aimed 1) to describe reporting of the Delphi method to develop quality indicators, 2) to discuss specific methodological skills for quality indicators selection 3) to give guidance about this practice. METHODOLOGY AND MAIN FINDING: Three electronic data bases were searched over a 30 years period (1978-2009). All articles that used the Delphi method to select quality indicators were identified. A standardized data extraction form was developed. Four domains (questionnaire preparation, expert panel, progress of the survey and Delphi results) were assessed. Of 80 included studies, quality of reporting varied significantly between items (9% for year's number of experience of the experts to 98% for the type of Delphi used). Reporting of methodological aspects needed to evaluate the reliability of the survey was insufficient: only 39% (31/80) of studies reported response rates for all rounds, 60% (48/80) that feedback was given between rounds, 77% (62/80) the method used to achieve consensus and 57% (48/80) listed quality indicators selected at the end of the survey. A modified Delphi procedure was used in 49/78 (63%) with a physical meeting of the panel members, usually between Delphi rounds. Median number of panel members was 17(Q1:11; Q3:31). In 40/70 (57%) studies, the panel included multiple stakeholders, who were healthcare professionals in 95% (38/40) of cases. Among 75 studies describing criteria to select quality indicators, 28 (37%) used validity and 17(23%) feasibility. CONCLUSION: The use and reporting of the Delphi method for quality indicators selection need to be improved. We provide some guidance to the investigators to improve the using and reporting of the method in future surveys

Expert Systems for Forecasting

Expert systems use rules to represent experts’ reasoning in solving problems. The rules are based on knowledge about methods and the problem domain. To acquire knowledge for an expert system, one should rely on a variety of sources, such as textbooks, research papers, interviews, surveys, and protocol analysis. Protocol analysis is especially useful if the area to be modeled is complex or if experts lack an awareness of their processes. Expert systems should be easy to use, incorporate the best available knowledge, and reveal the reasoning behind the recommendations they make. In forecasting, the most promising applications of expert systems are to replace unaided judgment in cases requiring many forecasts, to model complex problems where data on the dependent variable are of poor quality, and to handle semi-structured problems. We found 15 comparisons of forecast validity involving expert systems. As expected, expert systems were more accurate than unaided judgment, six comparisons to one, with one tie. Expert systems were less accurate than judgmental bootstrapping in two comparisons with two ties. There was little evidence with which to compare expert systems and econometric models; expert systems were better in one study and tied in two

Designing a participatory process for stakeholder involvement in a societal decision

For many societal decisions, governments and public bodies are beginning to involve stakeholders and the general public to a far greater extent than previously in the decision process. Stakeholder workshops, citizen juries, focus groups, electronic forums, web-polling and many other means of consultation are being used to draw citizens into the process of deciding between different options on the management of their communities. Politicians are drawn to such instruments because greater public involvement seems to achieve greater acceptance of the ultimate decision and, arguably in more objective terms, a better decision. Many academic studies have investigated participation and wider aspects of deliberative democracy and found that the politicians’ intuition is borne out in practice. However, while there have been many studies focused on specific instruments of participation, few have compared different ones. Moreover, there seems to be a dearth of advice on how to assemble a set of different instruments into a complete participatory decision making process. This paper offers a decision modelling framework which, firstly, provides a methodology which may be used to design participatory processes and, secondly, raises a number of questions which future comparative studies will need to address