Reliability and validity of the AGREE instrument used by physical therapists in assessment of clinical practice guidelines

BACKGROUND: The AGREE instrument has been validated for evaluating Clinical Practice Guidelines (CPG) pertaining to medical care. This study evaluated the reliability and validity of physical therapists using the AGREE to assess quality of CPGs relevant to physical therapy practice. METHODS: A total of 69 physical therapists participated and were classified as generalists, specialist or researchers. Pairs of appraisers within each category evaluated independently, a set of 6 CPG selected at random from a pool of 55 CPGs. RESULTS: Reliability between pairs of appraisers indicated low to high reliability depending on the domain and number of appraisers (0.17–0.81 for single appraiser; 0.30–0.96 when score averaged across a pair of appraisers). The highest reliability was achieved for Rigour of Development, which exceeded ICC> 0.79, if scores from pairs of appraisers were pooled. Adding more than 3 appraisers did not consistently improve reliability. Appraiser type did not determine reliability scores. End-users, including study participants and a separate sample of 102 physical therapy students, found the AGREE useful to guide critical appraisal. The construct validity of the AGREE was supported in that expected differences on Rigour of Development domains were observed between expert panels versus those with no/uncertain expertise (differences of 10–21% p = 0.09–0.001). Factor analysis with varimax rotation, produced a 4-factor solution that was similar, although not in exact agreement with the AGREE Domains. Validity was also supported by the correlation observed (Kendall-tao = 0.69) between Overall Assessment and the Rigour of Development domain. CONCLUSION: These findings suggest that the AGREE instrument is reliable and valid when used by physiotherapists to assess the quality of CPG pertaining to physical therapy health services

Development and Validation of a Pain Competence Assessment Tool (PCAT) Based on the IASP Core Competencies for Pain Management

Several interprofessional education programs have been developed to enhance primary care providers’ (PCPs) competency in pain management. This project aims to develop a valid and reliable competency-based assessment tool, suitable for all professions, to evaluate the impact of these programs. To develop a new pain competence assessment tool (PCAT), six clinical vignettes were created. Each vignette is followed by questions addressing one or more competencies derived from the core pain competencies developed by Fishman et al. in 2013. Experts in pain management representing different professions were asked to review and rate the appropriateness of the PCAT questions to confirm face validity through a modified Delphi study. The selected PCAT questions were mapped against the core competencies to ensure content validity. A pilot sample of PCPs was then asked to answer the PCAT through a cognitive interview study to assess feasibility. Currently, the PCAT was sent to a large sample of PCPs to assess its reliability through a cross-sectional study. The PCAT was also sent to healthcare prelicensure students to test for construct validity. Based on the Delphi results, nineteen questions were rated as appropriate confirming its face and content validity. The PCAT also showed adequate feasibility based on the cognitive interview study. Psychometric testing is still in process. Future implementation of a valid and reliable tool is necessary to reduce measurement error and produce results with high degree of credibility. Introduction/Aim: Several interprofessional education programs have been developed to enhance primary care providers’ (PCPs) competency in pain management. This project aims to develop a valid and reliable competency-based assessment tool, suitable for all professions, to evaluate the impact of these programs. Methods: To develop a new pain competence assessment tool (PCAT), six clinical vignettes were created. Each vignette is followed by questions addressing one or more competencies derived from the core pain competencies developed by Fishman et al. in 2013. Experts in pain management representing different professions were asked to review and rate the appropriateness of the PCAT questions to confirm face validity through a modified Delphi study. The selected PCAT questions were mapped against the core competencies to ensure content validity. A pilot sample of PCPs was then asked to answer the PCAT through a cognitive interview study to assess feasibility. Currently, the PCAT was sent to a large sample of PCPs to assess its reliability through a cross-sectional study. The PCAT was also sent to healthcare prelicensure students to test for construct validity. Results: Based on the Delphi results, nineteen questions were rated as appropriate confirming its face and content validity. The PCAT also showed adequate feasibility based on the cognitive interview study. Psychometric testing is still in process. Discussion/Conclusions: Future implementation of a valid and reliable tool is necessary to reduce measurement error and produce results with high degree of credibility