EFFECTS OF ITEM-LEVEL FEEDBACK ON THE RATINGS PROVIDED BY JUDGES IN A MODIFIED-ANGOFF STANDARD SETTING STUDY

Setting performance standards is a judgmental process involving human opinions and values as well as technical and empirical considerations and although all cut score decisions are by nature arbitrary, they should not be capricious. Establishing a minimum passing standard is the technical expression of a policy decision and the information gained through standard setting studies inform these policy decisions. To this end, it is necessary to conduct robust examinations of methods and techniques commonly applied to standard setting studies in order to better understand issues that may influence policy decisions. The modified-Angoff method remains one of the most popular methods for setting performance standards in testing and assessment. With this method, is common practice to provide content experts with feedback regarding the item difficulties; however, it is unclear how this feedback affects the ratings and recommendations of content experts. Recent research seems to indicate mixed results, noting that the feedback given to raters may or may not alter their judgments depending on the type of data provided, when the data was provided, and how raters collaborated within groups and between groups. This research seeks to examine issues related to the effects of item-level feedback on the judgment of raters. The results suggest that the most important factor related to item-level feedback is whether or not a Subject Matter Expert (SME) was able to correctly answer a question. If so, then the SMEs tended to rely on their own inherent sense of item difficulty rather than the data provided, in spite of empirical evidence to the contrary. The results of this research may hold implications for how standard setting studies are conducted with regard to the difficulty and ordering of items, the ability level of content experts invited to participate in these studies, and the types of feedback provided

Prevalence and Practice for Rare Diseases in Primary Care: A National Cross-Sectional Study in the USA

Objectives There are more than 7000 rare diseases in the USA, and they are prevalent in 8% of the population. Due to life-threatening risk and limited therapies, early detection and treatment are critical. The purpose of this study was to explore characteristics of visits for patients with rare diseases seen by primary care physicians (PCPs). Design The study used a cross sectional study using a national representative dataset, the National Ambulatory Medical Care Survey for the years 2012–2014. Setting Primary care setting. Participants Visits to PCPs (n=22 306 representing 354 507 772 office visits to PCPs). Primary outcome measures Prevalence of rare diseases in visits of PCPs was the primary outcome. Bivariate analyses and logistic regression analyses were used to compare patients with rare diseases and those without rare diseases and examined characteristics of PCP visits for rare diseases and practice pattern. Results Among outpatient visits to PCPs, rare diseases account for 1.6% of the visits. The majority of patients with rare diseases were established patients (93.0%) and almost half (49.0%) were enrolled in public insurance programmes. The time spent in visits for rare diseases (22.4 min) and visits for more common diseases (21.3 min) was not significantly different (p=0.09). In an adjusted model controlling for patient characteristics (age, sex, types of insurance, reason for this visit, total number of chronic disease, having a rare disease and established or new patient), patients with rare diseases were 52% more likely to be referred to another provider (OR 1.52, 95% CI, 1.01 to 2.28). Conclusions Visits for rare diseases are uncommon in primary care practice. Future research may help to explain whether this low level of management of rare diseases in primary care practice is consistent with a goal of a broad scope of care

Establishment of a Web-based System for Collection of Patient-reported Outcomes After Radical Prostatectomy in a Statewide Quality Improvement Collaborative.

OBJECTIVE: To report on the establishment of a unified, electronic patient-reported outcome (PRO) infrastructure and pilot results from the first 5 practices enrolled in the web-based collection system developed by the Michigan Urological Surgery Improvement Collaborative. MATERIALS AND METHODS: Eligible patients were those undergoing radical prostatectomy of 5 academic and community practices. PRO was obtained using a validated 21-item web-based questionnaire, regarding urinary function, erection function, and sexual interest and satisfaction. Data were collected preoperatively, at 3 months, and 6 months postoperatively. Patients were provided a link via email to complete the surveys. Perioperative and PRO data were analyzed as reports for individual patients and summary performance reports for individual surgeons. RESULTS: Among 773 eligible patients, 688 (89%) were enrolled preoperatively. Survey completion rate was 88%, 84%, and 90% preoperatively, at 3 months, and 6 months. Electronic completion rates preoperatively, at 3 months, and 6 months were 70%, 70%, and 68%, respectively. Mean urinary function scores were 18.3, 14.3, and 16.6 (good function ≥ 17), whereas mean erection scores were 18.7, 7.3, and 9.1 (good erection score ≥ 22) before surgery, at 3 months, and 6 months. Variation was noted for erectile function among the practices. CONCLUSION: Collection of electronic PRO via this unified, web-based format was successful and provided results that reflect expected recovery and identify opportunities for improvement. This will be extended to more practices statewide to improve outcomes after radical prostatectomy

Pelvic lymph node dissection at robot-assisted radical prostatectomy: Assessing utilization and nodal metastases within a statewide quality improvement consortium.

PURPOSE: Several guidelines recommend pelvic lymph node dissection (PLND) at robot-assisted radical prostatectomy (RARP) only when lymph node involvement (LN+) is \u3e2%. Individual surgeon use of PLND is not well-known. We sought to examine variability in PLND performance and detection of LN+ across the Michigan Urological Surgery Improvement Collaborative. METHODS: Data regarding all RARP (3/2012-9/2018) were prospectively collected, including patient and surgeon characteristics. Univariable and multivariable analyses of PLND rate and LN+ rate were performed. RESULTS: Among 9,751 men undergoing RARP, 79.8% had PLND performed (n = 7,781), of which 5.2% were LN+ (n = 404). In univariate and multivariable analyses, predictors of PLND included higher Prostate-Specific Antigen (PSA), biopsy Gleason grade (bGG), number of positive cores, and maximum core involvement at P \u3c 0.05 for each. Higher PSA, cT stage, bGG, number of positive cores, and maximum core involvement predicted LN+ when PLND was performed (P \u3c 0.05 for each). There was significant surgeon variation in the proportion of PLND performed at RARP, yet neither surgeon-annualized RARP volume nor % of PLND performed was associated with LN+ disease (P \u3e 0.05). Grade was associated with PLND (60.0%, 77.6%, 91.0%, 97.3%, and 98.5%; P \u3c 0.001) and LN+ (0.7%, 2.5%, 5.8%, 8.6%, and 19.9%; P \u3c 0.001) for bGG 1,2,3,4,5, respectively. Maximum core involvement also strongly predicted LN+ with rates of 1.5%, 3.8%, and 9.4% for65%, respectively (P \u3c 0.001). CONCLUSIONS: Nearly 80% of RARP in Michigan Urological Surgery Improvement Collaborative were performed with PLND, including 60% of bGG1 patients (with LN+ in only 0.7%), but significant variability exists between surgeons. Our data indicate limited benefit for favorable-risk CaP patients and support efforts to decrease PLND use going forward

A Statewide Quality Improvement Collaborative\u27s Adherence to the 2017 American Urological Association Guidelines Regarding Initial Evaluation of Patients With Clinical T1 Renal Masses

OBJECTIVE: To evaluate MUSIC-KIDNEY\u27s adherence to the American Urological Association (AUA) guidelines regarding the initial evaluation of patient\u27s with clinical T1 (cT1) renal masses. METHODS: We reviewed MUSIC-KIDNEY registry data for patients with newly diagnosed cT1 renal masses to assess for adherence with the 2017 AUA guideline statements regarding recommendations to obtain (1) CMP, (2) CBC, (3) UA, (4) abdominal cross-sectional imaging, and (5) chest imaging. An evaluation consisting of all 5 guideline measures was considered complete compliance. Variation with guideline adherence was assessed by contributing practice, management strategy, and renal mass size. RESULTS: We identified 1808 patients with cT1 renal masses in the MUSIC-KIDNEY registry, of which 30% met the definition of complete compliance. Most patients received care that was compliant with recommendations to obtain laboratory testing with 1448 (80%), 1545 (85%), and 1472 (81%) patients obtaining a CMP, CBC, and UA respectively. Only 862 (48%) patients underwent chest imaging. Significant variation exists in complete guideline compliance for contributing practices, ranging from 0% to 45% as well as for patients which underwent immediate intervention compared with initial observation (37% vs 23%) and patients with cT1b masses compared with cT1a masses (36% vs 28%). CONCLUSION: Complete guideline compliance in the initial evaluation of patients with cT1 renal masses is poor, which is mainly driven by omission of chest imaging. Significant variation in guideline adherence is seen across practices, as well as patients undergoing an intervention vs observation, and cT1a vs cT1b masses. There are ample quality improvement opportunities to increase adherence and decrease variability with guideline recommendations