Distill-and-Compare: Auditing Black-Box Models Using Transparent Model Distillation

Black-box risk scoring models permeate our lives, yet are typically proprietary or opaque. We propose Distill-and-Compare, a model distillation and comparison approach to audit such models. To gain insight into black-box models, we treat them as teachers, training transparent student models to mimic the risk scores assigned by black-box models. We compare the student model trained with distillation to a second un-distilled transparent model trained on ground-truth outcomes, and use differences between the two models to gain insight into the black-box model. Our approach can be applied in a realistic setting, without probing the black-box model API. We demonstrate the approach on four public data sets: COMPAS, Stop-and-Frisk, Chicago Police, and Lending Club. We also propose a statistical test to determine if a data set is missing key features used to train the black-box model. Our test finds that the ProPublica data is likely missing key feature(s) used in COMPAS.Comment: Camera-ready version for AAAI/ACM AIES 2018. Data and pseudocode at https://github.com/shftan/auditblackbox. Previously titled "Detecting Bias in Black-Box Models Using Transparent Model Distillation". A short version was presented at NIPS 2017 Symposium on Interpretable Machine Learnin

Quadriceps Tendon Repair Using Double-Row Suture Anchor Fixation

Quadriceps tendon ruptures compromise the knee extensor mechanism and cause an inability to ambulate and significant functional limitations. Therefore, the vast majority of quadriceps tendon ruptures are indicated for operative intervention to restore patient mobility and function. Although these injuries were traditionally repaired using a transosseous repair technique, recent literature has shown that suture anchor repair may offer biomechanical advantages. Additionally, research in other areas of orthopaedics has found that a double-row suture anchor construct can offer additional biomechanical strength to tendinous repair. This technical note describes a safe and effective quadriceps tendon repair using a double-row suture anchor construct

Preoperative PROMIS Scores Predict Postoperative Improvements Following Rotator Cuff Repair

Background: The Patient-Reported Outcomes Measurement Information System (PROMIS) has emerged as a valid and efficient means of collecting patient outcomes in patients with rotator cuff tear. The purpose of this study was to examine the role of pre-operative PROMIS computer adaptive test (CAT) scores in predicting post-operative PROMIS CAT scores as well as likelihood of achieving minimal clinically important difference (MCID) following rotator cuff repair. We hypothesize that pre-operative PROMIS CAT scores will directly impact both post-operative PROMIS CAT scores and likelihood of achieving MCID.Methods: Patients undergoing arthroscopic rotator cuff repair by one of three fellowship-trained surgeons were identified over a 12-month period. Only patients that completed pre-operative and 6-month post-operative PROMIS CAT assessments were included in this cohort. PROMIS CAT forms for upper extremity physical function (PROMIS-U), pain interference (PROMIS-PI), and depression (PROMIS-D) were utilized. MCID was calculated according to the distribution methodology, and receiver operating characteristics (ROC) were utilized to determine if pre-operative scores were predictive of post-operative outcomes. Preoperative cutoffs were used to predict which patients would likely meet MCID using 95% specificity. Results: A total of 80 patients met our inclusion criteria. PROMIS-UE, PROMIS-PI and PROMIS-D improved 6 months after surgery (p\u3c0.001). 76% of patients met MCID for PROMIS-UE, while 89% met MCID for PROMIS-PI, and 54% met MCID for PROMIS-D. Preoperative PROMIS scores were predictive of post-operative outcomes based on ROC analysis which demonstrated significant area under the curve (AUC) of .725 (p=0.003), .757 (p=0.013), and .789 (p\u3c0.001) for PROMIS-UE, PROMIS-PI, and PROMIS-D, respectively. Individuals with PROMIS-UE scores below 24.95 and PROMIS-PI scores above 65.65 yielded a 100% probability of achieving MCID, while a cutoff of 56.45 for PROMIS-D yielded a 91% probability of achieving MCID with 95% specificity. Conclusion: Patients undergoing arthroscopic rotator cuff repair experience significant improvements in upper extremity physical function, pain interference and depression as measured by PROMIS CAT domains. In particular, patients presenting with PROMIS upper extremity scores of \u3c24.95 are especially likely to achieve MCID.https://scholarlycommons.henryford.com/merf2019clinres/1053/thumbnail.jp

LiFT: A Scalable Framework for Measuring Fairness in ML Applications

Many internet applications are powered by machine learned models, which are usually trained on labeled datasets obtained through either implicit / explicit user feedback signals or human judgments. Since societal biases may be present in the generation of such datasets, it is possible for the trained models to be biased, thereby resulting in potential discrimination and harms for disadvantaged groups. Motivated by the need for understanding and addressing algorithmic bias in web-scale ML systems and the limitations of existing fairness toolkits, we present the LinkedIn Fairness Toolkit (LiFT), a framework for scalable computation of fairness metrics as part of large ML systems. We highlight the key requirements in deployed settings, and present the design of our fairness measurement system. We discuss the challenges encountered in incorporating fairness tools in practice and the lessons learned during deployment at LinkedIn. Finally, we provide open problems based on practical experience.Comment: Accepted for publication in CIKM 202

CT colonography with minimal bowel preparation: evaluation of tagging quality, patient acceptance and diagnostic accuracy in two iodine-based preparation schemes

PURPOSE: The aim of this study was to compare a 1-day with a 2-day iodine bowel preparation for CT colonography in a positive faecal occult blood test (FOBT) screening population. MATERIALS AND METHODS: One hundred consecutive patients underwent CT colonography and colonoscopy with segmental unblinding. The first 50 patients (group 1) ingested 7 50 ml iodinated contrast starting 2 days before CT colonography. The latter 50 patients (group 2) ingested 4 50 ml iodinated contrast starting 1 day before CT colonography. Per colonic segment measurements of residual stool attenuation and homogeneity were performed, and a subjective evaluation of tagging quality (grade 1-5) was done. Independently, two reviewers performed polyp and carcinoma detection. RESULTS: The tagging density was 638 and 618 HU (p = 0.458) and homogeneity 91 and 86 HU for groups 1 and 2, respectively (p = 0.145). The tagging quality was graded 5 (excellent) in 90% of all segments in group 1 and 91% in group 2 (p = 0.749). Mean per-polyp sensitivity for lesions >or=10 mm was 86% in group 1 and 97% in group 2 (p = 0.355). Patient burden from diarrhoea significantly decreased for patients in group 2. CONCLUSIONS: One-day preparation with meglumine ioxithalamate results in an improved patient acceptability compared with 2-day preparation and has a comparable, excellent image quality and good diagnostic performanc

Topical NSAIDs for chronic musculoskeletal pain: systematic review and meta-analysis

A previous systematic review reported that topical NSAIDs were effective in relieving pain in chronic conditions like osteoarthritis and tendinitis. More trials, a better understanding of trial quality and bias, and a reclassification of certain drugs necessitate a new review. Studies were identified by searching electronic databases, and writing to manufacturers. We identified randomised, double blind trials comparing topical NSAID with either placebo or another active treatment, in adults with chronic pain. The primary outcome was a reduction in pain of approximately 50% at two weeks, and secondary outcomes were local and systemic adverse events and adverse event-related withdrawals. Relative benefit and number-needed-to-treat (NNT), and relative harm and number-needed-to-harm (NNH) were calculated, and the effects of trial quality, validity and size, outcome reported, and condition treated, were examined by sensitivity analyses. Twelve new trials were added to 13 trials from a previous review. Fourteen double blind placebo-controlled trials had information from almost 1,500 patients. Topical NSAID was significantly better than placebo with relative benefit 1.9 (95% confidence interval 1.7 to 2.2), NNT 4.6 (95% confidence interval 3.8 to 5.9). Results were not affected by trial quality, validity or size, outcome reported, or condition treated. Three trials with 764 patients comparing a topical with an oral NSAID found no difference in efficacy. Local adverse events (6%), systemic adverse events (3%), or the numbers withdrawing due to an adverse event were the same for topical NSAID and placebo. Topical NSAIDs were effective and safe in treating chronic musculoskeletal conditions for two weeks. Larger and longer trials are necessary to fully elucidate the place of topical NSAIDs in clinical practice

Topical NSAIDs for acute pain: a meta-analysis

BACKGROUND: A previous systematic review reported that topical NSAIDs were effective in relieving pain in acute conditions like sprains and strains, with differences between individual drugs for efficacy. More trials, a better understanding of trial quality and bias, and a reclassification of certain drugs necessitate a new review. METHODS: Studies were identified by searching electronic databases and writing to manufacturers. We selected randomised double blind trials comparing topical NSAID with either placebo or another active treatment in adults with acute pain, and extracted dichotomous information approximating to a 50% reduction in pain at one week, together with details of adverse events and withdrawals. Relative benefit and number-needed-to-treat (NNT), and relative risk and number-needed-to-harm (NNH) were calculated, with sensitivity analyses where appropriate to investigate differences between individual drugs and aspects of trial design. RESULTS: Twenty-six double blind placebo controlled trials had information from 2,853 patients for evaluation of efficacy. Topical NSAID was significantly better than placebo in 19 of the 26 trials, with a pooled relative benefit of 1.6 (95% confidence interval 1.4 to 1.7), and NNT of 3.8 (95% confidence interval 3.4 to 4.4) compared with placebo for the outcome of half pain relief at seven days. Results were not affected by outcome reported, or condition treated, but smaller trials yielded a larger estimate of efficacy. Indirect comparisons of individual topical NSAIDs showed that ketoprofen was significantly better than all other topical NSAIDs, while indomethacin was barely distinguished from placebo. Three trials, with 433 patients, compared topical with oral NSAID (two trials compared the same drug, one compared different drugs) and found no difference in efficacy. Local adverse events, systemic adverse events, or withdrawals due to an adverse event were rare, and no different between topical NSAID and placebo. CONCLUSIONS: Topical NSAIDs were effective and safe in treating acute painful conditions for one week

Systematic review of dexketoprofen in acute and chronic pain

which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Background: Dexketoprofen, an NSAID used in the management of acute and chronic pains, is licensed in several countries but has not previously been the subjected of a systematic review. We used published and unpublished information from randomised clinical trials (RCTs) of dexketoprofen in painful conditions to assess evidence on efficacy and harm. Methods: PubMed and Cochrane Central were searched for RCTs of dexketoprofen for pain of any aetiology. Reference lists of retrieved articles and reviews were also searched. Menarini Group produced copies of published and unpublished studies (clinical trial reports). Data were abstracted into a standard form. For studies reporting results of single dose administration, the number of patients with at least 50 % pain relief was derived and used to calculate the relative benefit (RB) and number-needed-to-treat (NNT) for one patient to achieve at least 50 % pain relief compared with placebo. Results: Thirty-five trials were found in acute pain and chronic pain; 6,380 patients were included, 3,381 receiving dexketoprofen. Information from 16 trials (almost half the total patients) wa