Distill-and-Compare: Auditing Black-Box Models Using Transparent Model Distillation

Black-box risk scoring models permeate our lives, yet are typically proprietary or opaque. We propose Distill-and-Compare, a model distillation and comparison approach to audit such models. To gain insight into black-box models, we treat them as teachers, training transparent student models to mimic the risk scores assigned by black-box models. We compare the student model trained with distillation to a second un-distilled transparent model trained on ground-truth outcomes, and use differences between the two models to gain insight into the black-box model. Our approach can be applied in a realistic setting, without probing the black-box model API. We demonstrate the approach on four public data sets: COMPAS, Stop-and-Frisk, Chicago Police, and Lending Club. We also propose a statistical test to determine if a data set is missing key features used to train the black-box model. Our test finds that the ProPublica data is likely missing key feature(s) used in COMPAS.Comment: Camera-ready version for AAAI/ACM AIES 2018. Data and pseudocode at https://github.com/shftan/auditblackbox. Previously titled "Detecting Bias in Black-Box Models Using Transparent Model Distillation". A short version was presented at NIPS 2017 Symposium on Interpretable Machine Learnin

Evaluating undergraduate, laboratory-based learning experience in pharmacology

The distinctive role of laboratory-based teaching in science education is well recognised but whether the expected benefits of this style of teaching on student learning is realised is dependent upon multiple factors. Learning is a complex process, affected by different styles of learning and the educational learning space in which it occurs. For laboratory work to enhance student learning, laboratory experience needs to be positive. To evaluate students’ experience of laboratory-based learning in an undergraduate pharmacology course, we have used survey instruments developed and supported by ASELL - Advancing Science by Enhancing Laboratory Learning (http://www.asell.org/) with a view to improving student learning outcomes. At the University of Sydney, pharmacology is taught primarily to students enrolled in either a Bachelor of Science (BSc) or Bachelor of Medical Science (BMedSci) degree program. The unit of study “Pharmacology: Drugs and People” (PCOL2012) is offered in 2nd year, 2nd semester. Face-to-face teaching consists of 26 lectures, six workshops and four laboratories (two wet-labs and two computer-based, dry-labs). In the present study, two survey instruments were used to evaluate students’ laboratory experience. The first, Laboratory Program Evaluation was used to evaluate the students’ overall experience of laboratory teaching within PCOL2012 and the second, Student Evaluation of an Experiment was used to evaluate students’ experience of a specific wet-lab experiment entitled: “The effects of drugs on peristalsis in the guinea-pig ileum in vitro”. This experiment illustrates how drugs can be used to unravel physiological mechanisms controlling gut movements. Students are required to do pre- and post-lab work (creating a flow chart of experimental procedures, experimental data analyses and report writing). The surveys consisted of 14 closed questions and five (survey one) or four (survey two) open-ended questions. In each survey, the final question was: “Overall, as a learning experience, I would rate the experiment/these labs as ....” For PCOL2012, only 37% of students rated the overall laboratory experience as good or better. In contrast, the experiment “Drugs and Peristalsis” was rated by 65% of students as good or better. In reviewing comments, one criticism noted about the second wet-lab in PCOL2012 (entitled “Cholinesterase and Inhibitors”) was the use of a semi-quantitative colormetric assay to determine the hydrolysis rate of substrates by acetyl- and butyrylcholinesterase. To address this issue, we have revised the experiment for 2014. An ultra-fast, scanning absorbance microplate reader (SPECTROstar Nano, BMG LABTECH) will be used to measure, and display, the rate of hydrolysis in each of 48 wells. Additional changes will include holding the wet-lab in the recently opened “super-lab” (X-Lab, Charles Perkins Centre, The University of Sydney), with state-of-the art ICT support, and using LabTutor, ADInstruments (http://www.adinstruments.com/products/labtutor) for pre-lab work and to replace hard copy manuals. The revised wet-lab will be evaluated using the second of the ASELL survey instruments (see above). Details of ratings and comments will be reported in the presentation

The relationship between uncertainty tolerance and oncologists’ perceptions of large-panel genomic tumor testing

Introduction: Large-panel genomic tumor testing (GTT) is a new technology that promises to make cancer treatment more precise, but that currently poses many uncertainties regarding its clinical value and appropriate use. Uncertainty Tolerance (UT), a psychological construct that describes trait-level differences in individuals’ responses to uncertainty, may influence oncologists’ perceptions and attitudes regarding GTT

Deep Sequencing Analysis of Small Noncoding RNA and mRNA Targets of the Global Post-Transcriptional Regulator, Hfq

Recent advances in high-throughput pyrosequencing (HTPS) technology now allow a thorough analysis of RNA bound to cellular proteins, and, therefore, of post-transcriptional regulons. We used HTPS to discover the Salmonella RNAs that are targeted by the common bacterial Sm-like protein, Hfq. Initial transcriptomic analysis revealed that Hfq controls the expression of almost a fifth of all Salmonella genes, including several horizontally acquired pathogenicity islands (SPI-1, -2, -4, -5), two sigma factor regulons, and the flagellar gene cascade. Subsequent HTPS analysis of 350,000 cDNAs, derived from RNA co-immunoprecipitation (coIP) with epitope-tagged Hfq or control coIP, identified 727 mRNAs that are Hfq-bound in vivo. The cDNA analysis discovered new, small noncoding RNAs (sRNAs) and more than doubled the number of sRNAs known to be expressed in Salmonella to 64; about half of these are associated with Hfq. Our analysis explained aspects of the pleiotropic effects of Hfq loss-of-function. Specifically, we found that the mRNAs of hilD (master regulator of the SPI-1 invasion genes) and flhDC (flagellar master regulator) were bound by Hfq. We predicted that defective SPI-1 secretion and flagellar phenotypes of the hfq mutant would be rescued by overexpression of HilD and FlhDC, and we proved this to be correct. The combination of epitope-tagging and HTPS of immunoprecipitated RNA detected the expression of many intergenic chromosomal regions of Salmonella. Our approach overcomes the limited availability of high-density microarrays that have impeded expression-based sRNA discovery in microorganisms. We present a generic strategy that is ideal for the systems-level analysis of the post-transcriptional regulons of RNA-binding proteins and for sRNA discovery in a wide range of bacteria

In silico discovery of blood cell macromolecular associations

Background Physical molecular interactions are the basis of intracellular signalling and gene regulatory networks, and comprehensive, accessible databases are needed for their discovery. Highly correlated transcripts may reflect important functional associations, but identification of such associations from primary data are cumbersome. We have constructed and adapted a user-friendly web application to discover and identify putative macromolecular associations in human peripheral blood based on significant correlations at the transcriptional level. Methods The blood transcriptome was characterized by quantification of 17,328 RNA species, including 341 mature microRNAs in 105 clinically well-characterized postmenopausal women. Intercorrelation of detected transcripts signal levels generated a matrix with > 150 million correlations recognizing the human blood RNA interactome. The correlations with calculated adjusted p-values were made easily accessible by a novel web application. Results We found that significant transcript correlations within the giant matrix reflect experimentally documented interactions involving select ubiquitous blood relevant transcription factors (CREB1, GATA1, and the glucocorticoid receptor (GR, NR3C1)). Their responsive genes recapitulated up to 91% of these as significant correlations, and were replicated in an independent cohort of 1204 individual blood samples from the Framingham Heart Study. Furthermore, experimentally documented mRNAs/miRNA associations were also reproduced in the matrix, and their predicted functional co-expression described. The blood transcript web application is available at http://app.uio.no/med/klinmed/correlation-browser/blood/index.php and works on all commonly used internet browsers. Conclusions Using in silico analyses and a novel web application, we found that correlated blood transcripts across 105 postmenopausal women reflected experimentally proven molecular associations. Furthermore, the associations were reproduced in a much larger and more heterogeneous cohort and should therefore be generally representative. The web application lends itself to be a useful hypothesis generating tool for identification of regulatory mechanisms in complex biological data sets.publishedVersio

Community oncology clinicians’ knowledge, beliefs, and attitudes regarding genomic tumor testing

Introduction: Genomic tumor testing (GTT) is a new technology that promises to make cancer treatment more precise. However, little is known about clinicians’ knowledge, beliefs, and attitudes regarding GTT, particularly in community oncology settings

Bone mineral density and the risk of incident dementia:A meta-analysis

Background: It is not known whether bone mineral density (BMD) measured at baseline or as the rate of decline prior to baseline (prior bone loss) is a stronger predictor of incident dementia or Alzheimer's disease (AD). Methods:We performed a meta-analysis of three longitudinal studies, the Framingham Heart Study (FHS), the Rotterdam Study (RS), and the Rush Memory and Aging Project (MAP), modeling the time to diagnosis of dementia as a function of BMD measures accounting for covariates. We included individuals with one or two BMD assessments, aged ≥60 years, and free of dementia at baseline with follow-up available. BMD was measured at the hip femoral neck using dual-energy X-ray absorptiometry (DXA), or at the heel calcaneus using quantitative ultrasound to calculate estimated BMD (eBMD). BMD at study baseline (“baseline BMD”) and annualized percentage change in BMD prior to baseline (“prior bone loss”) were included as continuous measures. The primary outcome was incident dementia diagnosis within 10 years of baseline, and incident AD was a secondary outcome. Baseline covariates included age, sex, body mass index, ApoE4 genotype, and education. Results: The combined sample size across all three studies was 4431 with 606 incident dementia diagnoses, 498 of which were AD. A meta-analysis of baseline BMD across three studies showed higher BMD to have a significant protective association with incident dementia with a hazard ratio of 0.47 (95% CI: 0.23–0.96; p = 0.038) per increase in g/cm2, or 0.91 (95% CI: 0.84–0.995) per standard deviation increase. We observed a significant association between prior bone loss and incident dementia with a hazard ratio of 1.30 (95% CI: 1.12–1.51; p &lt; 0.001) per percent increase in prior bone loss only in the FHS cohort. Conclusions: Baseline BMD but not prior bone loss was associated with incident dementia in a meta-analysis across three studies.</p

Cultural representations of dementia

In a Perspective, Alexandra Hillman and Joanna Latimer discuss cultural representations of dementia in the media, film, and literature

Community oncologists\u27 perceptions and utilization of large-panel genomic tumor testing.

PURPOSE: Large-panel genomic tumor testing (GTT) is an emerging technology with great promise but uncertain clinical value. Previous research has documented variability in academic oncologists\u27 perceptions and use of GTT, but little is known about community oncologists\u27 perceptions of GTT and how perceptions relate to clinicians\u27 intentions to use GTT. METHODS: Community oncology physicians (N = 58) participating in a statewide initiative aimed at improving access to large-panel GTT completed surveys assessing their confidence in using GTT, attitudes regarding the value of GTT, perceptions of barriers to GTT implementation, and future intentions to use GTTs. Descriptive and multivariable regression analyses were conducted to characterize these perceptions and to explore the relationships between them. RESULTS: There was substantial variability in clinicians\u27 perceptions of GTT. Clinicians generally had moderate confidence in their ability to use GTT, but lower confidence in patients\u27 ability to understand test results and access targeted treatment. Clinicians had positive attitudes regarding the value of GTT. Clinicians\u27 future intentions to use GTT were associated with greater confidence in using GTT and greater perceived barriers to implementing GTT, but not with attitudes about the value of GTT. CONCLUSIONS: Community oncologists\u27 perceptions of large-panel genomic tumor testing are variable, and their future intentions to use GTT are associated with both their confidence in and perceived barriers to its use, but not with their attitudes towards GTT. More research is needed to understand other factors that determine how oncologists perceive and use GTT in clinical practice

The 2-degree Field Lensing Survey: design and clustering measurements

We present the 2-degree Field Lensing Survey (2dFLenS), a new galaxy redshift survey performed at the Anglo-Australian Telescope. 2dFLenS is the first wide-area spectroscopic survey specifically targeting the area mapped by deep-imaging gravitational lensing fields, in this case the Kilo-Degree Survey. 2dFLenS obtained 70 079 redshifts in the range z < 0.9 over an area of 731 deg2, and is designed to extend the data sets available for testing gravitational physics and promote the development of relevant algorithms for joint imaging and spectroscopic analysis. The redshift sample consists first of 40 531 Luminous Red Galaxies (LRGs), which enable analyses of galaxy–galaxy lensing, redshift-space distortion, and the overlapping source redshift distribution by cross-correlation. An additional 28 269 redshifts form a magnitude-limited (r < 19.5) nearly complete subsample, allowing direct source classification and photometric-redshift calibration. In this paper, we describe the motivation, target selection, spectroscopic observations, and clustering analysis of 2dFLenS. We use power spectrum multipole measurements to fit the redshift-space distortion parameter of the LRG sample in two redshift ranges 0.15 < z < 0.43 and 0.43 < z < 0.7 as β = 0.49 ± 0.15 and β = 0.26 ± 0.09, respectively. These values are consistent with those obtained from LRGs in the Baryon Oscillation Spectroscopic Survey. 2dFLenS data products will be released via our website http://2dflens.swin.edu.au