Judgment Sieve: Reducing Uncertainty in Group Judgments through Interventions Targeting Ambiguity versus Disagreement

When groups of people are tasked with making a judgment, the issue of uncertainty often arises. Existing methods to reduce uncertainty typically focus on iteratively improving specificity in the overall task instruction. However, uncertainty can arise from multiple sources, such as ambiguity of the item being judged due to limited context, or disagreements among the participants due to different perspectives and an under-specified task. A one-size-fits-all intervention may be ineffective if it is not targeted to the right source of uncertainty. In this paper we introduce a new workflow, Judgment Sieve, to reduce uncertainty in tasks involving group judgment in a targeted manner. By utilizing measurements that separate different sources of uncertainty during an initial round of judgment elicitation, we can then select a targeted intervention adding context or deliberation to most effectively reduce uncertainty on each item being judged. We test our approach on two tasks: rating word pair similarity and toxicity of online comments, showing that targeted interventions reduced uncertainty for the most uncertain cases. In the top 10% of cases, we saw an ambiguity reduction of 21.4% and 25.7%, and a disagreement reduction of 22.2% and 11.2% for the two tasks respectively. We also found through a simulation that our targeted approach reduced the average uncertainty scores for both sources of uncertainty as opposed to uniform approaches where reductions in average uncertainty from one source came with an increase for the other

Confidence Contours: Uncertainty-Aware Annotation for Medical Semantic Segmentation

Medical image segmentation modeling is a high-stakes task where understanding of uncertainty is crucial for addressing visual ambiguity. Prior work has developed segmentation models utilizing probabilistic or generative mechanisms to infer uncertainty from labels where annotators draw a singular boundary. However, as these annotations cannot represent an individual annotator's uncertainty, models trained on them produce uncertainty maps that are difficult to interpret. We propose a novel segmentation representation, Confidence Contours, which uses high- and low-confidence ``contours'' to capture uncertainty directly, and develop a novel annotation system for collecting contours. We conduct an evaluation on the Lung Image Dataset Consortium (LIDC) and a synthetic dataset. From an annotation study with 30 participants, results show that Confidence Contours provide high representative capacity without considerably higher annotator effort. We also find that general-purpose segmentation models can learn Confidence Contours at the same performance level as standard singular annotations. Finally, from interviews with 5 medical experts, we find that Confidence Contour maps are more interpretable than Bayesian maps due to representation of structural uncertainty.Comment: 10 pages content, 12 pages total. Accepted to HCOMP '2

Case Repositories: Towards Case-Based Reasoning for AI Alignment

Case studies commonly form the pedagogical backbone in law, ethics, and many other domains that face complex and ambiguous societal questions informed by human values. Similar complexities and ambiguities arise when we consider how AI should be aligned in practice: when faced with vast quantities of diverse (and sometimes conflicting) values from different individuals and communities, with whose values is AI to align, and how should AI do so? We propose a complementary approach to constitutional AI alignment, grounded in ideas from case-based reasoning (CBR), that focuses on the construction of policies through judgments on a set of cases. We present a process to assemble such a case repository by: 1) gathering a set of ``seed'' cases -- questions one may ask an AI system -- in a particular domain, 2) eliciting domain-specific key dimensions for cases through workshops with domain experts, 3) using LLMs to generate variations of cases not seen in the wild, and 4) engaging with the public to judge and improve cases. We then discuss how such a case repository could assist in AI alignment, both through directly acting as precedents to ground acceptable behaviors, and as a medium for individuals and communities to engage in moral reasoning around AI.Comment: MP2 workshop @ NeurIPS 202

Effect of Socioeconomic Status and Comorbidities on Thyroid Cancer Survival Outcomes

Purpose/Objectives: Thyroid cancer ranks among the most common head and neck malignancies and makes up 3% of new cancer cases per year, having increased in overall incidence in the last several decades. This study investigates patient characteristics, socioeconomic status (SES), and medical comorbidities as independent predictors of survival in patients with thyroid cancer, treated at a tertiary care hospital with a diverse, primarily low SES patient population. Materials/Methods: The Henry Ford Health System’s Virtual Data Warehouse Tumor Registry was used to identify patients with thyroid cancers from 1997 to 2016. Socioeconomic data was obtained from 2010 US Census. Comorbidities were quantified using the Charlson Comorbidity Index (CCI). Statistical analysis was performed using Kaplan-Meier estimator and Cox proportional hazards models. Results: There were 1042 patients with thyroid cancer. 5- and 10-year survival probabilities decreased with age \u3e=60 years (p\u3c0.001), stage 3/4 (p\u3c0.001), clinical N stage 1-3 (p\u3c0.001), black race (p=0.001), CCI\u3e1 (p\u3c0.001), and lowest quartile median household income (p\u3c0.001). In multivariate analysis, age (1-year increase HR: 1.97, 95% CI: 1.06-1.09), sex (female vs. male HR: 0.67, 95% CI: 0.48-0.95), and CCI (1-point increase HR: 1.26, 95% CI: 1.19-1.34) were significantly associated with survival outcomes. A CCI cutoff of \u3e1 was a reliable predictor of mortality (AUC: 0.759, 95% CI: 0.716-0.801). CCI was significantly correlated with stage (r=0.166, p=0.004) and median household income (r=-0.175, p\u3c0.001). Conclusion: Increased age, male sex, and medical comorbidities predicted significantly worse survival in thyroid cancer. Race and median household income were not independent significant prognostic indicators

Histone modifications influence mediator interactions with chromatin

The Mediator complex transmits activation signals from DNA bound transcription factors to the core transcription machinery. Genome wide localization studies have demonstrated that Mediator occupancy not only correlates with high levels of transcription, but that the complex also is present at transcriptionally silenced locations. We provide evidence that Mediator localization is guided by an interaction with histone tails, and that this interaction is regulated by their post-translational modifications. A quantitative, high-density genetic interaction map revealed links between Mediator components and factors affecting chromatin structure, especially histone deacetylases. Peptide binding assays demonstrated that pure wild-type Mediator forms stable complexes with the tails of Histone H3 and H4. These binding assays also showed Mediator—histone H4 peptide interactions are specifically inhibited by acetylation of the histone H4 lysine 16, a residue critical in transcriptional silencing. Finally, these findings were validated by tiling array analysis that revealed a broad correlation between Mediator and nucleosome occupancy in vivo, but a negative correlation between Mediator and nucleosomes acetylated at histone H4 lysine 16. Our studies show that chromatin structure and the acetylation state of histones are intimately connected to Mediator localization

Identification of human neutralizing antibodies against MERS-CoV and their role in virus adaptive evolution

The newly emerging Middle East Respiratory Syndrome coronavirus (MERS-CoV) causes a Severe Acute Respiratory Syndrome-like disease with ∼43% mortality. Given the recent detection of virus in dromedary camels, zoonotic transfer of MERS-CoV to humans is suspected. In addition, little is known about the role of human neutralizing Ab (nAb) pressure as a driving force in MERS-CoV adaptive evolution. Here, we used a well-characterized nonimmune human Ab-phage library and a panning strategy with proteoliposomes and cells to identify seven human nAbs against the receptor-binding domain (RBD) of the MERS-CoV Spike protein. These nAbs bind to three different epitopes in the RBD and human dipeptidyl peptidase 4 (hDPP4) interface with subnanomolar/nanomolar binding affinities and block the binding of MERS-CoV Spike protein with its hDPP4 receptor. Escape mutant assays identified five amino acid residues that are critical for neutralization escape. Despite the close proximity of the three epitopes on the RBD interface, escape from one epitope did not have a major impact on neutralization with Abs directed to a different epitope. Importantly, the majority of escape mutations had negative impacts on hDPP4 receptor binding and viral fitness. To our knowledge, these results provide the first report on human nAbs against MERS-CoV that may contribute to MERS-CoV clearance and evolution. Moreover, in the absence of a licensed vaccine or antiviral for MERS, this panel of nAbs offers the possibility of developing human mAb-based immunotherapy, especially for health-care workers

Long-term outcomes of ranibizumab therapy for diabetic macular edema: the 36-month results from two phase III trials: RISE and RIDE.

PURPOSE: To report 36-month outcomes of RIDE (NCT00473382) and RISE (NCT00473330), trials of ranibizumab in diabetic macular edema (DME). DESIGN: Phase III, randomized, multicenter, double-masked, 3-year trials, sham injection-controlled for 2 years. PARTICIPANTS: Adults with DME (n=759), baseline best-corrected visual acuity (BCVA) 20/40 to 20/320 Snellen equivalent, and central foveal thickness (CFT) ≥ 275 μm on optical coherence tomography. METHODS: Patients were randomized equally (1 eye per patient) to monthly 0.5 mg or 0.3 mg ranibizumab or sham injection. In the third year, sham patients, while still masked, were eligible to cross over to monthly 0.5 mg ranibizumab. Macular laser was available to all patients starting at month 3; panretinal laser was available as necessary. MAIN OUTCOME MEASURES: The proportion of patients gaining ≥15 Early Treatment Diabetic Retinopathy Study letters in BCVA from baseline at month 24. RESULTS: Visual acuity (VA) outcomes seen at month 24 in ranibizumab groups were consistent through month 36; the proportions of patients who gained ≥15 letters from baseline at month 36 in the sham/0.5 mg, 0.3 mg, and 0.5 mg ranibizumab groups were 19.2%, 36.8%, and 40.2%, respectively, in RIDE and 22.0%, 51.2%, and 41.6%, respectively, in RISE. In the ranibizumab arms, reductions in CFT seen at 24 months were, on average, sustained through month 36. After crossover to 1 year of treatment with ranibizumab, average VA gains in the sham/0.5 mg group were lower compared with gains seen in the ranibizumab patients after 1 year of treatment (2.8 vs. 10.6 and 11.1 letters). Per-injection rates of endophthalmitis remained low over time (∼0.06% per injection). The incidence of serious adverse events potentially related to systemic vascular endothelial growth factor inhibition was 19.7% in patients who received 0.5 mg ranibizumab compared with 16.8% in the 0.3 mg group. CONCLUSIONS: The strong VA gains and improvement in retinal anatomy achieved with ranibizumab at month 24 were sustained through month 36. Delayed treatment in patients receiving sham treatment did not seem to result in the same extent of VA improvement observed in patients originally randomized to ranibizumab. Ocular and systemic safety was generally consistent with the results seen at month 24. FINANCIAL DISCLOSURE(S): Proprietary or commercial disclosure may be found after the references

Multiplex primer prediction software for divergent targets

We describe a Multiplex Primer Prediction (MPP) algorithm to build multiplex compatible primer sets to amplify all members of large, diverse and unalignable sets of target sequences. The MPP algorithm is scalable to larger target sets than other available software, and it does not require a multiple sequence alignment. We applied it to questions in viral detection, and demonstrated that there are no universally conserved priming sequences among viruses and that it could require an unfeasibly large number of primers (∼3700 18-mers or ∼2000 10-mers) to generate amplicons from all sequenced viruses. We then designed primer sets separately for each viral family, and for several diverse species such as foot-and-mouth disease virus (FMDV), hemagglutinin (HA) and neuraminidase (NA) segments of influenza A virus, Norwalk virus, and HIV-1. We empirically demonstrated the application of the software with a multiplex set of 16 short (10 nt) primers designed to amplify the Poxviridae family to produce a specific amplicon from vaccinia virus

Genomic surveillance of Escherichia coli and Klebsiella spp. in hospital sink drains and patients

Escherichia coli and Klebsiella spp. are important human pathogens that cause a wide spectrum of clinical disease. In healthcare settings, sinks and other wastewater sites have been shown to be reservoirs of antimicrobial-resistant E. coli and Klebsiella spp., particularly in the context of outbreaks of resistant strains amongst patients. Without focusing exclusively on resistance markers or a clinical outbreak, we demonstrate that many hospital sink drains are abundantly and persistently colonised with diverse populations of E. coli, Klebsiella pneumoniae and Klebsiella oxytoca, including both antimicrobial-resistant and susceptible strains. Using whole genome sequencing (WGS) of 439 isolates, we show that environmental bacterial populations are largely structured by ward and sink, with only a handful of lineages, such as E. coli ST635, being widely distributed, suggesting different prevailing ecologies which may vary as a result of different inputs and selection pressures. WGS of 46 contemporaneous patient isolates identified one (2%; 95% CI 0.05-11%) E. coli urine infection-associated isolate with high similarity to a prior sink isolate, suggesting that sinks may contribute to up to 10% of infections caused by these organisms in patients on the ward over the same timeframe. Using metagenomics from 20 sink-timepoints, we show that sinks also harbour many clinically relevant antimicrobial resistance genes including blaCTX-M, blaSHV and mcr, and may act as niches for the exchange and amplification of these genes. Our study reinforces the potential role of sinks in contributing to Enterobacterales infection and antimicrobial resistance in hospital patients, something that could be amenable to intervention