Comparing Humans and Models on a Similar Scale: Towards Cognitive Gender Bias Evaluation in Coreference Resolution

Spurious correlations were found to be an important factor explaining model performance in various NLP tasks (e.g., gender or racial artifacts), often considered to be ''shortcuts'' to the actual task. However, humans tend to similarly make quick (and sometimes wrong) predictions based on societal and cognitive presuppositions. In this work we address the question: can we quantify the extent to which model biases reflect human behaviour? Answering this question will help shed light on model performance and provide meaningful comparisons against humans. We approach this question through the lens of the dual-process theory for human decision-making. This theory differentiates between an automatic unconscious (and sometimes biased) ''fast system'' and a ''slow system'', which when triggered may revisit earlier automatic reactions. We make several observations from two crowdsourcing experiments of gender bias in coreference resolution, using self-paced reading to study the ''fast'' system, and question answering to study the ''slow'' system under a constrained time setting. On real-world data humans make $\sim$3\% more gender-biased decisions compared to models, while on synthetic data models are $\sim$12\% more biased

The role of landscape and history on the genetic structure of peripheral populations of the Near Eastern fire salamander, Salamandra infraimmaculata, in Northern Israel

Genetic studies on core versus peripheral populations have yielded many patterns. This diversity in genetic patterns may reflect diversity in the meaning of peripheral populations as defined by geography, gene flow patterns, historical effects, and ecological conditions. Populations at the lower latitude periphery of a species' range are of particular concern because they may be at increased risk for extinction due to global climate change. In this work we aim to understand the impact of landscape and ecological factors on different geographical types of peripheral populations with respect to levels of genetic diversity and patterns of local population differentiation. We examined three geographical types of peripheral populations of the endangered salamander, Salamandra infraimmaculata, in Northern Israel, in the southernmost periphery of the genus Salamandra, by analyzing the variability in 15 microsatellite loci from 32 sites. Our results showed that: (1) genetic diversity decreases towards the geographical periphery of the species' range; (2) genetic diversity in geographically disjunct peripheral areas is low compared to the core or peripheral populations that are contiguous to the core and most likely affected by a founder effect; (3) ecologically marginal conditions enhance population subdivision. The patterns we found lead to the conclusion that genetic diversity is influenced by a combination of geographical, historical, and ecological factors. These complex patterns should be addressed when prioritizing areas for conservation.Peer reviewe

Direct Recognition of Fusobacterium nucleatum by the NK Cell Natural Cytotoxicity Receptor NKp46 Aggravates Periodontal Disease

Periodontitis is a common human chronic inflammatory disease that results in the destruction of the tooth attachment apparatus and tooth loss. Although infections with periopathogenic bacteria such as Porphyromonas gingivalis (P. gingivalis) and Fusobacterium nucleatum (F. nucleatum) are essential for inducing periodontitis, the nature and magnitude of the disease is determined by the host's immune response. Here, we investigate the role played by the NK killer receptor NKp46 (NCR1 in mice), in the pathogenesis of periodontitis. Using an oral infection periodontitis model we demonstrate that following F. nucleatum infection no alveolar bone loss is observed in mice deficient for NCR1 expression, whereas around 20% bone loss is observed in wild type mice and in mice infected with P. gingivalis. By using subcutaneous chambers inoculated with F. nucleatum we demonstrate that immune cells, including NK cells, rapidly accumulate in the chambers and that this leads to a fast and transient, NCR1-dependant TNF-α secretion. We further show that both the mouse NCR1 and the human NKp46 bind directly to F. nucleatum and we demonstrate that this binding is sensitive to heat, to proteinase K and to pronase treatments. Finally, we show in vitro that the interaction of NK cells with F. nucleatum leads to an NCR1-dependent secretion of TNF-α. Thus, the present study provides the first evidence that NCR1 and NKp46 directly recognize a periodontal pathogen and that this interaction influences the outcome of F. nucleatum-mediated periodontitis

Comparing Humans and Models on a Similar Scale: Towards Cognitive Gender Bias Evaluation in Coreference Resolution

Spurious correlations were found to be an important factor explaining model performance in various NLP tasks (e.g., gender or racial artifacts), often considered to be “shortcuts” to the actual task. However, humans tend to similarly make quick (and sometimes wrong) predictions based on societal and cognitive presuppositions. In this work we address the question: can we quantify the extent to which model biases reflect human behaviour? Answering this question will help shed light on model performance and provide meaningful comparisons against humans. We approach this question through the lens of the dual-process theory for human decision-making. This theory differentiates between an automatic unconscious (and sometimes biased) “fast system” and a “slow system”, which when triggered may revisit earlier automatic reactions. We make several observations from two crowdsourcing experiments of gender bias in coreference resolution, using self- paced reading to study the “fast” system, and question answering to study the “slow” system under a constrained time setting. On real-world data humans make ∼3% more gender- biased decisions compared to models, while on synthetic data models are ∼12% more biased. We make all our of our code and data publicly available

The infiltration of immune cells into the <i>F. nucleatum</i> challenged chambers is NCR1-independent.

<p>The presence of immune cells; (A) lymphocytes and (B) DC (identified by CD11c) and macrophages (identified by F4/80) in the chambers were analyzed by FACS, 2 hours following the injection of <i>F. nucleatum</i>. The mAb used for staining is indicated in the Y axis. IC is isotype control. The percentage of the various cells and the cell numbers are indicated in each quadrant. One representative experiment out of four is shown.</p

Survival of bacteria in chamber exudates.

<p>Viable CFU from chamber exudates were determined, 2 h and 24 h following inoculation of <i>P. gingivalis</i>, <i>F. nucleatum</i> or <i>E. coli</i>. Each symbol represents an individual mouse. One representative experiment out of four is shown. The y axis represents number of colonies (logarithmic scale).</p

Cytokine profiles in challenged chambers.

<p>TNF-α levels were determined by ELISA in chamber exudates of Ncr1<i>^+/+</i> (WT) and Ncr1<i>^gfp/gfp</i> (KO) mice at baseline (0), 2 h and 24 h following bacterial challenge. Data represent absorbance at 650 nm ± SD and are average of three different experiments. ** <i>p</i><0.01.</p

<i>In vivo</i> response to heat-treated <i>F. nucleatum</i> bacteria.

<p>(A) FACS analysis of chamber exudates, 2 hours and 24 hours following challenge with heat-killed <i>F. nucleatum</i>. Staining was performed with Phycoerythrin conjugated anti-DX5 mAb. The percentages of GFP⁺ NK cells are indicated. One representative experiment out of three is shown. (B) TNF-α level in chamber exudates of Ncr1<i>^+/+</i> (WT) and Ncr1<i>^gfp/gfp</i> (KO) was determined by ELISA, 2 and 24 hours following challenge with viable and heat-killed <i>F. nucleatum</i>. Data represent absorbance at 650 nm ± SD and are average of two different experiments. * <i>p</i><0.05, ** <i>p</i><0.01.</p

Direct binding of NCR1 and NKp46 to <i>F. nucleatum</i>.

<p><i>F. nucleatum</i> (A), <i>P. gingivalis</i> (B), uninfected 721.221 cells (C), and 721.221 cells infected with influenza PR8 virus (D), were stained with NCR1-Ig (left) and with NKp46-Ig (right) at two different dosages; 5 µg, blue line and 1 µg, red line. Staining was visualized using a Phycoerythrin conjugated anti-human Ig antibody. The filled grey histograms represent control staining with the NKp46D1-Ig fusion protein. The Median Fluorescence Intensity (MFI) values of the D1-Ig, NCR1-Ig and NKp46 staining are indicated in each of the histograms. One representative experiment out of five is shown. (E) <i>F. nucleatum</i> (upper) and <i>P. gingivalis</i> (lower) bacteria were stained with various activating NK cell receptors fused to Ig (indicated above the quadrants). Figure shows one representative experiment out of three performed.</p

NCR1-dependent bone loss.

<p>(A) Mesio-distal view on a random section. Red, yellow, gray and green represent the enamel, dentin, bone and the residual bone area above the reference line, respectively. (B) Ncr1<i>^+/+</i> (WT) and Ncr1<i>^gfp/gfp</i> (KO) mice (n = 8 in each group) were challenged orally three times at 2-day intervals with inoculums of <i>F. nucleatum</i> and of <i>P. gingivalis</i> (2×10⁸ CFU in 0.2 ml of PBS and 2% carboxymethylcellulose). Six weeks later, the jaws were harvested and the residual alveolar bone volume (mm³×10⁻³) was measured. Error bars represent SD. Data represent percentage of bone loss ± SD and are average of two independent experiments. ** <i>p</i><0.01. The difference in bone loss between the KO and the WT mice following <i>P. gingivalis</i> inoculation was not statistically significant, <i>p</i><0.09.</p