1 research outputs found
A Comprehensive Study of Gender Bias in Chemical Named Entity Recognition Models
Objective. Chemical named entity recognition (NER) models have the potential
to impact a wide range of downstream tasks, from identifying adverse drug
reactions to general pharmacoepidemiology. However, it is unknown whether these
models work the same for everyone. Performance disparities can potentially
cause harm rather than the intended good. Hence, in this paper, we measure
gender-related performance disparities of chemical NER systems.
Materials and Methods. We develop a framework to measure gender bias in
chemical NER models using synthetic data and a newly annotated dataset of over
92,405 words with self-identified gender information from Reddit. We applied
and evaluated state-of-the-art biomedical NER models.
Results. Our findings indicate that chemical NER models are biased. The
results of the bias tests on the synthetic dataset and the real-world data
multiple fairness issues. For example, for synthetic data, we find that
female-related names are generally classified as chemicals, particularly in
datasets containing many brand names rather than standard ones. For both
datasets, we find consistent fairness issues resulting in substantial
performance disparities between female- and male-related data.
Discussion. Our study highlights the issue of biases in chemical NER models.
For example, we find that many systems cannot detect contraceptives (e.g.,
birth control).
Conclusion. Chemical NER models are biased and can be harmful to
female-related groups. Therefore, practitioners should carefully consider the
potential biases of these models and take steps to mitigate them