14 research outputs found
Investigating Trade-offs For Fair Machine Learning Systems
Fairness in software systems aims to provide algorithms that operate in a nondiscriminatory manner, with respect to protected attributes such as gender, race,
or age. Ensuring fairness is a crucial non-functional property of data-driven Machine Learning systems. Several approaches (i.e., bias mitigation methods) have
been proposed in the literature to reduce bias of Machine Learning systems. However, this often comes hand in hand with performance deterioration. Therefore, this
thesis addresses trade-offs that practitioners face when debiasing Machine Learning
systems.
At first, we perform a literature review to investigate the current state of the
art for debiasing Machine Learning systems. This includes an overview of existing
debiasing techniques and how they are evaluated (e.g., how is bias measured).
As a second contribution, we propose a benchmarking approach that allows for
an evaluation and comparison of bias mitigation methods and their trade-offs (i.e.,
how much performance is sacrificed for improving fairness).
Afterwards, we propose a debiasing method ourselves, which modifies already
trained Machine Learning models, with the goal to improve both, their fairness and
accuracy.
Moreover, this thesis addresses the challenge of how to deal with fairness with
regards to age. This question is answered with an empirical evaluation on real-world
datasets
Privileged and Unprivileged Groups: An Empirical Study on the Impact of the Age Attribute on Fairness
An Empirical Study on the Fairness of Pre-trained Word Embeddings
Pre-trained word embedding models are easily distributed and applied, as they alleviate
users from the effort to train models themselves.
With widely distributed models, it is important to ensure that they do not exhibit undesired behaviour, such as biases against population groups. For this purpose, we carry out
an empirical study on evaluating the bias of
15 publicly available, pre-trained word embeddings model based on three training algorithms
(GloVe, word2vec, and fastText) with
regard to four bias metrics (WEAT, SEMBIAS,
DIRECT BIAS, and ECT). The choice of word
embedding models and bias metrics is motivated by a literature survey over 37 publications
which quantified bias on pre-trained word embeddings. Our results indicate that fastText
is the least biased model (in 8 out of 12 cases)
and small vector lengths lead to a higher bias
Py2Cy: A Genetic Improvement Tool To Speed Up Python
Due to its ease of use and wide range of custom libraries, Python has
quickly gained popularity and is used by a wide range of developers
all over the world. While Python allows for fast writing of source
code, the resulting programs are slow to execute when compared
to programs written in other programming languages like C. One
of the reasons for its slow execution time is the dynamic typing
of variables. Cython is an extension to Python, which can achieve
execution speed-ups by compiler optimization. One possibility for
improvements is the use of static typing, which can be added to
Python scripts by developers. To alleviate the need for manual effort,
we create Py2Cy, a Genetic Improvement tool for automatically
converting Python scripts to statically typed Cython scripts. To
show the feasibility of improving runtime with Py2Cy, we optimize
a Python script for generating Fibonacci numbers. The results show
that Py2Cy is able to speed up the execution time by up to a factor
of 18
The EarlyBIRD Catches the Bug: On Exploiting Early Layers of Encoder Models for More Efficient Code Classification
The use of modern Natural Language Processing (NLP) techniques has shown to
be beneficial for software engineering tasks, such as vulnerability detection
and type inference. However, training deep NLP models requires significant
computational resources. This paper explores techniques that aim at achieving
the best usage of resources and available information in these models.
We propose a generic approach, EarlyBIRD, to build composite representations
of code from the early layers of a pre-trained transformer model. We
empirically investigate the viability of this approach on the CodeBERT model by
comparing the performance of 12 strategies for creating composite
representations with the standard practice of only using the last encoder
layer.
Our evaluation on four datasets shows that several early layer combinations
yield better performance on defect detection, and some combinations improve
multi-class classification. More specifically, we obtain a +2 average
improvement of detection accuracy on Devign with only 3 out of 12 layers of
CodeBERT and a 3.3x speed-up of fine-tuning. These findings show that early
layers can be used to obtain better results using the same resources, as well
as to reduce resource usage during fine-tuning and inference.Comment: The content in this pre-print is the same as in the CRC accepted for
publication in the ACM Joint European Software Engineering Conference and
Symposium on the Foundations of Software Engineering (ESEC/FSE 2023
Fairness Testing: A Comprehensive Survey and Analysis of Trends
Unfair behaviors of Machine Learning (ML) software have garnered increasing
attention and concern among software engineers. To tackle this issue, extensive
research has been dedicated to conducting fairness testing of ML software, and
this paper offers a comprehensive survey of existing studies in this field. We
collect 100 papers and organize them based on the testing workflow (i.e., how
to test) and testing components (i.e., what to test). Furthermore, we analyze
the research focus, trends, and promising directions in the realm of fairness
testing. We also identify widely-adopted datasets and open-source tools for
fairness testing
Enhanced Fairness Testing via Generating Effective Initial Individual Discriminatory Instances
Fairness testing aims at mitigating unintended discrimination in the
decision-making process of data-driven AI systems. Individual discrimination
may occur when an AI model makes different decisions for two distinct
individuals who are distinguishable solely according to protected attributes,
such as age and race. Such instances reveal biased AI behaviour, and are called
Individual Discriminatory Instances (IDIs).
In this paper, we propose an approach for the selection of the initial seeds
to generate IDIs for fairness testing. Previous studies mainly used random
initial seeds to this end. However this phase is crucial, as these seeds are
the basis of the follow-up IDIs generation. We dubbed our proposed seed
selection approach I&D. It generates a large number of initial IDIs exhibiting
a great diversity, aiming at improving the overall performance of fairness
testing.
Our empirical study reveal that I&D is able to produce a larger number of
IDIs with respect to four state-of-the-art seed generation approaches,
generating 1.68X more IDIs on average. Moreover, we compare the use of I&D to
train machine learning models and find that using I&D reduces the number of
remaining IDIs by 29% when compared to the state-of-the-art, thus indicating
that I&D is effective for improving model fairnessComment: 19 pages, 7 figure
25th annual computational neuroscience meeting: CNS-2016
The same neuron may play different functional roles in the neural circuits to which it belongs. For example, neurons in the Tritonia pedal ganglia may participate in variable phases of the swim motor rhythms [1]. While such neuronal functional variability is likely to play a major role the delivery of the functionality of neural systems, it is difficult to study it in most nervous systems. We work on the pyloric rhythm network of the crustacean stomatogastric ganglion (STG) [2]. Typically network models of the STG treat neurons of the same functional type as a single model neuron (e.g. PD neurons), assuming the same conductance parameters for these neurons and implying their synchronous firing [3, 4]. However, simultaneous recording of PD neurons shows differences between the timings of spikes of these neurons. This may indicate functional variability of these neurons. Here we modelled separately the two PD neurons of the STG in a multi-neuron model of the pyloric network. Our neuron models comply with known correlations between conductance parameters of ionic currents. Our results reproduce the experimental finding of increasing spike time distance between spikes originating from the two model PD neurons during their synchronised burst phase. The PD neuron with the larger calcium conductance generates its spikes before the other PD neuron. Larger potassium conductance values in the follower neuron imply longer delays between spikes, see Fig. 17.Neuromodulators change the conductance parameters of neurons and maintain the ratios of these parameters [5]. Our results show that such changes may shift the individual contribution of two PD neurons to the PD-phase of the pyloric rhythm altering their functionality within this rhythm. Our work paves the way towards an accessible experimental and computational framework for the analysis of the mechanisms and impact of functional variability of neurons within the neural circuits to which they belong
Bias Mitigation for Machine Learning Classifiers: A Comprehensive Survey
This paper provides a comprehensive survey of bias mitigation methods for achieving fairness in Machine
Learning (ML) models. We collect a total of 341 publications concerning bias mitigation for ML classifiers.
These methods can be distinguished based on their intervention procedure (i.e., pre-processing, in-processing,
post-processing) and the technique they apply. We investigate how existing bias mitigation methods are
evaluated in the literature. In particular, we consider datasets, metrics and benchmarking. Based on the
gathered insights (e.g., What is the most popular fairness metric? How many datasets are used for evaluating
bias mitigation methods?), we hope to support practitioners in making informed choices when developing
and evaluating new bias mitigation methods