31 research outputs found
Sources of Irreproducibility in Machine Learning: A Review
Lately, several benchmark studies have shown that the state of the art in
some of the sub-fields of machine learning actually has not progressed despite
progress being reported in the literature. The lack of progress is partly
caused by the irreproducibility of many model comparison studies. Model
comparison studies are conducted that do not control for many known sources of
irreproducibility. This leads to results that cannot be verified by third
parties. Our objective is to provide an overview of the sources of
irreproducibility that are reported in the literature. We review the literature
to provide an overview and a taxonomy in addition to a discussion on the
identified sources of irreproducibility. Finally, we identify three lines of
further inquiry
Variance of ML-based software fault predictors: are we really improving fault prediction?
Software quality assurance activities become increasingly difficult as
software systems become more and more complex and continuously grow in size.
Moreover, testing becomes even more expensive when dealing with large-scale
systems. Thus, to effectively allocate quality assurance resources, researchers
have proposed fault prediction (FP) which utilizes machine learning (ML) to
predict fault-prone code areas. However, ML algorithms typically make use of
stochastic elements to increase the prediction models' generalizability and
efficiency of the training process. These stochastic elements, also known as
nondeterminism-introducing (NI) factors, lead to variance in the training
process and as a result, lead to variance in prediction accuracy and training
time. This variance poses a challenge for reproducibility in research. More
importantly, while fault prediction models may have shown good performance in
the lab (e.g., often-times involving multiple runs and averaging outcomes),
high variance of results can pose the risk that these models show low
performance when applied in practice. In this work, we experimentally analyze
the variance of a state-of-the-art fault prediction approach. Our experimental
results indicate that NI factors can indeed cause considerable variance in the
fault prediction models' accuracy. We observed a maximum variance of 10.10% in
terms of the per-class accuracy metric. We thus, also discuss how to deal with
such variance
Hyperparameter Optimization for Multi-Objective Reinforcement Learning
Reinforcement learning (RL) has emerged as a powerful approach for tackling
complex problems. The recent introduction of multi-objective reinforcement
learning (MORL) has further expanded the scope of RL by enabling agents to make
trade-offs among multiple objectives. This advancement not only has broadened
the range of problems that can be tackled but also created numerous
opportunities for exploration and advancement. Yet, the effectiveness of RL
agents heavily relies on appropriately setting their hyperparameters. In
practice, this task often proves to be challenging, leading to unsuccessful
deployments of these techniques in various instances. Hence, prior research has
explored hyperparameter optimization in RL to address this concern.
This paper presents an initial investigation into the challenge of
hyperparameter optimization specifically for MORL. We formalize the problem,
highlight its distinctive challenges, and propose a systematic methodology to
address it. The proposed methodology is applied to a well-known environment
using a state-of-the-art MORL algorithm, and preliminary results are reported.
Our findings indicate that the proposed methodology can effectively provide
hyperparameter configurations that significantly enhance the performance of
MORL agents. Furthermore, this study identifies various future research
opportunities to further advance the field of hyperparameter optimization for
MORL.Comment: Presented at the MODeM workshop https://modem2023.vub.ac.be/
Deep Learning in Unconventional Domains
Machine learning methods have dramatically improved in recent years thanks to advances in deep learning (LeCun et al., 2015), a set of methods for training high-dimensional, highly-parameterized, nonlinear functions. Yet deep learning progress has been concentrated in the domains of computer vision, vision-based reinforcement learning, and natural language processing. This dissertation is an attempt to extend deep learning into domains where it has thus far had little impact or has never been applied. It presents new deep learning algorithms and state-of-the-art results on tasks in the domains of source-code analysis, relational databases, and tabular data.</p
Neuroengineering of Clustering Algorithms
Cluster analysis can be broadly divided into multivariate data visualization, clustering algorithms, and cluster validation. This dissertation contributes neural network-based techniques to perform all three unsupervised learning tasks. Particularly, the first paper provides a comprehensive review on adaptive resonance theory (ART) models for engineering applications and provides context for the four subsequent papers. These papers are devoted to enhancements of ART-based clustering algorithms from (a) a practical perspective by exploiting the visual assessment of cluster tendency (VAT) sorting algorithm as a preprocessor for ART offline training, thus mitigating ordering effects; and (b) an engineering perspective by designing a family of multi-criteria ART models: dual vigilance fuzzy ART and distributed dual vigilance fuzzy ART (both of which are capable of detecting complex cluster structures), merge ART (aggregates partitions and lessens ordering effects in online learning), and cluster validity index vigilance in fuzzy ART (features a robust vigilance parameter selection and alleviates ordering effects in offline learning). The sixth paper consists of enhancements to data visualization using self-organizing maps (SOMs) by depicting in the reduced dimension and topology-preserving SOM grid information-theoretic similarity measures between neighboring neurons. This visualization\u27s parameters are estimated using samples selected via a single-linkage procedure, thereby generating heatmaps that portray more homogeneous within-cluster similarities and crisper between-cluster boundaries. The seventh paper presents incremental cluster validity indices (iCVIs) realized by (a) incorporating existing formulations of online computations for clusters\u27 descriptors, or (b) modifying an existing ART-based model and incrementally updating local density counts between prototypes. Moreover, this last paper provides the first comprehensive comparison of iCVIs in the computational intelligence literature --Abstract, page iv
Learning from alternative sources of supervision
With the rise of the internet, data of many varieties including: images, audio, text
and video are abundant. Unfortunately for a very specific task one might have, the
data for that problem is not typically abundant unless you are lucky. Typically one
might have only a small amount of labelled data, or only noisy labels, or labels for a
different task, or perhaps a simulator and reward function but no demonstrations, or
even a simulator but no reward function at all. However, arguably no task is truly novel
and so it is often possible for neural networks to benefit from the abundant data that
is related to your current task. This thesis documents three methods for learning from
alternative sources of supervision, an alternative to the more preferable case of simply
having unlimited amounts of direct examples of your task. Firstly we show how having
data from many related tasks could be described with a simple graphical model and
fit using a Variational-Autoencoder - directly modelling and representing the relations
amongst tasks. Secondly we investigate various forms of prediction-based intrinsic
rewards for agents in a simulator with no extrinsic rewards. Thirdly we introduce a
novel intrinsic reward and investigate how to best combine it with an extrinsic reward
for best performance
Revisiting Neural Program Smoothing for Fuzzing
Testing with randomly generated inputs (fuzzing) has gained significant
traction due to its capacity to expose program vulnerabilities automatically.
Fuzz testing campaigns generate large amounts of data, making them ideal for
the application of machine learning (ML). Neural program smoothing (NPS), a
specific family of ML-guided fuzzers, aims to use a neural network as a smooth
approximation of the program target for new test case generation.
In this paper, we conduct the most extensive evaluation of NPS fuzzers
against standard gray-box fuzzers (>11 CPU years and >5.5 GPU years), and make
the following contributions: (1) We find that the original performance claims
for NPS fuzzers do not hold; a gap we relate to fundamental, implementation,
and experimental limitations of prior works. (2) We contribute the first
in-depth analysis of the contribution of machine learning and gradient-based
mutations in NPS. (3) We implement Neuzz++, which shows that addressing the
practical limitations of NPS fuzzers improves performance, but that standard
gray-box fuzzers almost always surpass NPS-based fuzzers. (4) As a consequence,
we propose new guidelines targeted at benchmarking fuzzing based on machine
learning, and present MLFuzz, a platform with GPU access for easy and
reproducible evaluation of ML-based fuzzers. Neuzz++, MLFuzz, and all our data
are public.Comment: Accepted as conference paper at ESEC/FSE 202
Evolutionary Computation 2020
Intelligent optimization is based on the mechanism of computational intelligence to refine a suitable feature model, design an effective optimization algorithm, and then to obtain an optimal or satisfactory solution to a complex problem. Intelligent algorithms are key tools to ensure global optimization quality, fast optimization efficiency and robust optimization performance. Intelligent optimization algorithms have been studied by many researchers, leading to improvements in the performance of algorithms such as the evolutionary algorithm, whale optimization algorithm, differential evolution algorithm, and particle swarm optimization. Studies in this arena have also resulted in breakthroughs in solving complex problems including the green shop scheduling problem, the severe nonlinear problem in one-dimensional geodesic electromagnetic inversion, error and bug finding problem in software, the 0-1 backpack problem, traveler problem, and logistics distribution center siting problem. The editors are confident that this book can open a new avenue for further improvement and discoveries in the area of intelligent algorithms. The book is a valuable resource for researchers interested in understanding the principles and design of intelligent algorithms