49,683 research outputs found
On Testing Machine Learning Programs
Nowadays, we are witnessing a wide adoption of Machine learning (ML) models
in many safety-critical systems, thanks to recent breakthroughs in deep
learning and reinforcement learning. Many people are now interacting with
systems based on ML every day, e.g., voice recognition systems used by virtual
personal assistants like Amazon Alexa or Google Home. As the field of ML
continues to grow, we are likely to witness transformative advances in a wide
range of areas, from finance, energy, to health and transportation. Given this
growing importance of ML-based systems in our daily life, it is becoming
utterly important to ensure their reliability. Recently, software researchers
have started adapting concepts from the software testing domain (e.g., code
coverage, mutation testing, or property-based testing) to help ML engineers
detect and correct faults in ML programs. This paper reviews current existing
testing practices for ML programs. First, we identify and explain challenges
that should be addressed when testing ML programs. Next, we report existing
solutions found in the literature for testing ML programs. Finally, we identify
gaps in the literature related to the testing of ML programs and make
recommendations of future research directions for the scientific community. We
hope that this comprehensive review of software testing practices will help ML
engineers identify the right approach to improve the reliability of their
ML-based systems. We also hope that the research community will act on our
proposed research directions to advance the state of the art of testing for ML
programs.Comment: This manuscript is part of a submission to the Journal of Systems and
Softwar
On End-to-End Program Generation from User Intention by Deep Neural Networks
This paper envisions an end-to-end program generation scenario using
recurrent neural networks (RNNs): Users can express their intention in natural
language; an RNN then automatically generates corresponding code in a
characterby-by-character fashion. We demonstrate its feasibility through a case
study and empirical analysis. To fully make such technique useful in practice,
we also point out several cross-disciplinary challenges, including modeling
user intention, providing datasets, improving model architectures, etc.
Although much long-term research shall be addressed in this new field, we
believe end-to-end program generation would become a reality in future decades,
and we are looking forward to its practice.Comment: Submitted to 2016 International Conference of Software Engineering
"Vision of 2025 and Beyond" trac
Data Generation as Sequential Decision Making
We connect a broad class of generative models through their shared reliance
on sequential decision making. Motivated by this view, we develop extensions to
an existing model, and then explore the idea further in the context of data
imputation -- perhaps the simplest setting in which to investigate the relation
between unconditional and conditional generative modelling. We formulate data
imputation as an MDP and develop models capable of representing effective
policies for it. We construct the models using neural networks and train them
using a form of guided policy search. Our models generate predictions through
an iterative process of feedback and refinement. We show that this approach can
learn effective policies for imputation problems of varying difficulty and
across multiple datasets.Comment: Accepted for publication at Advances in Neural Information Processing
Systems (NIPS) 201
Learning with Pseudo-Ensembles
We formalize the notion of a pseudo-ensemble, a (possibly infinite)
collection of child models spawned from a parent model by perturbing it
according to some noise process. E.g., dropout (Hinton et. al, 2012) in a deep
neural network trains a pseudo-ensemble of child subnetworks generated by
randomly masking nodes in the parent network. We present a novel regularizer
based on making the behavior of a pseudo-ensemble robust with respect to the
noise process generating it. In the fully-supervised setting, our regularizer
matches the performance of dropout. But, unlike dropout, our regularizer
naturally extends to the semi-supervised setting, where it produces
state-of-the-art results. We provide a case study in which we transform the
Recursive Neural Tensor Network of (Socher et. al, 2013) into a
pseudo-ensemble, which significantly improves its performance on a real-world
sentiment analysis benchmark.Comment: To appear in Advances in Neural Information Processing Systems 27
(NIPS 2014), Advances in Neural Information Processing Systems 27, Dec. 201
StructVAE: Tree-structured Latent Variable Models for Semi-supervised Semantic Parsing
Semantic parsing is the task of transducing natural language (NL) utterances
into formal meaning representations (MRs), commonly represented as tree
structures. Annotating NL utterances with their corresponding MRs is expensive
and time-consuming, and thus the limited availability of labeled data often
becomes the bottleneck of data-driven, supervised models. We introduce
StructVAE, a variational auto-encoding model for semisupervised semantic
parsing, which learns both from limited amounts of parallel data, and
readily-available unlabeled NL utterances. StructVAE models latent MRs not
observed in the unlabeled data as tree-structured latent variables. Experiments
on semantic parsing on the ATIS domain and Python code generation show that
with extra unlabeled data, StructVAE outperforms strong supervised models.Comment: ACL 201
MaskDGA: A Black-box Evasion Technique Against DGA Classifiers and Adversarial Defenses
Domain generation algorithms (DGAs) are commonly used by botnets to generate
domain names through which bots can establish a resilient communication channel
with their command and control servers. Recent publications presented deep
learning, character-level classifiers that are able to detect algorithmically
generated domain (AGD) names with high accuracy, and correspondingly,
significantly reduce the effectiveness of DGAs for botnet communication. In
this paper we present MaskDGA, a practical adversarial learning technique that
adds perturbation to the character-level representation of algorithmically
generated domain names in order to evade DGA classifiers, without the attacker
having any knowledge about the DGA classifier's architecture and parameters.
MaskDGA was evaluated using the DMD-2018 dataset of AGD names and four recently
published DGA classifiers, in which the average F1-score of the classifiers
degrades from 0.977 to 0.495 when applying the evasion technique. An additional
evaluation was conducted using the same classifiers but with adversarial
defenses implemented: adversarial re-training and distillation. The results of
this evaluation show that MaskDGA can be used for improving the robustness of
the character-level DGA classifiers against adversarial attacks, but that
ideally DGA classifiers should incorporate additional features alongside
character-level features that are demonstrated in this study to be vulnerable
to adversarial attacks.Comment: 12 pages, 2 figure
DRIT++: Diverse Image-to-Image Translation via Disentangled Representations
Image-to-image translation aims to learn the mapping between two visual
domains. There are two main challenges for this task: 1) lack of aligned
training pairs and 2) multiple possible outputs from a single input image. In
this work, we present an approach based on disentangled representation for
generating diverse outputs without paired training images. To synthesize
diverse outputs, we propose to embed images onto two spaces: a domain-invariant
content space capturing shared information across domains and a domain-specific
attribute space. Our model takes the encoded content features extracted from a
given input and attribute vectors sampled from the attribute space to
synthesize diverse outputs at test time. To handle unpaired training data, we
introduce a cross-cycle consistency loss based on disentangled representations.
Qualitative results show that our model can generate diverse and realistic
images on a wide range of tasks without paired training data. For quantitative
evaluations, we measure realism with user study and Fr\'{e}chet inception
distance, and measure diversity with the perceptual distance metric,
Jensen-Shannon divergence, and number of statistically-different bins.Comment: IJCV Journal extension for ECCV 2018 "Diverse Image-to-Image
Translation via Disentangled Representations" arXiv:1808.00948. Project Page:
http://vllab.ucmerced.edu/hylee/DRIT_pp/ Code:
https://github.com/HsinYingLee/DRI
Multi-source weak supervision for saliency detection
The high cost of pixel-level annotations makes it appealing to train saliency
detection models with weak supervision. However, a single weak supervision
source usually does not contain enough information to train a well-performing
model. To this end, we propose a unified framework to train saliency detection
models with diverse weak supervision sources. In this paper, we use category
labels, captions, and unlabelled data for training, yet other supervision
sources can also be plugged into this flexible framework. We design a
classification network (CNet) and a caption generation network (PNet), which
learn to predict object categories and generate captions, respectively,
meanwhile highlight the most important regions for corresponding tasks. An
attention transfer loss is designed to transmit supervision signal between
networks, such that the network designed to be trained with one supervision
source can benefit from another. An attention coherence loss is defined on
unlabelled data to encourage the networks to detect generally salient regions
instead of task-specific regions. We use CNet and PNet to generate pixel-level
pseudo labels to train a saliency prediction network (SNet). During the testing
phases, we only need SNet to predict saliency maps. Experiments demonstrate the
performance of our method compares favourably against unsupervised and weakly
supervised methods and even some supervised methods.Comment: cvpr201
Data Augmentation Using GANs
In this paper we propose the use of Generative Adversarial Networks (GAN) to
generate artificial training data for machine learning tasks. The generation of
artificial training data can be extremely useful in situations such as
imbalanced data sets, performing a role similar to SMOTE or ADASYN. It is also
useful when the data contains sensitive information, and it is desirable to
avoid using the original data set as much as possible (example: medical data).
We test our proposal on benchmark data sets using different network
architectures, and show that a Decision Tree (DT) classifier trained using the
training data generated by the GAN reached the same, (and surprisingly
sometimes better), accuracy and recall than a DT trained on the original data
set.Comment: Submitted for ACML 201
When Provably Secure Steganography Meets Generative Models
Steganography is the art and science of hiding secret messages in public
communication so that the presence of the secret messages cannot be detected.
There are two provably secure steganographic frameworks, one is black-box
sampling based and the other is compression based. The former requires a
perfect sampler which yields data following the same distribution, and the
latter needs explicit distributions of generative objects. However, these two
conditions are too strict even unrealistic in the traditional data environment,
because it is hard to model the explicit distribution of natural image. With
the development of deep learning, generative models bring new vitality to
provably secure steganography, which can serve as the black-box sampler or
provide the explicit distribution of generative media. Motivated by this, this
paper proposes two types of provably secure stegosystems with generative
models. Specifically, we first design block-box sampling based provably secure
stegosystem for broad generative models without explicit distribution, such as
GAN, VAE, and flow-based generative models, where the generative network can
serve as the perfect sampler. For compression based stegosystem, we leverage
the generative models with explicit distribution such as autoregressive models
instead, where the adaptive arithmetic coding plays the role of the perfect
compressor, decompressing the encrypted message bits into generative media, and
the receiver can compress the generative media into the encrypted message bits.
To show the effectiveness of our method, we take DFC-VAE, Glow, WaveNet as
instances of generative models and demonstrate the perfectly secure performance
of these stegosystems with the state-of-the-art steganalysis methods
- …