20 research outputs found
Dynamic Human Evaluation for Relative Model Comparisons
Collecting human judgements is currently the most reliable evaluation method
for natural language generation systems. Automatic metrics have reported flaws
when applied to measure quality aspects of generated text and have been shown
to correlate poorly with human judgements. However, human evaluation is time
and cost-intensive, and we lack consensus on designing and conducting human
evaluation experiments. Thus there is a need for streamlined approaches for
efficient collection of human judgements when evaluating natural language
generation systems. Therefore, we present a dynamic approach to measure the
required number of human annotations when evaluating generated outputs in
relative comparison settings. We propose an agent-based framework of human
evaluation to assess multiple labelling strategies and methods to decide the
better model in a simulation and a crowdsourcing case study. The main results
indicate that a decision about the superior model can be made with high
probability across different labelling strategies, where assigning a single
random worker per task requires the least overall labelling effort and thus the
least cost.Comment: accepted at LREC 202
Distributed Learning over Unreliable Networks
Most of today's distributed machine learning systems assume {\em reliable
networks}: whenever two machines exchange information (e.g., gradients or
models), the network should guarantee the delivery of the message. At the same
time, recent work exhibits the impressive tolerance of machine learning
algorithms to errors or noise arising from relaxed communication or
synchronization. In this paper, we connect these two trends, and consider the
following question: {\em Can we design machine learning systems that are
tolerant to network unreliability during training?} With this motivation, we
focus on a theoretical problem of independent interest---given a standard
distributed parameter server architecture, if every communication between the
worker and the server has a non-zero probability of being dropped, does
there exist an algorithm that still converges, and at what speed? The technical
contribution of this paper is a novel theoretical analysis proving that
distributed learning over unreliable network can achieve comparable convergence
rate to centralized or distributed learning over reliable networks. Further, we
prove that the influence of the packet drop rate diminishes with the growth of
the number of \textcolor{black}{parameter servers}. We map this theoretical
result onto a real-world scenario, training deep neural networks over an
unreliable network layer, and conduct network simulation to validate the system
improvement by allowing the networks to be unreliable
SHiFT: An Efficient, Flexible Search Engine for Transfer Learning
Transfer learning can be seen as a data- and compute-efficient alternative to
training models from scratch. The emergence of rich model repositories, such as
TensorFlow Hub, enables practitioners and researchers to unleash the potential
of these models across a wide range of downstream tasks. As these repositories
keep growing exponentially, efficiently selecting a good model for the task at
hand becomes paramount. By carefully comparing various selection and search
strategies, we realize that no single method outperforms the others, and hybrid
or mixed strategies can be beneficial. Therefore, we propose SHiFT, the first
downstream task-aware, flexible, and efficient model search engine for transfer
learning. These properties are enabled by a custom query language SHiFT-QL
together with a cost-based decision maker, which we empirically validate.
Motivated by the iterative nature of machine learning development, we further
support efficient incremental executions of our queries, which requires a
careful implementation when jointly used with our optimizations
A Data Quality-Driven View of MLOps
Developing machine learning models can be seen as a process similar to the
one established for traditional software development. A key difference between
the two lies in the strong dependency between the quality of a machine learning
model and the quality of the data used to train or perform evaluations. In this
work, we demonstrate how different aspects of data quality propagate through
various stages of machine learning development. By performing a joint analysis
of the impact of well-known data quality dimensions and the downstream machine
learning process, we show that different components of a typical MLOps pipeline
can be efficiently designed, providing both a technical and theoretical
perspective
Decoding EEG brain activity for multi-modal natural language processing
Until recently, human behavioral data from reading has mainly been of
interest to researchers to understand human cognition. However, these human
language processing signals can also be beneficial in machine learning-based
natural language processing tasks. Using EEG brain activity to this purpose is
largely unexplored as of yet. In this paper, we present the first large-scale
study of systematically analyzing the potential of EEG brain activity data for
improving natural language processing tasks, with a special focus on which
features of the signal are most beneficial. We present a multi-modal machine
learning architecture that learns jointly from textual input as well as from
EEG features. We find that filtering the EEG signals into frequency bands is
more beneficial than using the broadband signal. Moreover, for a range of word
embedding types, EEG data improves binary and ternary sentiment classification
and outperforms multiple baselines. For more complex tasks such as relation
detection, further research is needed. Finally, EEG data shows to be
particularly promising when limited training data is available
Co-design Hardware and Algorithm for Vector Search
Vector search has emerged as the foundation for large-scale information
retrieval and machine learning systems, with search engines like Google and
Bing processing tens of thousands of queries per second on petabyte-scale
document datasets by evaluating vector similarities between encoded query texts
and web documents. As performance demands for vector search systems surge,
accelerated hardware offers a promising solution in the post-Moore's Law era.
We introduce \textit{FANNS}, an end-to-end and scalable vector search framework
on FPGAs. Given a user-provided recall requirement on a dataset and a hardware
resource budget, \textit{FANNS} automatically co-designs hardware and
algorithm, subsequently generating the corresponding accelerator. The framework
also supports scale-out by incorporating a hardware TCP/IP stack in the
accelerator. \textit{FANNS} attains up to 23.0 and 37.2 speedup
compared to FPGA and CPU baselines, respectively, and demonstrates superior
scalability to GPUs, achieving 5.5 and 7.6 speedup in median
and 95\textsuperscript{th} percentile (P95) latency within an eight-accelerator
configuration. The remarkable performance of \textit{FANNS} lays a robust
groundwork for future FPGA integration in data centers and AI supercomputers.Comment: 11 page
Building Data-Centric Systems for Machine Learning Development and Operations
Developing machine learning (ML) models can be seen as a process similar to the one established for traditional software development. Over the last years, practitioners have started adopting well-established concepts of classical software engineering to ML projects. One prominent example is MLOps, where tools and techniques from development and operations (DevOps) are transferred to ML to shorten the system development life cycle and provide continuous delivery into production while ensuring high quality of the artifacts.
A key difference between classical software and ML development lies in the strong dependence of ML models on the data used to train or evaluate their performance. Therefore, many tools and best-practices of DevOps can not be directly applied to ML workloads or could lead to lower quality if blindly taken over. In this thesis, we provide three novel data-centric solutions to support data scientists and ML engineers to handle ML workloads in a principled and efficient way: 1. We present Ease.ML/Snoopy, designed to perform a systematic and theoretically founded feasibility study before building ML applications. We approach this problem by estimating the irreducible error of the underlying task, also known as the Bayes error rate (BER), which stems from data quality issues in datasets used to train or evaluate ML model artifacts. We design a practical Bayes error estimator by aggregating over a collection of 1NN-based estimator over publicly available pre-trained feature transformations. Furthermore, by including our systematic feasibility study with additional signals into the iterative label cleaning process, we demonstrate in end-to-end experiments how users are able to save substantial labeling time and monetary efforts. 2. We propose SHiFT, the first task-aware (i.e., taking the dataset into account), flexible, and efficient model search engine for transfer learning, which can be seen as a data- and compute-efficient alternative to training models from scratch, in analogy to code-reuse in classical software engineering. The emergence of rich model repositories, such as TensorFlow Hub, enables practitioners and researchers to unleash the potential of these models across a wide range of tasks. As these repositories keep growing exponentially, efficiently selecting a good model for the task at hand becomes paramount. By carefully comparing various selection and search strategies, we realize that no single method outperforms the others, and hybrid or mixed strategies can be beneficial. The flexibility and efficiency of SHiFT are enabled by a custom query language SHiFT-QL together with a cost-based decision maker. Motivated by the iterative nature of machine learning development, we further support efficient incremental executions of our queries, which requires a careful implementation when jointly used with our optimizations. 3. We introduce Ease.ML/CI, the first continuous integration system for machine learning with statistical guarantees. The challenge of building Ease.ML/CI is to provide these rigorous guarantees (e.g., single accuracy point error tolerance with 0.999 reliability) with a practical amount of labeling effort (e.g., 2000 labels per test). We design a declarative scripting language that allows users to specify integration conditions with reliability constraints, and develop simple novel optimizations that can lower the number of labels required by up to two orders of magnitude for test conditions commonly used in real production systems
Observer dependent lossy image compression
Deep neural networks have recently advanced the state-of-the-art in image compression and surpassed many traditional compression algorithms. The training of such networks involves carefully trading off entropy of the latent representation against reconstruction quality. The term quality crucially depends on the observer of the images which, in the vast majority of literature, is assumed to be human. In this paper, we aim to go beyond this notion of compression quality and look at human visual perception and image classification simultaneously. To that end, we use a family of loss functions that allows to optimize deep image compression depending on the observer and to interpolate between human perceived visual quality and classification accuracy, enabling a more unified view on image compression. Our extensive experiments show that using perceptual loss functions to train a compression system preserves classification accuracy much better than traditional codecs such as BPG without requiring retraining of classifiers on compressed images. For example, compressing ImageNet to 0.25 bpp reduces Inception-ResNet classification accuracy by only 2%. At the same time, when using a human friendly loss function, the same compression system achieves competitive performance in terms of MS-SSIM. By combining these two objective functions, we show that there is a pronounced trade-off in compression quality between the human visual system and classification accuracy