95 research outputs found
Debugging Machine Learning Pipelines
Machine learning tasks entail the use of complex computational pipelines to
reach quantitative and qualitative conclusions. If some of the activities in a
pipeline produce erroneous or uninformative outputs, the pipeline may fail or
produce incorrect results. Inferring the root cause of failures and unexpected
behavior is challenging, usually requiring much human thought, and is both
time-consuming and error-prone. We propose a new approach that makes use of
iteration and provenance to automatically infer the root causes and derive
succinct explanations of failures. Through a detailed experimental evaluation,
we assess the cost, precision, and recall of our approach compared to the state
of the art. Our source code and experimental data will be available for
reproducibility and enhancement.Comment: 10 page
A Theory of the Acquisition of Episodic Memory
Case-based reasoning (CBR) has been viewed by many as just a methodology for building systems, but the foundations of CBR are psychological theories. Dynamic Memory (Schank, 1982) was the first attempt to describe a theory for learning in computers and people, based on particular forms of data structures and processes, that nowadays are widely used in a variety of forms in CBR. In addition to being useful for system building, CBR provides a way of discussing a range of issues concerned with cognition. This focus on the practical uses of CBR has deflected attention from the need to develop further the underlying theory. In particular, the issue of knowledge acquisition, in not adequately handled by the existing theory. This paper discusses this theoretical weakness and then proposes an enhanced model of learning which is compatible with the CBR paradigm
MDB: Interactively Querying Datasets and Models
As models are trained and deployed, developers need to be able to
systematically debug errors that emerge in the machine learning pipeline. We
present MDB, a debugging framework for interactively querying datasets and
models. MDB integrates functional programming with relational algebra to build
expressive queries over a database of datasets and model predictions. Queries
are reusable and easily modified, enabling debuggers to rapidly iterate and
refine queries to discover and characterize errors and model behaviors. We
evaluate MDB on object detection, bias discovery, image classification, and
data imputation tasks across self-driving videos, large language models, and
medical records. Our experiments show that MDB enables up to 10x faster and
40\% shorter queries than other baselines. In a user study, we find developers
can successfully construct complex queries that describe errors of machine
learning models
- …