18 research outputs found
Invariances and Data Augmentation for Supervised Music Transcription
This paper explores a variety of models for frame-based music transcription,
with an emphasis on the methods needed to reach state-of-the-art on human
recordings. The translation-invariant network discussed in this paper, which
combines a traditional filterbank with a convolutional neural network, was the
top-performing model in the 2017 MIREX Multiple Fundamental Frequency
Estimation evaluation. This class of models shares parameters in the
log-frequency domain, which exploits the frequency invariance of music to
reduce the number of model parameters and avoid overfitting to the training
data. All models in this paper were trained with supervision by labeled data
from the MusicNet dataset, augmented by random label-preserving pitch-shift
transformations.Comment: 6 page
Robust Distortion-free Watermarks for Language Models
We propose a methodology for planting watermarks in text from an
autoregressive language model that are robust to perturbations without changing
the distribution over text up to a certain maximum generation budget. We
generate watermarked text by mapping a sequence of random numbers -- which we
compute using a randomized watermark key -- to a sample from the language
model. To detect watermarked text, any party who knows the key can align the
text to the random number sequence. We instantiate our watermark methodology
with two sampling schemes: inverse transform sampling and exponential minimum
sampling. We apply these watermarks to three language models -- OPT-1.3B,
LLaMA-7B and Alpaca-7B -- to experimentally validate their statistical power
and robustness to various paraphrasing attacks. Notably, for both the OPT-1.3B
and LLaMA-7B models, we find we can reliably detect watermarked text () from tokens even after corrupting between -\% of the tokens
via random edits (i.e., substitutions, insertions or deletions). For the
Alpaca-7B model, we conduct a case study on the feasibility of watermarking
responses to typical user instructions. Due to the lower entropy of the
responses, detection is more difficult: around of the responses -- whose
median length is around tokens -- are detectable with , and
the watermark is also less robust to certain automated paraphrasing attacks we
implement
MAUVE Scores for Generative Models: Theory and Practice
Generative artificial intelligence has made significant strides, producing
text indistinguishable from human prose and remarkably photorealistic images.
Automatically measuring how close the generated data distribution is to the
target distribution is central to diagnosing existing models and developing
better ones. We present MAUVE, a family of comparison measures between pairs of
distributions such as those encountered in the generative modeling of text or
images. These scores are statistical summaries of divergence frontiers
capturing two types of errors in generative modeling. We explore three
approaches to statistically estimate these scores: vector quantization,
non-parametric estimation, and classifier-based estimation. We provide
statistical bounds for the vector quantization approach.
Empirically, we find that the proposed scores paired with a range of
-divergences and statistical estimation methods can quantify the gaps
between the distributions of human-written text and those of modern neural
language models by correlating with human judgments and identifying known
properties of the generated texts. We demonstrate in the vision domain that
MAUVE can identify known properties of generated images on par with or better
than existing metrics. In conclusion, we present practical recommendations for
using MAUVE effectively with language and image modalities.Comment: Published in Journal of Machine Learning Researc
Evaluating Human-Language Model Interaction
Many real-world applications of language models (LMs), such as writing
assistance and code autocomplete, involve human-LM interaction. However, most
benchmarks are non-interactive in that a model produces output without human
involvement. To evaluate human-LM interaction, we develop a new framework,
Human-AI Language-based Interaction Evaluation (HALIE), that defines the
components of interactive systems and dimensions to consider when designing
evaluation metrics. Compared to standard, non-interactive evaluation, HALIE
captures (i) the interactive process, not only the final output; (ii) the
first-person subjective experience, not just a third-party assessment; and
(iii) notions of preference beyond quality (e.g., enjoyment and ownership). We
then design five tasks to cover different forms of interaction: social
dialogue, question answering, crossword puzzles, summarization, and metaphor
generation. With four state-of-the-art LMs (three variants of OpenAI's GPT-3
and AI21 Labs' Jurassic-1), we find that better non-interactive performance
does not always translate to better human-LM interaction. In particular, we
highlight three cases where the results from non-interactive and interactive
metrics diverge and underscore the importance of human-LM interaction for LM
evaluation.Comment: Authored by the Center for Research on Foundation Models (CRFM) at
the Stanford Institute for Human-Centered Artificial Intelligence (HAI