148 research outputs found
Spectral Methods from Tensor Networks
A tensor network is a diagram that specifies a way to "multiply" a collection
of tensors together to produce another tensor (or matrix). Many existing
algorithms for tensor problems (such as tensor decomposition and tensor PCA),
although they are not presented this way, can be viewed as spectral methods on
matrices built from simple tensor networks. In this work we leverage the full
power of this abstraction to design new algorithms for certain continuous
tensor decomposition problems.
An important and challenging family of tensor problems comes from orbit
recovery, a class of inference problems involving group actions (inspired by
applications such as cryo-electron microscopy). Orbit recovery problems over
finite groups can often be solved via standard tensor methods. However, for
infinite groups, no general algorithms are known. We give a new spectral
algorithm based on tensor networks for one such problem: continuous
multi-reference alignment over the infinite group SO(2). Our algorithm extends
to the more general heterogeneous case.Comment: 30 pages, 8 figure
Smoothed Analysis of Tensor Decompositions
Low rank tensor decompositions are a powerful tool for learning generative
models, and uniqueness results give them a significant advantage over matrix
decomposition methods. However, tensors pose significant algorithmic challenges
and tensors analogs of much of the matrix algebra toolkit are unlikely to exist
because of hardness results. Efficient decomposition in the overcomplete case
(where rank exceeds dimension) is particularly challenging. We introduce a
smoothed analysis model for studying these questions and develop an efficient
algorithm for tensor decomposition in the highly overcomplete case (rank
polynomial in the dimension). In this setting, we show that our algorithm is
robust to inverse polynomial error -- a crucial property for applications in
learning since we are only allowed a polynomial number of samples. While
algorithms are known for exact tensor decomposition in some overcomplete
settings, our main contribution is in analyzing their stability in the
framework of smoothed analysis.
Our main technical contribution is to show that tensor products of perturbed
vectors are linearly independent in a robust sense (i.e. the associated matrix
has singular values that are at least an inverse polynomial). This key result
paves the way for applying tensor methods to learning problems in the smoothed
setting. In particular, we use it to obtain results for learning multi-view
models and mixtures of axis-aligned Gaussians where there are many more
"components" than dimensions. The assumption here is that the model is not
adversarially chosen, formalized by a perturbation of model parameters. We
believe this an appealing way to analyze realistic instances of learning
problems, since this framework allows us to overcome many of the usual
limitations of using tensor methods.Comment: 32 pages (including appendix
Fourier PCA and Robust Tensor Decomposition
Fourier PCA is Principal Component Analysis of a matrix obtained from higher
order derivatives of the logarithm of the Fourier transform of a
distribution.We make this method algorithmic by developing a tensor
decomposition method for a pair of tensors sharing the same vectors in rank-
decompositions. Our main application is the first provably polynomial-time
algorithm for underdetermined ICA, i.e., learning an matrix
from observations where is drawn from an unknown product
distribution with arbitrary non-Gaussian components. The number of component
distributions can be arbitrarily higher than the dimension and the
columns of only need to satisfy a natural and efficiently verifiable
nondegeneracy condition. As a second application, we give an alternative
algorithm for learning mixtures of spherical Gaussians with linearly
independent means. These results also hold in the presence of Gaussian noise.Comment: Extensively revised; details added; minor errors corrected;
exposition improve
AutoBiasTest: Controllable Sentence Generation for Automated and Open-Ended Social Bias Testing in Language Models
Social bias in Pretrained Language Models (PLMs) affects text generation and
other downstream NLP tasks. Existing bias testing methods rely predominantly on
manual templates or on expensive crowd-sourced data. We propose a novel
AutoBiasTest method that automatically generates sentences for testing bias in
PLMs, hence providing a flexible and low-cost alternative. Our approach uses
another PLM for generation and controls the generation of sentences by
conditioning on social group and attribute terms. We show that generated
sentences are natural and similar to human-produced content in terms of word
length and diversity. We illustrate that larger models used for generation
produce estimates of social bias with lower variance. We find that our bias
scores are well correlated with manual templates, but AutoBiasTest highlights
biases not captured by these templates due to more diverse and realistic test
sentences. By automating large-scale test sentence generation, we enable better
estimation of underlying bias distribution
Finding Social Media Trolls: Dynamic Keyword Selection Methods for Rapidly-Evolving Online Debates
Online harassment is a significant social problem. Prevention of online
harassment requires rapid detection of harassing, offensive, and negative
social media posts. In this paper, we propose the use of word embedding models
to identify offensive and harassing social media messages in two aspects:
detecting fast-changing topics for more effective data collection and
representing word semantics in different domains. We demonstrate with
preliminary results that using the GloVe (Global Vectors for Word
Representation) model facilitates the discovery of new and relevant keywords to
use for data collection and trolling detection. Our paper concludes with a
discussion of a research agenda to further develop and test word embedding
models for identification of social media harassment and trolling.Comment: AI for Social Good workshop at NeurIPS (2019
OrbNet: Deep learning for quantum chemistry using symmetry-adapted atomic-orbital features
We introduce a machine learning method in which energy solutions from the Schrödinger equation are predicted using symmetry adapted atomic orbital features and a graph neural-network architecture. OrbNet is shown to outperform existing methods in terms of learning efficiency and transferability for the prediction of density functional theory results while employing low-cost features that are obtained from semi-empirical electronic structure calculations. For applications to datasets of drug-like molecules, including QM7b-T, QM9, GDB-13-T, DrugBank, and the conformer benchmark dataset of Folmsbee and Hutchison [Int. J. Quantum Chem. (published online) (2020)], OrbNet predicts energies within chemical accuracy of density functional theory at a computational cost that is 1000-fold or more reduced
Can You Label Less by Using Out-of-Domain Data? Active & Transfer Learning with Few-shot Instructions
Labeling social-media data for custom dimensions of toxicity and social bias
is challenging and labor-intensive. Existing transfer and active learning
approaches meant to reduce annotation effort require fine-tuning, which suffers
from over-fitting to noise and can cause domain shift with small sample sizes.
In this work, we propose a novel Active Transfer Few-shot Instructions (ATF)
approach which requires no fine-tuning. ATF leverages the internal linguistic
knowledge of pre-trained language models (PLMs) to facilitate the transfer of
information from existing pre-labeled datasets (source-domain task) with
minimum labeling effort on unlabeled target data (target-domain task). Our
strategy can yield positive transfer achieving a mean AUC gain of 10.5%
compared to no transfer with a large 22b parameter PLM. We further show that
annotation of just a few target-domain samples via active learning can be
beneficial for transfer, but the impact diminishes with more annotation effort
(26% drop in gain between 100 and 2000 annotated examples). Finally, we find
that not all transfer scenarios yield a positive gain, which seems related to
the PLMs initial performance on the target-domain task.Comment: Accepted to NeurIPS Workshop on Transfer Learning for Natural
Language Processing, 2022, New Orlean
Multi-task learning for electronic structure to predict and explore molecular potential energy surfaces
We refine the OrbNet model to accurately predict energy, forces, and other response properties for molecules using a graph neural-network architecture based on features from low-cost approximated quantum operators in the symmetry-adapted atomic orbital basis. The model is end-to-end differentiable due to the derivation of analytic gradients for all electronic structure terms, and is shown to be transferable across chemical space due to the use of domain-specific features. The learning efficiency is improved by incorporating physically motivated constraints on the electronic structure through multi-task learning. The model outperforms existing methods on energy prediction tasks for the QM9 dataset and for molecular geometry optimizations on conformer datasets, at a computational cost that is thousand-fold or more reduced compared to conventional quantum-chemistry calculations (such as density functional theory) that offer similar accuracy
- …