72 research outputs found
Finding NEEMo: Geometric Fitting using Neural Estimation of the Energy Mover's Distance
A novel neural architecture was recently developed that enforces an exact
upper bound on the Lipschitz constant of the model by constraining the norm of
its weights in a minimal way, resulting in higher expressiveness compared to
other techniques. We present a new and interesting direction for this
architecture: estimation of the Wasserstein metric (Earth Mover's Distance) in
optimal transport by employing the Kantorovich-Rubinstein duality to enable its
use in geometric fitting applications. Specifically, we focus on the field of
high-energy particle physics, where it has been shown that a metric for the
space of particle-collider events can be defined based on the Wasserstein
metric, referred to as the Energy Mover's Distance (EMD). This metrization has
the potential to revolutionize data-driven collider phenomenology. The work
presented here represents a major step towards realizing this goal by providing
a differentiable way of directly calculating the EMD. We show how the
flexibility that our approach enables can be used to develop novel clustering
algorithms.Comment: 5 pages, 4 figure
CLIP: Cheap Lipschitz Training of Neural Networks
Despite the large success of deep neural networks (DNN) in recent years, most
neural networks still lack mathematical guarantees in terms of stability. For
instance, DNNs are vulnerable to small or even imperceptible input
perturbations, so called adversarial examples, that can cause false
predictions. This instability can have severe consequences in applications
which influence the health and safety of humans, e.g., biomedical imaging or
autonomous driving. While bounding the Lipschitz constant of a neural network
improves stability, most methods rely on restricting the Lipschitz constants of
each layer which gives a poor bound for the actual Lipschitz constant.
In this paper we investigate a variational regularization method named CLIP
for controlling the Lipschitz constant of a neural network, which can easily be
integrated into the training procedure. We mathematically analyze the proposed
model, in particular discussing the impact of the chosen regularization
parameter on the output of the network. Finally, we numerically evaluate our
method on both a nonlinear regression problem and the MNIST and Fashion-MNIST
classification databases, and compare our results with a weight regularization
approach.Comment: 12 pages, 2 figures, accepted at SSVM 202
Expressive Monotonic Neural Networks
The monotonic dependence of the outputs of a neural network on some of its
inputs is a crucial inductive bias in many scenarios where domain knowledge
dictates such behavior. This is especially important for interpretability and
fairness considerations. In a broader context, scenarios in which monotonicity
is important can be found in finance, medicine, physics, and other disciplines.
It is thus desirable to build neural network architectures that implement this
inductive bias provably. In this work, we propose a weight-constrained
architecture with a single residual connection to achieve exact monotonic
dependence in any subset of the inputs. The weight constraint scheme directly
controls the Lipschitz constant of the neural network and thus provides the
additional benefit of robustness. Compared to currently existing techniques
used for monotonicity, our method is simpler in implementation and in theory
foundations, has negligible computational overhead, is guaranteed to produce
monotonic dependence, and is highly expressive. We show how the algorithm is
used to train powerful, robust, and interpretable discriminators that achieve
competitive performance compared to current state-of-the-art methods across
various benchmarks, from social applications to the classification of the
decays of subatomic particles produced at the CERN Large Hadron Collider.Comment: 9 pages, 4 figures, ICLR 2023 final submissio
- …