3,514 research outputs found
Revisiting Tropical Polynomial Division: Theory, Algorithms and Application to Neural Networks
Tropical geometry has recently found several applications in the analysis of
neural networks with piecewise linear activation functions. This paper presents
a new look at the problem of tropical polynomial division and its application
to the simplification of neural networks. We analyze tropical polynomials with
real coefficients, extending earlier ideas and methods developed for
polynomials with integer coefficients. We first prove the existence of a unique
quotient-remainder pair and characterize the quotient in terms of the convex
bi-conjugate of a related function. Interestingly, the quotient of tropical
polynomials with integer coefficients does not necessarily have integer
coefficients. Furthermore, we develop a relationship of tropical polynomial
division with the computation of the convex hull of unions of convex polyhedra
and use it to derive an exact algorithm for tropical polynomial division. An
approximate algorithm is also presented, based on an alternation between data
partition and linear programming. We also develop special techniques to divide
composite polynomials, described as sums or maxima of simpler ones. Finally, we
present some numerical results to illustrate the efficiency of the algorithms
proposed, using the MNIST handwritten digit and CIFAR-10 datasets
Alternating Minimization for Regression with Tropical Rational Functions
We propose an alternating minimization heuristic for regression over the
space of tropical rational functions with fixed exponents. The method
alternates between fitting the numerator and denominator terms via tropical
polynomial regression, which is known to admit a closed form solution. We
demonstrate the behavior of the alternating minimization method experimentally.
Experiments demonstrate that the heuristic provides a reasonable approximation
of the input data. Our work is motivated by applications to ReLU neural
networks, a popular class of network architectures in the machine learning
community which are closely related to tropical rational functions
Adapter Pruning using Tropical Characterization
Adapters are widely popular parameter-efficient transfer learning approaches
in natural language processing that insert trainable modules in between layers
of a pre-trained language model. Apart from several heuristics, however, there
has been a lack of studies analyzing the optimal number of adapter parameters
needed for downstream applications. In this paper, we propose an adapter
pruning approach by studying the tropical characteristics of trainable modules.
We cast it as an optimization problem that aims to prune parameters from the
adapter layers without changing the orientation of underlying tropical
hypersurfaces. Our experiments on five NLP datasets show that tropical geometry
tends to identify more relevant parameters to prune when compared with the
magnitude-based baseline, while a combined approach works best across the
tasks.Comment: Accepted at EMNLP 2023, Finding
- …