213 research outputs found
PAC-Bayesian High Dimensional Bipartite Ranking
This paper is devoted to the bipartite ranking problem, a classical
statistical learning task, in a high dimensional setting. We propose a scoring
and ranking strategy based on the PAC-Bayesian approach. We consider nonlinear
additive scoring functions, and we derive non-asymptotic risk bounds under a
sparsity assumption. In particular, oracle inequalities in probability holding
under a margin condition assess the performance of our procedure, and prove its
minimax optimality. An MCMC-flavored algorithm is proposed to implement our
method, along with its behavior on synthetic and real-life datasets
An Oracle Inequality for Quasi-Bayesian Non-Negative Matrix Factorization
The aim of this paper is to provide some theoretical understanding of
quasi-Bayesian aggregation methods non-negative matrix factorization. We derive
an oracle inequality for an aggregated estimator. This result holds for a very
general class of prior distributions and shows how the prior affects the rate
of convergence.Comment: This is the corrected version of the published paper P. Alquier, B.
Guedj, An Oracle Inequality for Quasi-Bayesian Non-negative Matrix
Factorization, Mathematical Methods of Statistics, 2017, vol. 26, no. 1, pp.
55-67. Since then Arnak Dalalyan (ENSAE) found a mistake in the proofs. We
fixed the mistake at the price of a slightly different logarithmic term in
the boun
Pycobra: A Python Toolbox for Ensemble Learning and Visualisation
We introduce \texttt{pycobra}, a Python library devoted to ensemble learning
(regression and classification) and visualisation. Its main assets are the
implementation of several ensemble learning algorithms, a flexible and generic
interface to compare and blend any existing machine learning algorithm
available in Python libraries (as long as a \texttt{predict} method is given),
and visualisation tools such as Voronoi tessellations. \texttt{pycobra} is
fully \texttt{scikit-learn} compatible and is released under the MIT
open-source license. \texttt{pycobra} can be downloaded from the Python Package
Index (PyPi) and Machine Learning Open Source Software (MLOSS). The current
version (along with Jupyter notebooks, extensive documentation, and continuous
integration tests) is available at
\href{https://github.com/bhargavvader/pycobra}{https://github.com/bhargavvader/pycobra}
and official documentation website is
\href{https://modal.lille.inria.fr/pycobra}{https://modal.lille.inria.fr/pycobra}
A Quasi-Bayesian Perspective to Online Clustering
When faced with high frequency streams of data, clustering raises theoretical
and algorithmic pitfalls. We introduce a new and adaptive online clustering
algorithm relying on a quasi-Bayesian approach, with a dynamic (i.e.,
time-dependent) estimation of the (unknown and changing) number of clusters. We
prove that our approach is supported by minimax regret bounds. We also provide
an RJMCMC-flavored implementation (called PACBO, see
https://cran.r-project.org/web/packages/PACBO/index.html) for which we give a
convergence guarantee. Finally, numerical experiments illustrate the potential
of our procedure
Wasserstein PAC-Bayes Learning: A Bridge Between Generalisation and Optimisation
PAC-Bayes learning is an established framework to assess the generalisation
ability of learning algorithm during the training phase. However, it remains
challenging to know whether PAC-Bayes is useful to understand, before training,
why the output of well-known algorithms generalise well. We positively answer
this question by expanding the \emph{Wasserstein PAC-Bayes} framework, briefly
introduced in \cite{amit2022ipm}. We provide new generalisation bounds
exploiting geometric assumptions on the loss function. Using our framework, we
prove, before any training, that the output of an algorithm from
\citet{lambert2022variational} has a strong asymptotic generalisation ability.
More precisely, we show that it is possible to incorporate optimisation results
within a generalisation framework, building a bridge between PAC-Bayes and
optimisation algorithms
Sequential Learning of Principal Curves: Summarizing Data Streams on the Fly
When confronted with massive data streams, summarizing data with dimension
reduction methods such as PCA raises theoretical and algorithmic pitfalls.
Principal curves act as a nonlinear generalization of PCA and the present paper
proposes a novel algorithm to automatically and sequentially learn principal
curves from data streams. We show that our procedure is supported by regret
bounds with optimal sublinear remainder terms. A greedy local search
implementation (called \texttt{slpc}, for Sequential Learning Principal Curves)
that incorporates both sleeping experts and multi-armed bandit ingredients is
presented, along with its regret computation and performance on synthetic and
real-life data
Differentiable PAC–Bayes Objectives with Partially Aggregated Neural Networks
We make two related contributions motivated by the challenge of training stochastic neural networks, particularly in a PAC–Bayesian setting: (1) we show how averaging over an ensemble of stochastic neural networks enables a new class of partially-aggregated estimators, proving that these lead to unbiased lower-variance output and gradient estimators; (2) we reformulate a PAC–Bayesian bound for signed-output networks to derive in combination with the above a directly optimisable, differentiable objective and a generalisation guarantee, without using a surrogate loss or loosening the bound. We show empirically that this leads to competitive generalisation guarantees and compares favourably to other methods for training such networks. Finally, we note that the above leads to a simpler PAC–Bayesian training scheme for sign-activation networks than previous work
PAC-Bayes Generalisation Bounds for Heavy-Tailed Losses through Supermartingales
While PAC-Bayes is now an established learning framework for light-tailed
losses (\emph{e.g.}, subgaussian or subexponential), its extension to the case
of heavy-tailed losses remains largely uncharted and has attracted a growing
interest in recent years. We contribute PAC-Bayes generalisation bounds for
heavy-tailed losses under the sole assumption of bounded variance of the loss
function. Under that assumption, we extend previous results from
\citet{kuzborskij2019efron}. Our key technical contribution is exploiting an
extention of Markov's inequality for supermartingales. Our proof technique
unifies and extends different PAC-Bayesian frameworks by providing bounds for
unbounded martingales as well as bounds for batch and online learning with
heavy-tailed losses.Comment: New Section 3 on Online PAC-Baye
Comparing Comparators in Generalization Bounds
We derive generic information-theoretic and PAC-Bayesian generalization
bounds involving an arbitrary convex comparator function, which measures the
discrepancy between the training and population loss. The bounds hold under the
assumption that the cumulant-generating function (CGF) of the comparator is
upper-bounded by the corresponding CGF within a family of bounding
distributions. We show that the tightest possible bound is obtained with the
comparator being the convex conjugate of the CGF of the bounding distribution,
also known as the Cram\'er function. This conclusion applies more broadly to
generalization bounds with a similar structure. This confirms the
near-optimality of known bounds for bounded and sub-Gaussian losses and leads
to novel bounds under other bounding distributions.Comment: AISTATS 202
Differentiable PAC-Bayes Objectives with Partially Aggregated Neural Networks
We make three related contributions motivated by the challenge of training
stochastic neural networks, particularly in a PAC-Bayesian setting: (1) we show
how averaging over an ensemble of stochastic neural networks enables a new
class of \emph{partially-aggregated} estimators; (2) we show that these lead to
provably lower-variance gradient estimates for non-differentiable signed-output
networks; (3) we reformulate a PAC-Bayesian bound for these networks to derive
a directly optimisable, differentiable objective and a generalisation
guarantee, without using a surrogate loss or loosening the bound. This bound is
twice as tight as that of Letarte et al. (2019) on a similar network type. We
show empirically that these innovations make training easier and lead to
competitive guarantees
- …