Sub-Weibull distributions: generalizing sub-Gaussian and sub-Exponential properties to heavier-tailed distributions

We propose the notion of sub-Weibull distributions, which are characterised by tails lighter than (or equally light as) the right tail of a Weibull distribution. This novel class generalises the sub-Gaussian and sub-Exponential families to potentially heavier-tailed distributions. Sub-Weibull distributions are parameterized by a positive tail index $\theta$ and reduce to sub-Gaussian distributions for $\theta=1/2$ and to sub-Exponential distributions for $\theta=1$. A characterisation of the sub-Weibull property based on moments and on the moment generating function is provided and properties of the class are studied. An estimation procedure for the tail parameter is proposed and is applied to an example stemming from Bayesian deep learning.Comment: 10 pages, 3 figure

A Primer on Bayesian Neural Networks: Review and Debates

Neural networks have achieved remarkable performance across various problem domains, but their widespread applicability is hindered by inherent limitations such as overconfidence in predictions, lack of interpretability, and vulnerability to adversarial attacks. To address these challenges, Bayesian neural networks (BNNs) have emerged as a compelling extension of conventional neural networks, integrating uncertainty estimation into their predictive capabilities. This comprehensive primer presents a systematic introduction to the fundamental concepts of neural networks and Bayesian inference, elucidating their synergistic integration for the development of BNNs. The target audience comprises statisticians with a potential background in Bayesian methods but lacking deep learning expertise, as well as machine learners proficient in deep neural networks but with limited exposure to Bayesian statistics. We provide an overview of commonly employed priors, examining their impact on model behavior and performance. Additionally, we delve into the practical considerations associated with training and inference in BNNs. Furthermore, we explore advanced topics within the realm of BNN research, acknowledging the existence of ongoing debates and controversies. By offering insights into cutting-edge developments, this primer not only equips researchers and practitioners with a solid foundation in BNNs, but also illuminates the potential applications of this dynamic field. As a valuable resource, it fosters an understanding of BNNs and their promising prospects, facilitating further advancements in the pursuit of knowledge and innovation.Comment: 65 page

Bayesian neural network priors at the level of units

International audienc

Bayesian neural networks become heavier-tailed with depth

International audienceWe investigate deep Bayesian neural networks with Gaussian priors on the weights and ReLU-like nonlinearities, shedding light on novel distribution properties at the level of the neural network units. The main thrust of the paper is to establish that the prior distribution induced on the units before and after activation becomes increasingly heavier-tailed with depth. We show that first layer units are Gaussian, second layer units are sub-Exponential, and we introduce sub-Weibull distributions to characterize the deeper layers units. This result provides new theoretical insight on deep Bayesian neural networks, underpinning their practical potential. The workshop paper is based on the original paper Vladimirova et al. (2018)

Dependence between Bayesian neural network units

International audienceThe connection between Bayesian neural networks and Gaussian processes gained a lot of attention in the last few years, with the flagship result that hidden units converge to a Gaussian process limit when the layers width tends to infinity. Underpinning this result is the fact that hidden units become independent in the infinite-width limit. Our aim is to shed some light on hidden units dependence properties in practical finite-width Bayesian neural networks. In addition to theoretical results, we assess empirically the depth and width impacts on hidden units dependence properties

Bayesian neural networks increasingly sparsify their units with depth

We investigate deep Bayesian neural networks with Gaussian priors on the weights and ReLU-like nonlinearities, shedding light on novel sparsity-inducing mechanisms at the level of the units of the network, both pre-and post-nonlinearities. The main thrust of the paper is to establish that the units prior distribution becomes increasingly heavy-tailed with depth. We show that first layer units are Gaussian, second layer units are sub-Exponential, and we introduce sub-Weibull distributions to characterize the deeper layers units. Bayesian neural networks with Gaussian priors are well known to induce the weight decay penalty on the weights. In contrast, our result indicates a more elaborate regularization scheme at the level of the units, ranging from convex penalties for the first two layers-weight decay for the first and Lasso for the second to non convex penalties for deeper layers. Thus, despite weight decay does not allow for the weights to be set exactly to zero, sparse solutions tend to be selected for the units from the second layer onward. This result provides new theoretical insight on deep Bayesian neural networks, underpinning their natural shrinkage properties and practical potential

Bayesian neural network priors at the level of units

International audienceWe investigate deep Bayesian neural networks with Gaussian priors on the weights and ReLU-like nonlinearities, shedding light on novel sparsity-inducing mechanisms at the level of the units of the network. Bayesian neural networks with Gaussian priors are well known to induce the weight decay penalty on the weights. In contrast, our result indicates a more elaborate regularization scheme at the level of the units, ranging from convex penalties for the first two layers-L 2 regularization for the first and Lasso for the second-to non convex penalties for deeper layers. Thus, although weight decay does not allow for the weights to be set exactly to zero, sparse solutions tend to be selected for the units from the second layer onward. This result provides new theoretical insight on deep Bayesian neural networks, underpinning their natural shrinkage properties and practical potential

Distributional properties of Bayesian neural networks

Les réseaux neuronaux (RN) sont des outils efficaces qui atteignent des performances de pointe dans divers problèmes, notamment la vision par ordinateur, la reconnaissance d'objets et le traitement du langage naturel. Cependant, la principale préoccupation concernant les réseaux neuronaux est leur nature de boîte noire, car il n'existe toujours pas de théorie permettant de retracer le chemin entre l'entrée et la sortie. Comme les résultats des RNs ne peuvent pas être expliqués intuitivement, ils sont perçus comme peu fiables pour certaines applications, en particulier dans les traitements médicaux et les voitures à conduite autonome. L'inférence bayésienne est considérée comme l'une des solutions à la fiabilité car elle permet de fournir une certaine incertitude pour les sorties. Cependant, l'approche bayésienne n'ouvre pas la boîte noire des modèles basés sur les RN.Cette thèse aborde la modélisation bayésienne des RNs en caractérisant les distributions dans l'espace des fonctions qui apparaissent dans les RNs avant l'entraînement. La connexion entre les RNs bayésiens et les processus gaussiens a gagné beaucoup d'attention ces dernières années. Le résultat phare est que les prieurs dans l'espace des fonctions convergent vers un processus gaussien lorsque la largeur des couches tend vers l'infini. Nous étendons ce résultat aux réseaux bayésiens de largeur finie en fournissant une caractérisation de la distribution marginale des priorités des unités. Nous fournissons une caractérisation précise des queues des unités cachées par le biais de descriptions des queues de sous-Weibull et de Weibull. Les résultats obtenus illustrent la nature à queue lourde des unités cachées dans les couches profondes pour différentes priorités de poids. Enfin, nous décrivons analytiquement et empiriquement la dépendance positive et négative induite entre les unités cachées pour les RNs bayésiens de différentes largeurs et profondeurs. Nous pensons que ces caractérisations aident à comprendre les mécanismes internes des RNs et à suggérer des améliorations du modèle.Neural networks (NNs) are efficient tools that achieve state-of-the-art performance in various problems including computer vision, object recognition, natural language processing. However, the primary concern about NNs is their black-box nature due to the fact that some theory that could trace back the path from input to output is still lacking. Since NNs' output cannot be intuitively explained, they are perceived as being unreliable for some applications, especially in medical treatment and self-driving cars. Bayesian inference is considered to be one of the solutions to trustworthiness as it allows to provide some uncertainty for the outputs. However, the Bayesian approach does not open the black box of NN-based models.This thesis addresses the Bayesian modeling of NNs by characterizing the distributions in function space arising in NNs before training. The connection between Bayesian NNs and Gaussian processes gained much attention in the last few years. The flagship result is that function-space priors converge to a Gaussian process when the layers' width tends to infinity. We extend this result to finite-width Bayesian NNs by providing a characterization of the marginal prior distribution of the units. We provide an accurate characterization of hidden units tails through sub-Weibull and Weibull-tail descriptions. The obtained results illustrate the heavy-tailed nature of hidden units in deep layers for different weight priors. Finally, we describe analytically and empirically the induced positive and negative dependence between hidden units for Bayesian NNs of different widths and depths. We believe that these characterizations help to understand the internal mechanisms of NNs and to suggest model improvements

XX: Bordando una mujer de Instagram.

[ES] En el presente proyecto hemos querido reflexionar sobre la imagen femenina en las redes sociales a través del bordado de punto de cruz, tomando como referencia unas imágenes de Instagram. Hemos planteado, además, profundizar en las posibilidades estéticas del punto de cruz en el contexto del arte contemporáneo, en tanto que es una práctica artesanal ejercida tradicionalmente por las mujeres. Este proyecto surge a partir del interés en la técnica en sí y con la intención de analizar un tema actual, contando sobre la experiencia propia a través del trabajo práctico y una reflexión escrita. Después de haber utilizado dicha técnica ocasionalmente en diferentes prácticas artísticas a lo largo de la carrera, continúa dándonos posibilidades para expresar algunas cuestiones que nos interesan.[EN] In this project we wanted to reflect on the female image in social media through cross-stitching embroidery, taking real images from Instagram as a reference. We have also proposed to delve into aesthetic possibilities of cross-stitching as a valid medium of contemporary art, bearing in mind that it is a craft practice traditionally attributed to women. This project stems from the interest in the technique itself and the philosophy of the aforementioned topic. Our own experience is also reflected in both practical work and written material. This technique was applied occasionally in different artistic practices throughout the studies and as means of self-expression. It continues to give us possibilities to tackle certain matters of interest.Vladimirova, M. (2021). XX: Bordando una mujer de Instagram. Universitat Politècnica de València. http://hdl.handle.net/10251/169589TFG

Bayesian neural networks become heavier-tailed with depth

International audienceWe investigate deep Bayesian neural networks with Gaussian priors on the weights and ReLU-like nonlinearities, shedding light on novel distribution properties at the level of the neural network units. The main thrust of the paper is to establish that the prior distribution induced on the units before and after activation becomes increasingly heavier-tailed with depth. We show that first layer units are Gaussian, second layer units are sub-Exponential, and we introduce sub-Weibull distributions to characterize the deeper layers units. This result provides new theoretical insight on deep Bayesian neural networks, underpinning their practical potential. The workshop paper is based on the original paper Vladimirova et al. (2018)