From Mutual Information to Expected Dynamics: New Generalization Bounds for Heavy-Tailed SGD

Understanding the generalization abilities of modern machine learning algorithms has been a major research topic over the past decades. In recent years, the learning dynamics of Stochastic Gradient Descent (SGD) have been related to heavy-tailed dynamics. This has been successfully applied to generalization theory by exploiting the fractal properties of those dynamics. However, the derived bounds depend on mutual information (decoupling) terms that are beyond the reach of computability. In this work, we prove generalization bounds over the trajectory of a class of heavy-tailed dynamics, without those mutual information terms. Instead, we introduce a geometric decoupling term by comparing the learning dynamics (depending on the empirical risk) with an expected one (depending on the population risk). We further upper-bound this geometric term, by using techniques from the heavy-tailed and the fractal literature, making it fully computable. Moreover, as an attempt to tighten the bounds, we propose a PAC-Bayesian setting based on perturbed dynamics, in which the same geometric term plays a crucial role and can still be bounded using the techniques described above.Comment: Accepted in the NeurIPS 2023 Workshop Heavy Tails in Machine Learnin

Learning via Wasserstein-Based High Probability Generalisation Bounds

Minimising upper bounds on the population risk or the generalisation gap has been widely used in structural risk minimisation (SRM) -- this is in particular at the core of PAC-Bayesian learning. Despite its successes and unfailing surge of interest in recent years, a limitation of the PAC-Bayesian framework is that most bounds involve a Kullback-Leibler (KL) divergence term (or its variations), which might exhibit erratic behavior and fail to capture the underlying geometric structure of the learning problem -- hence restricting its use in practical applications. As a remedy, recent studies have attempted to replace the KL divergence in the PAC-Bayesian bounds with the Wasserstein distance. Even though these bounds alleviated the aforementioned issues to a certain extent, they either hold in expectation, are for bounded losses, or are nontrivial to minimize in an SRM framework. In this work, we contribute to this line of research and prove novel Wasserstein distance-based PAC-Bayesian generalisation bounds for both batch learning with independent and identically distributed (i.i.d.) data, and online learning with potentially non-i.i.d. data. Contrary to previous art, our bounds are stronger in the sense that (i) they hold with high probability, (ii) they apply to unbounded (potentially heavy-tailed) losses, and (iii) they lead to optimizable training objectives that can be used in SRM. As a result we derive novel Wasserstein-based PAC-Bayesian learning algorithms and we illustrate their empirical advantage on a variety of experiments.Comment: Accepted to NeurIPS 202

The chemical properties of dissolved organic matter as a function of seasonal and microbiological factors

Dissolved organic matter (DOM) is the pool of molecules predominantly produced from cellular growth in both terrestrial and aquatic systems and forms a reservoir of 662 Pg of carbon in the ocean. With as many as 10⁵ to 10⁷ different chemicals held in a single sample, the chemical diversity typically outstrips the capability of analytical techniques and the human capacity to effectively monitor the effects of environmental factors on their individual abundance. To address this issue, we adopted a “fingerprinting” approach and performed two sets of experiments to monitor the behavior of co-clustered compounds. In the first experiment, the change of DOM under seasonal, spatial, and reactivity variables was delineated using a size-exclusion chromatography approach applying multiple detectors and a computing technique called PARAFAC. The model showed how the molar mass and fluorescent properties of DOM change with the impact of biological activity and photodegradation in terrestrial aquatic systems, as well as leaching from different soil and sediment profiles. The second experiment analyzed the production of DOM moieties during the growth of several mixed diatom assemblages. Various patterns in fluorescent molecules and NMR bands were observed characterizing better the deep biological imprint of primary producers on DOM in estuaries. These two experiments, both performed in boreal regions, were complementary both in processes (i.e., production vs degradation) and in techniques (e.g., mass spectrometry vs NMR). It also enabled us to map those effects across the aquatic gradient (i.e., rivers and coasts). Interesting findings were yielded, and each time a limited number of factors were able to explain most of the data variance which allowed me to situate and discuss my results within the context of various DOM studies

Méthodologie pour l’étude de l’évolution des comportements des voyageurs de transport collectif urbain

RÉSUMÉ : Démocratisé depuis déjà plusieurs années les systèmes tarifaires automatisés, relatifs à l’accès aux transports en commun, génèrent des masses de données encore trop peu exploitées. Ces données issues de cartes à puce sont devenues si volumineuses que leur analyse représente un véritable défi pour l’homme, mais également un immense potentiel pour la planification en transport en commun. Ce mémoire s’inscrit dans le cadre de la valorisation de volumétries importantes de données quotidiennes. Ouvrant un projet commun avec des exploitants de transports, il s’agit de s’intéresser à l’analyse de la demande. L’ensemble des méthodes seront développées à partir de trois ans de données de transaction issues de l’utilisation du transport par bus à Gatineau. L’objectif principal de la recherche est de présenter une méthodologie simple et complète, relative à l’étude longitudinale des comportements d’usage des cartes à puces sur long terme en utilisant différentes techniques d’exploration de données. À terme, cette méthode d’analyse fournit des résultats aidant le travail d’un planificateur de réseau. Les sous-objectifs de l’étude sont les suivants : - Développer un algorithme permettant une analyse comportementale des usagers. - Développer un algorithme expérimental améliorant la méthode précédente, afin que l’analyste puisse suivre l’évolution des comportements des usagers à travers le temps. - Proposer une méthode de prévision des évolutions, enrichissant ainsi les connaissances apportées à la planification. Ce mémoire débute par une revue de littérature présentant l’intérêt de l’utilisation des cartes à puces en analyse. Il s’agit de s’intéresser aux diverses études réalisées, notamment dans le cadre d’analyses comportementales. Une partie de la littérature s’intéresse aux techniques d’exploration de données, particulièrement dans le cas de segmentations et de prévisions. La section méthodologie présente les raisonnements répondant aux trois sous-objectifs, et la dernière partie les résultats des diverses expérimentations effectuées sur les données fournies par la STO. Les contributions apportées par ce mémoire sont : - La présentation d’une méthode classique d’analyse comportementale des cartes à puce à partir de leurs utilisations. Un travail de segmentation est effectué sur l’ensemble des déplacements hebdomadaires en transports en commun afin de repérer les similarités entre comportements. - La conception et la critique d’une méthode expérimentale basée sur une segmentation hebdomadaire visant à montrer l’évolution des comportements des cartes à travers le temps. - Différents indicateurs de qualité et de stabilité de segmentation sont proposés afin de comparer les diverses méthodes engagées, et de caractériser la population de cartes étudiée. - Jouant sur une possible évolution comportementale des cartes, une critique sur la fiabilité de l’utilisation de méthodes de prévision est réalisée. Les prévisions sont appliquées sur l’évolution comportementale des groupes ainsi que l’évolution de la taille de leur population. En conclusion, ce projet présente une méthode classique, fonctionnelle et applicable en industrie permettant l’analyse comportementale des usagers. Prenant comme entrée un jeu de données de cartes à puce, la méthode exporte les résultats de segmentation liés à l’utilisation des transports en commun. Par cela, elle définit 6 groupes d’identifiants aux comportements similaires dont les caractéristiques propres permettent l’aide à la décision en planification des transports. Il s’agit de trois groupes dont les déplacements récurrents en semaine ressemblent à ceux de travailleurs à temps plein et à mi-temps. Deux groupes représentent les comportements de déplacements occasionnels et le dernier contient l’ensemble des cartes qui produisent le plus de déplacement. Le tout est réalisé en un temps relativement long : 11 minutes pour la segmentation de 10 millions de déplacements. La méthode expérimentale, quant à elle, se concentre sur l’évolution comportementale possible des cartes. Sans pour autant être parfaite, elle admet un potentiel énorme. En effet, elle permet une analyse des comportements des 6 groupes de cartes en un temps de calcul très court (38 secondes pour une qualité similaire). Il s’agit de groupes dont les caractéristiques sont très proches de ceux issus de la méthode classique, mais le principe incrémental de la segmentation rend possible l'étude de l’évolution comportementale, jugée fixe dans la méthode classique.----------ABSTRACT : For several years automated fare systems related to public transport access are generating an, not enough, exploited massive volume of data. These smart card data became so voluminous they represent a challenge for humans and a huge potential to public transport planning too. This work aims to value massive volumes of daily data. Opening a common project with transit services, the analysis is based on studying demand. All methods were developed thanks to the three years of transactions from the usage of Gatineau’s bus network. Presenting a simple and a complete methodology to apply a longitudinal analysis on smart card usage behavior using different data mining techniques, represent the main purpose of the research. At the end, the analysis methodology gives the results helping to do the transit planners job. The sub-objectives are the following ones: - Develop an algorithm allowing users’ behavior analysis - Develop an improved algorithm (experimental), allowing to follow the users’ behavior evolution through time. - Propose an evolution prevision methodology, enhancing transit planning knowledge. This works starts with a literature review presenting smart card data usage in analysis, particularly through different studies done in behavior analysis. The second part of the literature review is about data mining techniques like clustering and forecasting. The methodology section describes the three sub-objectives, and the final section presents the different applications on STO’s data. The main achievements of this project are: - The presentation of a classical methodology allowing to analyze smart cards’ behavior through their activities. A clustering technique is applied on all weekly usage of public transit to find similarity between behaviors. - The conception and critic of an experimental method based on week-to-week clustering aiming to show the users’ behavior evolution through time. - Different quality and stability indicators are proposed to compare the methods applied and to characterize the population. Knowing that users’ behavior can evolve, a critic on prevision technique viability is applied. Forecasts methods are used on clusters’ behavior evolution and their population size evolution. Finally, this project presents an industrially applicable methodology on transit users’ behavior. Taking smart card data as input, the algorithm exports the transit usage clustering results. This way it defines 6 groups of IDs with similar behavior which the proper characteristics help the transit planner to take decisions. There are three groups which the trips patterns look like full time and part-time workers trip patterns. Two of the groups represent occasional trip behavior and the last one holds the cards with the most trips. The computation time is relatively high: 11 minutes for the clustering of 10 million transactions. The experimental method focuses on the possible smart cards’ behavior evolution. Without being perfect, it shows a huge potential. Indeed, the method allows a behavior analysis of 6 groups with a shorter computation time (only 38 seconds for a similar quality). These groups present the same characteristics as those from the traditional method, but the way the algorithm works makes the behavior evolution analysis possible in this case

Tighter Generalisation Bounds via Interpolation

This paper contains a recipe for deriving new PAC-Bayes generalisation bounds based on the $(f, \Gamma)$-divergence, and, in addition, presents PAC-Bayes generalisation bounds where we interpolate between a series of probability divergences (including but not limited to KL, Wasserstein, and total variation), making the best out of many worlds depending on the posterior distributions properties. We explore the tightness of these bounds and connect them to earlier results from statistical learning, which are specific cases. We also instantiate our bounds as training objectives, yielding non-trivial guarantees and practical performances

Du conte à la scène, l'exemple de Blanche-Neige

Le présent mémoire de recherche-création a pour objet les réécritures théâtrales des contes. Si le conte merveilleux et le théâtre sont a priori deux genres littéraires distincts de par leur origine, leur nature et leur mode d’expression, ils présentent néanmoins des points communs qui facilitent la transgression des frontières génériques. Aussi les dramaturges se sont-ils parfois risqués à proposer des adaptations de contes merveilleux pour le théâtre. C’est par exemple le cas de Joël Pommerat, qui a récemment proposé des transpositions scéniques de trois contes traditionnels, revisitant et actualisant thèmes et personnages. Dans cet essai, je m’intéresse tout d’abord aux éléments qui favorisent la réécriture théâtrale des contes. Je m’interroge ensuite sur la manière dont Pommerat, qui fonde sa réflexion sur les apports de la philosophie, de la sociologie et de la psychanalyse, prépare ses spectacles ; ce que j’illustre en mettant en lumière la méditation menée par l’auteur sur le travail du deuil dans son spectacle Cendrillon. À l’instar de Pommerat, je présente enfin une étude psychanalytique des personnages du conte Blanche-Neige des frères Grimm. La création que je propose consiste en une transposition théâtrale du conte Blanche-Neige des frères Grimm, qui déplace situations et personnages dans le monde contemporain de l’entreprise.The purpose of this research-creation master thesis is to rewrite the theatrical aspects of the fairy tales. While storytelling and theatre are a priori two literary genres distinct in origin, nature and mode of expression, they nevertheless share common qualities that facilitate the transgression of generic boundaries. So playwrights have sometimes ventured to offer adaptations of fairy tales for the theatre. This is the case, for example, of Joël Pommerat, who recently proposed scenic transpositions of three traditional fairy tales, revisiting and updating themes and characters. In this essay, I am first interested in the elements that promote the theatrical rewriting of fairy tales. I then ask myself how Pommerat, who bases his reflection on the contributions of philosophy, sociology and psychoanalysis, prepares his shows ; what I illustrate by highlighting the author's reflections on the work of mourning in his play Cinderella. Like Pommerat, I finally present a psychoanalytical study of the characters in the Grimm brothers' Snow White tale. The creation I propose consists of a theatrical transposition of the Grimm brothers' Snow White tale, which moves situations and characters into the contemporary world of business

Perturbations of the CD8+ T-cell repertoire in CVID patients with complications

AbstractA higher chronic expansion of effector cytotoxic CD8+DR+ T-lymphocytes has been reported in common variable immunodeficiency (CVID) patients with complications such as splenomegaly, autoimmune disease and/or granulomatous disease. In order to document the features associated with this T cell activation involving the CD8+ T-compartment, we examined the diversity of the alpha/beta TCR repertoire of the patient's CD8+ T-lymphocytes using the qualitative analysis of the CDR3 lengths (Immunoscope).Ten CIVD patients were enrolled in this study, four without complications (Group 1), six with complications (Group 2). All patients exhibited non-gaussian altered CDR3 length distributions, albeit to different extent within the different Vβ families. CVID patients with activated CD8+ T-cells show a reduction of their TCR repertoire diversity which is more severe in patients with complications. Viral reactivations such as CMV are suspected to be part of the mechanisms underlying immunosenescence

Learning Stochastic Majority Votes by Minimizing a PAC-Bayes Generalization Bound

We investigate a stochastic counterpart of majority votes over finite ensembles of classifiers, and study its generalization properties. While our approach holds for arbitrary distributions, we instantiate it with Dirichlet distributions: this allows for a closed-form and differentiable expression for the expected risk, which then turns the generalization bound into a tractable training objective.The resulting stochastic majority vote learning algorithm achieves state-of-the-art accuracy and benefits from (non-vacuous) tight generalization bounds, in a series of numerical experiments when compared to competing algorithms which also minimize PAC-Bayes objectives -- both with uninformed (data-independent) and informed (data-dependent) priors

Immunological markers after long-term treatment interruption in chronically HIV-1 infected patients with CD4 cell count above 400 x 10(6) cells/l.

OBJECTIVE: To analyse immunological markers associated with CD4+ lymphocyte T-cell count (CD4+) evolution during 12-month follow-up after treatment discontinuation. METHOD: Prospective observational study of chronically HIV-1 infected patients with CD4+ above 400 x 10(6) cells/l. RESULTS: CD4+ changes took place in two phases: an initial rapid decrease in the first month (-142 x 10(6) cells/l on average), followed by a slow decline (-17 x 10(6) cells/l on average) The second slope of CD4+ decline was not correlated with the first and only baseline plasma HIV RNA was associated with it. The decline in CD4+ during the first month was steeper in patients with higher CD4+ and weaker plasma HIV RNA baseline levels. Moreover, the decline was less pronounced (P < 10(-4)) in patients with CD4+ nadir above 350 x 10(6) cells/l (-65 x 10(6) cells/l per month) in comparison with those below 350 x 10(6) cells/l (-200 x 10(6) cells/l per month). A high number of dendritic cells (DCs) whatever the type was associated with high CD4+ at the time of treatment interruption and its steeper decline over the first month. Moreover, the myeloid DC level was stable whereas the lymphoid DC count, which tended to decrease in association with decrease in CD4+, was negatively correlated with the HIV RNA load slope. CONCLUSIONS: The results support the use of the CD4+ nadir to predict the CD4+ dynamic after treatment interruption and consideration of the CD4+ count after 1-month of interruption merely reflects the 12-month level of CD4+. Although DCs seem to be associated with the CD4+ dynamic, the benefit of monitoring them has still to be defined