Premiers résultats empiriques sur l’apprentissage par renforcement symbolique

Reinforcement learning is a subfield of machine learning that is concerned with how agents learn to make decisions in an environment in order to maximize some notion of reward. It has shown great promise in a variety of domains, but it often struggles with generalization and interpretability due to its reliance on black-box models. One approach to address this issue is to incorporate knowledge representation into reinforcement learning. In this work, we explore the benefits of using symbolic representation in reinforcement learning and present preliminary results on its performance compared to standard reinforcement learning techniques. Our experiments show that the use of symbolic representation can significantly improve the generalization capabilities of reinforcement learning agents. We also discuss the potential challenges and limitations of this approach and suggest avenues for future research.L’apprentissage par renforcement est un sous-domaine de l’apprentissage automatique qui s’intéresse à la manière dont les agents apprennent à prendre des décisions dans un environnement afin de maximiser une récompense. Il s’est avéré très prometteur dans divers domaines, mais il se heurte souvent à des problèmes de généralisation et d’interprétabilité en raison de sa dépendance à l’égard de modèles de type "boîte noire". Une approche pour résoudre ce problème consiste à incorporer la représentation des connaissances dans l’apprentissage par renforcement. Dans ce travail, nous explorons les avantages de l’utilisation de la représentation symbolique dans l’apprentissage par renforcement et présentons des résultats préliminaires sur sa performance par rapport aux techniques d’apprentissage par renforcement standard. Nos expériences montrent que l’utilisation de la représentation symbolique peut améliorer de manièresignificative les capacités de généralisation des agents d’apprentissage par renforcement. Nous discutons également des défis potentiels et des limites de cette approche et suggérons des pistes de recherche pour l’avenir

Evolving Reservoirs for Meta Reinforcement Learning

Animals often demonstrate a remarkable ability to adapt to their environments during their lifetime. They do so partly due to the evolution of morphological and neural structures. These structures capture features of environments shared between generations to bias and speed up lifetime learning. In this work, we propose a computational model for studying a mechanism that can enable such a process. We adopt a computational framework based on meta reinforcement learning as a model of the interplay between evolution and development. At the evolutionary scale, we evolve reservoirs, a family of recurrent neural networks that differ from conventional networks in that one optimizes not the weight values but hyperparameters of the architecture: the later control macro-level properties, such as memory and dynamics. At the developmental scale, we employ these evolved reservoirs to facilitate the learning of a behavioral policy through Reinforcement Learning (RL). Within an RL agent, a reservoir encodes the environment state before providing it to an action policy. We evaluate our approach on several 2D and 3D simulated environments. Our results show that the evolution of reservoirs can improve the learning of diverse challenging tasks. We study in particular three hypotheses: the use of an architecture combining reservoirs and reinforcement learning could enable (1) solving tasks with partial observability, (2) generating oscillatory dynamics that facilitate the learning of locomotion tasks, and (3) facilitating the generalization of learned behaviors to new tasks unknown during the evolution phase

B2LiVe, a label-free 1D-NMR method to quantify the binding of amphitropic peptides or proteins to membrane vesicles

Posted December 16, 2022 on bioRxiv.Amphitropic proteins and peptides reversibly partition from solution to membrane, a key process that regulates their functions. Experimental approaches, such as fluorescence and circular dichroism, are classically used to measure the partitioning of amphitropic peptides and proteins into lipid bilayers, yet hardly usable when the peptides or proteins do not exhibit significant polarity and/or conformational changes upon membrane binding. Here, we describe B2LiVe ( i . e ., Binding to Lipid Vesicles), a simple, robust, and widely applicable NMR method to determine the solution-to-membrane partitioning of unlabeled proteins or peptides. The experimental strategy proposed here relies on previously described proton 1D NMR fast pulsing techniques with selective adiabatic pulses. Membrane partitioning induces a large line broadening leading to a progressive loss of protein signals, and therefore, the decrease of the NMR signal directly measures the fraction of membrane-bound protein. The B2LiVe method uses low polypeptide concentrations and has been validated on several membrane-interacting peptides and proteins, ranging from 3 to 54 kDa, with membrane vesicles of different sizes and various lipid compositions. Motivation Characterization of the interaction of peptides and proteins with lipid membranes is involved in various biological processes and is often challenging for polypeptides which do not possess intrinsic fluorophores, do not exhibit significant structural content changes, as well as for those characterized by low affinities for membranes. To meet these challenges, we have developed a simple and robust label-free NMR-based experimental approach, named B2LiVe, to measure the binding of polypeptides to lipid vesicles. The experimental strategy relies on previously described proton 1D NMR fast pulsing techniques with selective adiabatic pulses to excite the amide resonances. B2LiVe is a label-free method based on the observation of amide hydrogen nuclei which are naturally present in all protein and peptide backbones. Our results validate the B2LiVe method and indicate that it compares well with established technics to quantify polypeptide-membrane interactions. Overall, B2LiVe should efficiently complement the arsenal of label-free biophysical assays available to characterize protein-membrane interactions. In brief We describe a robust label-free NMR-based experimental approach (B2LiVe) to measure interactions between peptides or proteins with membranes. The validity of this approach has been established on several polypeptides and on various membrane vesicles. The B2LiVe method efficiently complements the arsenal of label-free biophysical techniques to characterize protein-membrane interactions. Highlights B2LiVe is a simple and robust NMR-based method to quantify affinity of proteins and peptides for membranes B2LiVe is a label-free approach that relies on proton 1D NMR fast pulsing techniques with selective excitation of amide resonances B2LiVe has been validated on several membrane-interacting peptides and proteins B2LiVe coupled to DOSY can pinpoint the presence within a membrane-bound protein of polypeptide segments remaining in solutio

Gallocin A, an Atypical Two-Peptide Bacteriocin with Intramolecular Disulfide Bonds Required for Activity

International audienceStreptococcus gallolyticus subsp. gallolyticus (SGG) is an opportunistic gut pathogen associated with colorectal cancer. We previously showed that colonization of the murine colon by SGG in tumoral conditions was strongly enhanced by the production of gallocin A, a two-peptide bacteriocin. Here, we aimed to characterize the mechanisms of its action and resistance. Using a genetic approach, we demonstrated that gallocin A is composed of two peptides, GllA1 and GllA2, which are inactive alone and act together to kill "target" bacteria. We showed that gallocin A can kill phylogenetically close relatives of the pathogen. Importantly, we demonstrated that gallocin A peptides can insert themselves into membranes and permeabilize lipid bilayer vesicles. Next, we showed that the third gene of the gallocin A operon, gip, is necessary and sufficient to confer immunity to gallocin A. Structural modeling of GllA1 and GllA2 mature peptides suggested that both peptides form alpha-helical hairpins stabilized by intramolecular disulfide bridges. The presence of a disulfide bond in GllA1 and GllA2 was confirmed experimentally. Addition of disulfide-reducing agents abrogated gallocin A activity. Likewise, deletion of a gene encoding a surface protein with a thioredoxin-like domain impaired the ability of gallocin A to kill Enterococcus faecalis. Structural modeling of GIP revealed a hairpin-like structure strongly resembling those of the GllA1 and GllA2 mature peptides, suggesting a mechanism of immunity by competition with GllA1/2. Finally, identification of other class IIb bacteriocins exhibiting a similar alpha-helical hairpin fold stabilized with an intramolecular disulfide bridge suggests the existence of a new subclass of class IIb bacteriocins. IMPORTANCE Streptococcus gallolyticus subsp. gallolyticus (SGG), previously named Streptococcus bovis biotype I, is an opportunistic pathogen responsible for invasive infections (septicemia, endocarditis) in elderly people and is often associated with colon tumors. SGG is one of the first bacteria to be associated with the occurrence of colorectal cancer in humans. Previously, we showed that tumor-associated conditions in the colon provide SGG with an ideal environment to proliferate at the expense of phylogenetically and metabolically closely related commensal bacteria such as enterococci (1). SGG takes advantage of CRC-associated conditions to outcompete and substitute commensal members of the gut microbiota using a specific bacteriocin named gallocin, recently renamed gallocin A following the discovery of gallocin D in a peculiar SGG isolate. Here, we showed that gallocin A is a two-peptide bacteriocin and that both GllA1 and GllA2 peptides are required for antimicrobial activity. Gallocin A was shown to permeabilize bacterial membranes and kill phylogenetically closely related bacteria such as most streptococci, lactococci, and enterococci, probably through membrane pore formation. GllA1 and GllA2 secreted peptides are unusually long (42 and 60 amino acids long) and have very few charged amino acids compared to well-known class IIb bacteriocins. In silico modeling revealed that both GllA1 and GllA2 exhibit a similar hairpin-like conformation stabilized by an intramolecular disulfide bond. We also showed that the GIP immunity peptide forms a hairpin-like structure similar to GllA1/GllA2. Thus, we hypothesize that GIP blocks the formation of the GllA1/GllA2 complex by interacting with GllA1 or GllA2. Gallocin A may constitute the first class IIb bacteriocin which displays disulfide bridges important for its structure and activity and might be the founding member of a subtype of class IIb bacteriocins

Ligand-induced conformational switch in an artificial bidomain protein scaffold

Abstract Artificial proteins binding any predefined “target” protein can now be efficiently generated using combinatorial libraries based on robust protein scaffolds. αRep is such a family of artificial proteins, based on an α-solenoid protein repeat scaffold. The low aggregation propensity of the specific “binders” generated from this library opens new protein engineering opportunities such as the creation of biosensors within multidomain constructions. Here, we have explored the properties of two new types of artificial bidomain proteins based on αRep structures. Their structural and functional properties are characterized in detail using biophysical methods. The results clearly show that both bidomain proteins adopt a closed bivalve shell-like conformation, in the ligand free form. However, the presence of ligands induces a conformational transition, and the proteins adopt an open form in which each domain can bind its cognate protein partner. The open/closed equilibria alter the affinities of each domain and induce new cooperative effects. The binding-induced relative domain motion was monitored by FRET. Crystal structures of the chimeric proteins indicate that the conformation of each constituting domain is conserved but that their mutual interactions explain the emergent properties of these artificial bidomain proteins. The ligand-induced structural transition observed in these bidomain proteins should be transferable to other αRep proteins with different specificity and could provide the basis of a new generic biosensor design

Dynamics and structural changes of calmodulin upon interaction with the antagonist calmidazolium

International audienceBackground Calmodulin (CaM) is an evolutionarily conserved eukaryotic multifunctional protein that functions as the major sensor of intracellular calcium signaling. Its calcium-modulated function regulates the activity of numerous effector proteins involved in a variety of physiological processes in diverse organs, from proliferation and apoptosis, to memory and immune responses. Due to the pleiotropic roles of CaM in normal and pathological cell functions, CaM antagonists are needed for fundamental studies as well as for potential therapeutic applications. Calmidazolium (CDZ) is a potent small molecule antagonist of CaM and one the most widely used inhibitors of CaM in cell biology. Yet, CDZ, as all other CaM antagonists described thus far, also affects additional cellular targets and its lack of selectivity hinders its application for dissecting calcium/CaM signaling. A better understanding of CaM:CDZ interaction is key to design analogs with improved selectivity. Here, we report a molecular characterization of CaM:CDZ complexes using an integrative structural biology approach combining SEC-SAXS, X-ray crystallography, HDX-MS, and NMR. Results We provide evidence that binding of a single molecule of CDZ induces an open-to-closed conformational reorientation of the two domains of CaM and results in a strong stabilization of its structural elements associated with a reduction of protein dynamics over a large time range. These CDZ-triggered CaM changes mimic those induced by CaM-binding peptides derived from physiological protein targets, despite their distinct chemical natures. CaM residues in close contact with CDZ and involved in the stabilization of the CaM:CDZ complex have been identified. Conclusion Our results provide molecular insights into CDZ-induced dynamics and structural changes of CaM leading to its inhibition and open the way to the rational design of more selective CaM antagonists. Graphical abstract Calmidazolium is a potent and widely used inhibitor of calmodulin, a major mediator of calcium-signaling in eukaryotic cells. Structural characterization of calmidazolium-binding to calmodulin reveals that it triggers open-to-closed conformational changes similar to those induced by calmodulin-binding peptides derived from enzyme targets. These results provide molecular insights into CDZ-induced dynamics and structural changes of CaM leading to its inhibition and open the way to the rational design of more selective CaM antagonists