Curriculum Learning for Cumulative Return Maximization

Curriculum learning has been successfully used in reinforcement learning to accelerate the learning process, through knowledge transfer between tasks of increasing complexity. Critical tasks, in which suboptimal exploratory actions must be minimized, can benefit from curriculum learning, and its ability to shape exploration through transfer. We propose a task sequencing algorithm maximizing the cumulative return, that is, the return obtained by the agent across all the learning episodes. By maximizing the cumulative return, the agent not only aims at achieving high rewards as fast as possible, but also at doing so while limiting suboptimal actions. We experimentally compare our task sequencing algorithm to several popular metaheuristic algorithms for combinatorial optimization, and show that it achieves significantly better performance on the problem of cumulative return maximization. Furthermore, we validate our algorithm on a critical task, optimizing a home controller for a micro energy grid

Rheology and microrheology of deformable droplet suspensions

Dense suspensions of soft colloidal particles display a broad range of physical and rheological properties which are still far from being fully understood. To elucidate the role of deformability on colloidal flow, we employ computer simulations to measure the apparent viscosity of a system of droplets of variable surface tension subjected to a pressure-driven flow. We confirm that our suspension generically undergoes discontinuous shear thinning, and determine the dependence of the onset of the discontinuity on surface tension. We find that the effective viscosity of the suspension is mainly determined by a capillary number. We present active microrheology simulations, where a single droplet is dragged through the suspension. These also show a dynamical phase transition, analogous to the one associated with discontinuous shear thinning in our interpretation. Such a transition is signalled by a discontinuity in the droplet velocity versus applied force

An Optimization Framework for Task Sequencing in Curriculum Learning

Curriculum learning in reinforcement learning is used to shape exploration by presenting the agent with increasingly complex tasks. The idea of curriculum learning has been largely applied in both animal training and pedagogy. In reinforcement learning, all previous task sequencing methods have shaped exploration with the objective of reducing the time to reach a given performance level. We propose novel uses of curriculum learning, which arise from choosing different objective functions. Furthermore, we define a general optimization framework for task sequencing and evaluate the performance of popular metaheuristic search methods on several tasks. We show that curriculum learning can be successfully used to: improve the initial performance, take fewer suboptimal actions during exploration, and discover better policies

Flow of Deformable Droplets:Discontinuous Shear Thinning and Velocity Oscillations

We study the rheology of a suspension of soft deformable droplets subjected to a pressure-driven flow. Through computer simulations, we measure the apparent viscosity as a function of droplet concentration and pressure gradient, and provide evidence of a discontinuous shear thinning behaviour, which occurs at a concentration-dependent value of the forcing. We further show that this response is associated with a nonequilibrium transition between a `hard' (or less deformable) phase, which is nearly jammed and flows very slowly, and a `soft' (or more deformable) phase, which flows much more easily. The soft phase is characterised by flow-induced time dependent shape deformations and internal currents, which are virtually absent in the hard phase. Close to the transition, we find sustained oscillations in both the droplet and fluid velocities. Polydisperse systems show similar phenomenology but with a smoother transition, and less regular oscillations

A New Highly Conserved Antibiotic Sensing/Resistance Pathway in Firmicutes Involves an ABC Transporter Interplaying with a Signal Transduction System

Signal transduction systems and ABC transporters often contribute jointly to adaptive bacterial responses to environmental changes. In Bacillus subtilis, three such pairs are involved in responses to antibiotics: BceRSAB, YvcPQRS and YxdJKLM. They are characterized by a histidine kinase belonging to the intramembrane sensing kinase family and by a translocator possessing an unusually large extracytoplasmic loop. It was established here using a phylogenomic approach that systems of this kind are specific but widespread in Firmicutes, where they originated. The present phylogenetic analyses brought to light a highly dynamic evolutionary history involving numerous horizontal gene transfers, duplications and lost events, leading to a great variety of Bce-like repertories in members of this bacterial phylum. Based on these phylogenetic analyses, it was proposed to subdivide the Bce-like modules into six well-defined subfamilies. Functional studies were performed on members of subfamily IV comprising BceRSAB from B. subtilis, the expression of which was found to require the signal transduction system as well as the ABC transporter itself. The present results suggest, for the members of this subfamily, the occurrence of interactions between one component of each partner, the kinase and the corresponding translocator. At functional and/or structural levels, bacitracin dependent expression of bceAB and bacitracin resistance processes require the presence of the BceB translocator loop. Some other members of subfamily IV were also found to participate in bacitracin resistance processes. Taken together our study suggests that this regulatory mechanism might constitute an important common antibiotic resistance mechanism in Firmicutes. [Supplemental material is available online at http://www.genome.org.

Computation tool for rotor's life consumption in steam turbines according to industrial specifications

Life consumption evaluation presently represents a high priority topic for Industry and University. In the specific work of this paper, we performed an analysis about the effects of thermal transients on life consumption of steam turbine rotors and an optimization of the thermal transient during runups. This kind of operation, more and more recurrent in power generation, produces a stress in the mechanical elements that could compromise their integrity. The work was developed in its larger part in the MGMV laboratory of the University of Genova, using the equipment (hardware and software) present in this structure, in cooperation with industry. The study moves from a classification of the damage mechanism and from a description of the most important manufacturing technologies of turbine rotors. Then, after a glance at the most important commercial software for estimating life consumption, a classification is given referring to the checks that Ansaldo, the industrial collaborator, performs during its service interventions. Finally, the paper reports the calculation module developed according to the specifications given by the industrial collaborator; the software was developed in Matlab and needs as input the characteristic constants of the rotor and the temperature measurements (with high precision transducers) at the inlet of the HP and IP sections of the turbine. This study confirms that thermal transients are very important to define the rotor\u2019s residual life; the results could help the manufacturer, who designed the machines, and the operators, who run the power plant, to ensure a correct use of the installation, without unexpected failures that could create an economical damage connected with a missed production

A novel optimization perspective to the problem of designing sequences of tasks in a reinforcement learning framework

Training agents over sequences of tasks is often employed in deep reinforcement learning to let the agents progress more quickly towards better behaviours. This problem, known as curriculum learning, has been mainly tackled in the literature by numerical methods based on enumeration strategies, which, however, can handle only small size problems. In this work, we define a new optimization perspective to the curriculum learning problem with the aim of developing efficient solution methods for solving complex reinforcement learning tasks. Specifically, we show how the curriculum learning problem can be viewed as an optimization problem with a nonsmooth and nonconvex objective function and with an integer feasible region. We reformulate it by defining a grey-box function that includes a suitable scheduling problem. Numerical results on a benchmark environment in the reinforcement learning community show the effectiveness of the proposed approaches in reaching better performance also on large problems

Two-component systems in Pseudomonas aeruginosa: Why so many?

cited By 143International audienceScreening the Pseudomonas aeruginosa genome has led to the identification of the highest number of putative genes encoding two-component regulatory systems of all bacterial genomes sequenced to date (64 and 63 encoding response regulators and histidine kinases, respectively). Sixteen atypical kinases, among them 11 devoid of an Hpt domain, and three independent Hpt modules were retrieved. These data suggest that P. aeruginosa possesses complex control strategies with which to respond to environmental challenges. Copyright (C) 2000 Elsevier Science Ltd

The global activator GacA of Pseudomonas aeruginosa PAO positively controls the production of the autoinducer N-butyryl-homoserine lactone and the formation of the virulence factors pyocyanin, cyanide, and lipase.

The global activator GacA, a highly conserved response regulator in Gram-negative bacteria, is required for the production of exoenzymes and secondary metabolites in Pseudomonas spp. The gacA gene of Pseudomonas aeruginosa PAO1 was isolated and its role in cell-density-dependent gene expression was characterized. Mutational inactivation of gacA resulted in delayed and reduced formation of the cell-density signal N-butyryl-L-homoserine lactone (BHL), of the cognate transcriptional activator RhIR (VsmR), and of the transcriptional activator LasR, which is known to positively regulate RhIR expression. Amplification of gacA on a multicopy plasmid caused precocious and enhanced production of BHL, RhIR and LasR. In parallel, the gacA gene dosage markedly influenced the BHL/RhIR-dependent formation of the cytotoxic compounds pyocyanin and cyanide and the exoenzyme lipase. However, the concentrations of another known cell-density signal of P. aeruginosa, N-oxododecanoyl-L-homoserine lactone, did not always match BHL concentrations. A model accounting for these observations places GacA function upstream of LasR and RhIR in the complex, cell-density-dependent signal-transduction pathway regulating several exoproducts and virulence factors of P. aeruginosa via BHL