Predicting optimal value functions by interpolating reward functions in scalarized multi-objective reinforcement learning

A common approach for defining a reward function for Multi-objective Reinforcement Learning (MORL) problems is the weighted sum of the multiple objectives. The weights are then treated as design parameters dependent on the expertise (and preference) of the person performing the learning, with the typical result that a new solution is required for any change in these settings. This paper investigates the relationship between the reward function and the optimal value function for MORL; specifically addressing the question of how to approximate the optimal value function well beyond the set of weights for which the optimization problem was actually solved, thereby avoiding the need to recompute for any particular choice. We prove that the value function transforms smoothly given a transformation of weights of the reward function (and thus a smooth interpolation in the policy space). A Gaussian process is used to obtain a smooth interpolation over the reward function weights of the optimal value function for three well-known examples: GridWorld, Objectworld and Pendulum. The results show that the interpolation can provide very robust values for sample states and action space in discrete and continuous domain problems. Significant advantages arise from utilizing this interpolation technique in the domain of autonomous vehicles: easy, instant adaptation of user preferences while driving and true randomization of obstacle vehicle behavior preferences during training.Comment: Accepted at ICRA 202

Computational Prediction and Experimental Verification of New MAP Kinase Docking Sites and Substrates Including Gli Transcription Factors

In order to fully understand protein kinase networks, new methods are needed to identify regulators and substrates of kinases, especially for weakly expressed proteins. Here we have developed a hybrid computational search algorithm that combines machine learning and expert knowledge to identify kinase docking sites, and used this algorithm to search the human genome for novel MAP kinase substrates and regulators focused on the JNK family of MAP kinases. Predictions were tested by peptide array followed by rigorous biochemical verification with in vitro binding and kinase assays on wild-type and mutant proteins. Using this procedure, we found new ‘D-site’ class docking sites in previously known JNK substrates (hnRNP-K, PPM1J/PP2Czeta), as well as new JNK-interacting proteins (MLL4, NEIL1). Finally, we identified new D-site-dependent MAPK substrates, including the hedgehog-regulated transcription factors Gli1 and Gli3, suggesting that a direct connection between MAP kinase and hedgehog signaling may occur at the level of these key regulators. These results demonstrate that a genome-wide search for MAP kinase docking sites can be used to find new docking sites and substrates

Metabolic profiling identifies trehalose as an abundant and diurnally fluctuating metabolite in the microalga Ostreococcus tauri

© 2017, The Author(s).Introduction: The picoeukaryotic alga Ostreococcus tauri (Chlorophyta) belongs to the widespread group of marine prasinophytes. Despite its ecological importance, little is known about the metabolism of this alga. Objectives: In this work, changes in the metabolome were quantified when O. tauri was grown under alternating cycles of 12 h light and 12 h darkness. Methods: Algal metabolism was analyzed by gas chromatography-mass spectrometry. Using fluorescence-activated cell sorting, the bacteria associated with O. tauri were depleted to below 0.1% of total cells at the time of metabolic profiling. Results: Of 111 metabolites quantified over light–dark cycles, 20 (18%) showed clear diurnal variations. The strongest fluctuations were found for trehalose. With an intracellular concentration of 1.6 mM in the dark, this disaccharide was six times more abundant at night than during the day. This fluctuation pattern of trehalose may be a consequence of starch degradation or of the synchronized cell cycle. On the other hand, maltose (and also sucrose) was below the detection limit (~10 μM). Accumulation of glycine in the light is in agreement with the presence of a classical glycolate pathway of photorespiration. We also provide evidence for the presence of fatty acid methyl and ethyl esters in O. tauri. Conclusions: This study shows how the metabolism of O. tauri adapts to day and night and gives new insights into the configuration of the carbon metabolism. In addition, several less common metabolites were identified