Long-term Safe Reinforcement Learning with Binary Feedback

Safety is an indispensable requirement for applying reinforcement learning (RL) to real problems. Although there has been a surge of safe RL algorithms proposed in recent years, most existing work typically 1) relies on receiving numeric safety feedback; 2) does not guarantee safety during the learning process; 3) limits the problem to a priori known, deterministic transition dynamics; and/or 4) assume the existence of a known safe policy for any states. Addressing the issues mentioned above, we thus propose Long-term Binaryfeedback Safe RL (LoBiSaRL), a safe RL algorithm for constrained Markov decision processes (CMDPs) with binary safety feedback and an unknown, stochastic state transition function. LoBiSaRL optimizes a policy to maximize rewards while guaranteeing a long-term safety that an agent executes only safe state-action pairs throughout each episode with high probability. Specifically, LoBiSaRL models the binary safety function via a generalized linear model (GLM) and conservatively takes only a safe action at every time step while inferring its effect on future safety under proper assumptions. Our theoretical results show that LoBiSaRL guarantees the long-term safety constraint, with high probability. Finally, our empirical results demonstrate that our algorithm is safer than existing methods without significantly compromising performance in terms of reward.Comment: Accepted to AAAI-2

Neural Controller Synthesis for Signal Temporal Logic Specifications Using Encoder-Decoder Structured Networks

In this paper, we propose a control synthesis method for signal temporal logic (STL) specifications with neural networks (NNs). Most of the previous works consider training a controller for only a given STL specification. These approaches, however, require retraining the NN controller if a new specification arises and needs to be satisfied, which results in large consumption of memory and inefficient training. To tackle this problem, we propose to construct NN controllers by introducing encoder-decoder structured NNs with an attention mechanism. The encoder takes an STL formula as input and encodes it into an appropriate vector, and the decoder outputs control signals that will meet the given specification. As the encoder, we consider three NN structures: sequential, tree-structured, and graph-structured NNs. All the model parameters are trained in an end-to-end manner to maximize the expected robustness that is known to be a quantitative semantics of STL formulae. We compare the control performances attained by the above NN structures through a numerical experiment of the path planning problem, showing the efficacy of the proposed approach.Comment: submitted for publicatio

Bacteria with a mouth: Discovery and new insights into cell surface structure and macromolecule transport

A bacterium with a "mouth"-like pit structure isolated for the first time in the history of microbiology was a Gram-negative rod, containing glycosphingolipids in the cell envelope, and named Sphingomonas sp. strain A1. The pit was dynamic, with repetitive opening and closing during growth on alginate, and directly included alginate concentrated around the pit, particularly by flagellins, an alginate-binding protein localized on the cell surface. Alginate incorporated into the periplasm was subsequently transferred to the cytoplasm by cooperative interactions of periplasmic solute-binding proteins and an ATP-binding cassette transporter in the cytoplasmic membrane. The mechanisms of assembly, functions, and interactions between the above-mentioned molecules were clarified using structural biology. The pit was transplanted into other strains of sphingomonads, and the pitted recombinant cells were effectively applied to the production of bioethanol, bioremediation for dioxin removal, and other tasks. Studies of the function of the pit shed light on the biological significance of cell surface structures and macromolecule transport in bacteria

Hydration of vinyl ether groups by unsaturated glycoside hydrolases and their role in bacterial pathogenesis

Many pathogenic microorganisms invade mammalian and/or plant cells by producing polysaccharide-degrading enzymes (lyases and hydrolases). Mammalian glycosaminoglycans and plant pectins that form part of the cell surface matrix are typical targets for these microbial enzymes. Unsaturated glycoside hydrolase catalyzes the hydrolytic release of an unsaturated uronic acid from oligosaccharides, which are produced through the reaction of matrix-degrading polysaccharide lyase. This enzymatic ability suggests that unsaturated glycoside hydrolases function as virulence factors in microbial infection. This review focuses on the molecular identification, bacterial distribution, and structure/function relationships of these enzymes. In contrast to general glycoside hydrolases, in which the catalytic mechanism involves the retention or inversion of an anomeric configuration, unsaturated glycoside hydrolases uniquely trigger the hydrolysis of vinyl ether groups in unsaturated saccharides but not of their glycosidic bonds. [Int Microbiol 2007; 10(4):233-243

A novel bleb-dependent polysaccharide export system in nitrogen-fixing Azotobacter vinelandii subjected to low nitrogen gas levels

The alginate biofilm-producing bacterium Azotobacter vinelandii aerobically fixes nitrogen by oxygensensitivenitrogenases. Here we investigated the bacterial response to nitrogen/oxygen gas mixtures. A. vinelandii cellswere cultured in nitrogen-free minimal media containing gas mixtures differing in their ratios of nitrogen and oxygen.The bacteria did not grow at oxygen concentrations >75 % but grew well in the presence of 5 % nitrogen/25 % oxygen.Growth of wild-type and alginate-deficient strains when cultured with 50 % oxygen did not differ substantially, indicatingthat alginate is not required for the protection of nitrogenases from oxygen damage. In response to decreasing nitrogenlevels, A. vinelandii produced greater amounts of alginate, accompanied by the formation of blebs on the cell surface. Theencystment of vegetative cells occurred in tandem with the release of blebs and the development of a multilayered exine.Immunoelectron microscopy using anti alginate-antibody revealed that the blebs contained alginate molecules. Bycontrast, alginate-deficient mutants could not form blebs. Taken together, our data provide evidence for a novel blebdependentpolysaccharide export system in A. vinelandii that is activated in response to low nitrogen gas levels. [IntMicrobiol 2013; 16(1):35-44

The Optimal Cut-offValue of Ankle Brachial Index for Screening Cardiovascular Disease Risk in Hemodialysis Patients

Article信州医学雑誌 64(3): 135-146(2016)journal articl

Polyunsaturated fatty acids-enriched lipid from reduced sugar alcohol mannitol by marine yeast Rhodosporidiobolus fluvialis Y2

Brown macroalgae is a promising marine biomass for the production of bioethanol and biodiesel fuels. Here we investigate the biochemical processes used by marine oleaginous yeast for assimilating the major carbohydrate found in brown macroalgae. Briefly, yeast Rhodosporidiobolus fluvialis strain Y2 was isolated from seawater and grown in minimal medium containing reduced sugar alcohol mannitol as the sole carbon source with a salinity comparable to seawater. Conditions limiting nitrogen were used to facilitate lipid synthesis. R. fluvialis Y2 yielded 55.1% (w/w) and 39.1% (w/w) of lipids, per dry cell weight, from mannitol in the absence and presence of salinity, respectively. Furthermore, mannitol, as a sugar source, led to an increase in the composition of polyunsaturated fatty acids, linoleic acid (C18:2) and linolenic acid (C18:3), compared to glucose. This suggests that oxidation of mannitol leads to the activation of NADH-dependent fatty acid desaturases in R. fluvialis Y2. Such fatty acid composition may contribute to the cold-flow properties of biodiesel fuels. Our results identified a salt-tolerant oleaginous yeast species with unique metabolic traits, demonstrating a key role as a decomposer in the global carbon cycle through marine ecosystems. This is the first study on mannitol-induced synthesis of lipids enriched with polyunsaturated fatty acids by marine yeast

Bacterial inducible expression of plant cell wall-binding protein YesO through conflict between Glycine max and saprophytic Bacillus subtilis

大豆と納豆菌のせめぎ合いの仕組みを解明 --生きた大豆は納豆菌を嫌い、納豆菌は死んだ大豆が好き--. 京都大学プレスリリース. 2020-11-02.Saprophytic bacteria and plants compete for limited nutrient sources. Bacillus subtilis grows well on steamed soybeans Glycine max to produce the fermented food, natto. Here we focus on bacterial responses in conflict between B. subtilis and G. max. B. subtilis cells maintained high growth rates specifically on non-germinating, dead soybean seeds. On the other hand, viable soybean seeds with germinating capability attenuated the initial growth of B. subtilis. Thus, B. subtilis cells may trigger saprophytic growth in response to the physiological status of G. max. Scanning electron microscope observation indicated that B. subtilis cells on steamed soybeans undergo morphological changes to form apertures, demonstrating cell remodeling during saprophytic growth. Further, transcriptomic analysis of B. subtilis revealed upregulation of the gene cluster, yesOPQR, in colonies growing on steamed soybeans. Recombinant YesO protein, a putative, solute-binding protein for the ATP-binding cassette transporter system, exhibited an affinity for pectin-derived oligosaccharide from plant cell wall. The crystal structure of YesO, in complex with the pectin oligosaccharide, was determined at 1.58 Å resolution. This study expands our knowledge of defensive and offensive strategies in interspecies competition, which may be promising targets for crop protection and fermented food production

Enhanced propagation of Granulicatella adiacens from human oral microbiota by hyaluronan

Host determinants for formation/composition of human oral microbiota remain to be clarified, although microorganisms entering the mouth cannot necessarily colonize the oral environment. Here we show that human oral-abundant bacteria degraded host glycosaminoglycans (GAGs) in saliva and gingiva, and certain bacteria significantly grew on hyaluronan (HA), a kind of GAGs. Microbial communities from teeth or gingiva of healthy donors assimilated HA. Metagenomic analysis of human oral microbiota under different carbon sources revealed HA-driven Granulicatella growth. HA-degrading bacterial strains independently isolated from teeth and gingiva were identified as Granulicatella adiacens producing extracellular 130 kDa polysaccharide lyase as a HA-degrading enzyme encoded in a peculiar GAG genetic cluster containing genes for isomerase KduI and dehydrogenase DhuD. These findings demonstrated that GAGs are one of the host determinants for formation/composition of oral microbiota not only for colonization but also for the adaptation to the host niche. Especially, HA enhanced the G. adiacens propagation