Model-free reinforcement learning for stochastic parity games

This paper investigates the use of model-free reinforcement learning to compute the optimal value in two-player stochastic games with parity objectives. In this setting, two decision makers, player Min and player Max, compete on a finite game arena - a stochastic game graph with unknown but fixed probability distributions - to minimize and maximize, respectively, the probability of satisfying a parity objective. We give a reduction from stochastic parity games to a family of stochastic reachability games with a parameter ε, such that the value of a stochastic parity game equals the limit of the values of the corresponding simple stochastic games as the parameter ε tends to 0. Since this reduction does not require the knowledge of the probabilistic transition structure of the underlying game arena, model-free reinforcement learning algorithms, such as minimax Q-learning, can be used to approximate the value and mutual best-response strategies for both players in the underlying stochastic parity game. We also present a streamlined reduction from 112-player parity games to reachability games that avoids recourse to nondeterminism. Finally, we report on the experimental evaluations of both reductions

Modification of the Two-Point Scaling Theory for the Description of the Phase Transition in Solution. Analysis of Sodium Octanoate Aqueous Solutions

On the basis of conventional scaling theory, the two-point scaling theory was modified in order to describe the influence of composition on the partial molar heat capacity and volume during the micellization process. To verify the theory, isobaric heat capacities and densities of aqueous sodium octanoate solutions were measured over wide composition and temperature ranges and the modified approach was used to analyze the calculated partial molar heat capacities and volumes of the surfactant in water. The results obtained indicate that the micellization process is subject to the scaling laws. The results were compared with those for other systems. Peculiar behavior of the critical indices was observed and correlated with the structure of the micelles

Transport of Cytoplasmically Synthesized Proteins into the Mitochondria in a Cell Free System from Neurospora crassa

Synthesis and transport of mitochondrial proteins were followed in a cell-free homogenate of Neurospora crassa in which mitochondrial translation was inhibited. Proteins synthesized on cytoplasmic ribosomes are transferred into the mitochondrial fraction. The relative amounts of proteins which are transferred in vitro are comparable to those transferred in whole cells. Cycloheximide and puromycin inhibit the synthesis of mitochondrial proteins but not their transfer into mitochondria. The transfer of immunoprecipitable mitochondrial proteins was demonstrated for matrix proteins, carboxyatractyloside-binding protein and cytochrome c. Import of proteins into mitochondria exhibits a degree of specificity. The transport mechanism differentiates between newly synthesized proteins and preexistent mitochondrial proteins, at least in the case of matrix proteins. In the cell-free homogenate membrane-bound ribosomes are more active in the synthesis of mitochondrial proteins than are free ribosomes. The finished translation products appear to be released from the membrane-bound ribosomes into the cytosol rather than into the membrane vesicles. The results suggest that the transport of cytoplasmically synthesized mitochondrial proteins is essentially independent of cytoplasmic translation; that cytoplasmically synthesized mitochondrial proteins exist in an extramitochondrial pool prior to import; that the site of this pool is the cytosol for at least some of the mitochondrial proteins; and that the precursors in the extramitochondrial pool differ in structure or conformation from the functional proteins in the mitochondria

Synthesis and structural analysis of the N-terminal domain of the thyroid hormone-binding protein transthyretin

Transthyretin (TTR) is a 55 kDa protein responsible for the transport of thyroid hormones and retinol in human serum. Misfolded forms of the protein are implicated in the amyloid diseases familial amyloidotic polyneuropathy and senile systemic amyloidosis. Its folding properties and stabilization by ligands are of current interest due to their importance in understanding and combating these diseases. To assist in such studies we developed a method for the solid phase synthesis of the monomeric unit of a TTR analogue and its folding to form a functional 55 kDa tetramer. The monomeric unit of the protein was chemically synthesized in three parts, comprising amino acid residues 151, 5499 and 102127, and ligated using chemoselective thioether ligation chemistry. The synthetic protein was folded and assembled to a tetrameric structure in the presence of the TTRs native ligand, thyroxine, as shown by gel filtration chromatography, native gel electrophoresis, TTR antibody recognition and thyroid hormone binding. In the current study the solution structure of the first of these fragment peptides, TTR(151) is examined to determine its intrinsic propensity to form beta-sheet structure, potentially involved in amyloid fibril formation by TTR. Despite the presence of extensive beta-structure in the native form of the protein, the Nterminal fragment adopts an essentially random coil conformation in solution

A phenomenological approach to the simulation of metabolism and proliferation dynamics of large tumour cell populations

A major goal of modern computational biology is to simulate the collective behaviour of large cell populations starting from the intricate web of molecular interactions occurring at the microscopic level. In this paper we describe a simplified model of cell metabolism, growth and proliferation, suitable for inclusion in a multicell simulator, now under development (Chignola R and Milotti E 2004 Physica A 338 261-6). Nutrients regulate the proliferation dynamics of tumor cells which adapt their behaviour to respond to changes in the biochemical composition of the environment. This modeling of nutrient metabolism and cell cycle at a mesoscopic scale level leads to a continuous flow of information between the two disparate spatiotemporal scales of molecular and cellular dynamics that can be simulated with modern computers and tested experimentally.Comment: 58 pages, 7 figures, 3 tables, pdf onl

Optimal Control for Multi-mode Systems with Discrete Costs

This paper studies optimal time-bounded control in multi-mode systems with discrete costs. Multi-mode systems are an important subclass of linear hybrid systems, in which there are no guards on transitions and all invariants are global. Each state has a continuous cost attached to it, which is linear in the sojourn time, while a discrete cost is attached to each transition taken. We show that an optimal control for this model can be computed in NEXPTIME and approximated in PSPACE. We also show that the one-dimensional case is simpler: although the problem is NP-complete (and in LOGSPACE for an infinite time horizon), we develop an FPTAS for finding an approximate solution.Comment: extended version of a FORMATS 2017 pape

Hormone affinity and fibril formation of piscine transthyretin: the role of the N-terminal

Transthyretin (TTR) transports thyroid hormones (THs), thyroxine (T4) and triiodothyronine (T3) in the blood of vertebrates. TH-binding sites are highly conserved in vertebrate TTR however, piscine TTR has a longer N-terminus which is thought to influence TH-binding affinity and may influence TTR stability. We produced recombinant wild-type sea bream TTR (sbTTRWT) plus two mutants in which six (sbTTRM6) and twelve (sbTTRM12) N-terminal residues were removed. Ligandbinding studies revealed similar affinities for T3 (Kd=10.6±1.7nM) and T4 (Kd=9.8±0.97nM) binding to sbTTRWT. Affinity for THs was unaltered in sbTTRM12 but sbTTRM6 had poorer affinity for T4 (Kd=252.3±15.8nM) implying that some residues in the N-terminus can influence T4 binding. sbTTRM6 inhibited acid-mediated fibril formation in vitro as shown by fluorometric measurements using thioflavine-T.In contrast, fibril formation by sbTTRM12 was significant, probably due to decreased stability of the tetramer. Such studies also suggested that sbTTRWT is more resistant to fibril formation than human TTR

An amphitropic cAMP-binding protein in yeast mitochondria

ABSTRACT: We describe the first example of a mitochondrial protein with a covalently attached phos-phatidylinositol moiety acting as a membrane anchor. The protein can be metabolically labeled with both stearic acid and inositol. The stearic acid label is removed by phospholipase D whereupon the protein with the retained inositol label is released from the membrane. This protein is a cAMP receptor of the yeast Saccharomyces cereuisiae and tightly associated with the inner mitochondrial membrane. However, it is converted into a soluble form during incubation of isolated mitochondria with Ca2+ and phospholipid (or lipid derivatives). This transition requires the action of a proteinaceous, N-ethylmaleimide-sensitive component of the intermembrane space and is accompanied by a decrease in the lipophilicity of the cAMP receptor. We propose that the component of the intermembrane space triggers the amphitropic behavior of the mitochondrial lipid-modified CAMP-binding protein through a phospholipase activity. Only in recent years specific fatty acids have been recog-nized to play important roles in the association of proteins with membranes. Both noncovalent and covalent interactions be-tween fatty acids and proteins have been reported. Among the latter are GTP-binding proteins (Molenaar et al., 1988)

One-counter Markov decision processes

We study the computational complexity of central analysis problems for One-Counter Markov Decision Processes (OC-MDPs), a class of finitely-presented, countable-state MDPs. OC-MDPs are equivalent to a controlled extension of (discrete-time) Quasi-Birth-Death processes (QBDs), a stochastic model studied heavily in queueing theory and applied probability. They can thus be viewed as a natural ``adversarial'' version of a classic stochastic model. Alternatively, they can also be viewed as a natural probabilistic/controlled extension of classic one-counter automata. OC-MDPs also subsume (as a very restricted special case) a recently studied MDP model called ``solvency games'' that model a risk-averse gambling scenario. Basic computational questions about these models include ``termination'' questions and ``limit'' questions, such as the following: does the controller have a ``strategy'' (or ``policy'') to ensure that the counter (which may for example count the number of jobs in the queue) will hit value 0 (the empty queue) almost surely (a.s.)? Or that it will have infinite limsup value, a.s.? Or, that it will hit value 0 in selected terminal states, a.s.? Or, in case these are not satisfied a.s., compute the maximum (supremum) such probability over all strategies. We provide new upper and lower bounds on the complexity of such problems. For some of them we present a polynomial-time algorithm, whereas for others we show PSPACE- or BH-hardness and give an EXPTIME upper bound. Our upper bounds combine techniques from the theory of MDP reward models, the theory of random walks, and a variety of automata-theoretic methods