Value Iteration for Long-run Average Reward in Markov Decision Processes

Markov decision processes (MDPs) are standard models for probabilistic systems with non-deterministic behaviours. Long-run average rewards provide a mathematically elegant formalism for expressing long term performance. Value iteration (VI) is one of the simplest and most efficient algorithmic approaches to MDPs with other properties, such as reachability objectives. Unfortunately, a naive extension of VI does not work for MDPs with long-run average rewards, as there is no known stopping criterion. In this work our contributions are threefold. (1) We refute a conjecture related to stopping criteria for MDPs with long-run average rewards. (2) We present two practical algorithms for MDPs with long-run average rewards based on VI. First, we show that a combination of applying VI locally for each maximal end-component (MEC) and VI for reachability objectives can provide approximation guarantees. Second, extending the above approach with a simulation-guided on-demand variant of VI, we present an anytime algorithm that is able to deal with very large models. (3) Finally, we present experimental results showing that our methods significantly outperform the standard approaches on several benchmarks

Interrupt Timed Automata: verification and expressiveness

We introduce the class of Interrupt Timed Automata (ITA), a subclass of hybrid automata well suited to the description of timed multi-task systems with interruptions in a single processor environment. While the reachability problem is undecidable for hybrid automata we show that it is decidable for ITA. More precisely we prove that the untimed language of an ITA is regular, by building a finite automaton as a generalized class graph. We then establish that the reachability problem for ITA is in NEXPTIME and in PTIME when the number of clocks is fixed. To prove the first result, we define a subclass ITA- of ITA, and show that (1) any ITA can be reduced to a language-equivalent automaton in ITA- and (2) the reachability problem in this subclass is in NEXPTIME (without any class graph). In the next step, we investigate the verification of real time properties over ITA. We prove that model checking SCL, a fragment of a timed linear time logic, is undecidable. On the other hand, we give model checking procedures for two fragments of timed branching time logic. We also compare the expressive power of classical timed automata and ITA and prove that the corresponding families of accepted languages are incomparable. The result also holds for languages accepted by controlled real-time automata (CRTA), that extend timed automata. We finally combine ITA with CRTA, in a model which encompasses both classes and show that the reachability problem is still decidable. Additionally we show that the languages of ITA are neither closed under complementation nor under intersection

Widest Paths and Global Propagation in Bounded Value Iteration for Stochastic Games

Solving stochastic games with the reachability objective is a fundamental problem, especially in quantitative verification and synthesis. For this purpose, bounded value iteration (BVI) attracts attention as an efficient iterative method. However, BVI's performance is often impeded by costly end component (EC) computation that is needed to ensure convergence. Our contribution is a novel BVI algorithm that conducts, in addition to local propagation by the Bellman update that is typical of BVI, global propagation of upper bounds that is not hindered by ECs. To conduct global propagation in a computationally tractable manner, we construct a weighted graph and solve the widest path problem in it. Our experiments show the algorithm's performance advantage over the previous BVI algorithms that rely on EC computation.Comment: v2: an URL to the implementation is adde

High-throughput DNA sequencing of museum specimens sheds light on the long-missing species of the Bokermannohyla claresignata group (Anura: Hylidae: Cophomantini)

The two species of the Bokermannohyla claresignata species group (Anura: Hylidae) have not been collected for the last four decades. It is the only species group of the hyline tribe Cophomantini that has not yet been analysed genetically. Its phylogenetic position is thus uncertain, and it has a combination of adult and larval character states that make this group a crucial missing piece that hinders our understanding of Cophomantini phylogenetics and character evolution. We obtained DNA sequences from a museum larval specimen of Bok. claresignata, using specialized extraction methods and high-throughput DNA sequencing, and combined the molecular phylogenetic results with available phenotypic information to provide new insights into the taxonomy and phylogenetic relationships of its species group. Our phylogenetic results place Bok. claresignata as sister to the Boana pulchella group, supporting its inclusion in Boana, together with Bokermannohyla clepsydra. In light of this new finding, we recognize a newly defined Boana claresignata group to accommodate these species, thus resolving both the polyphyly of Bokermannohyla and the paraphyly of Boana. Considering the phylogenetic relationships of the Boana claresignata group, we also discuss the evolution of suctorial tadpoles and mature oocyte/egg pigmentation in Cophomantini

Acute pressure overload of the right ventricle. Comparison of two models of right-left shunt. Pulmonary artery to left atrium and right atrium to left atrium: experimental study

<p>Abtract</p> <p>Background</p> <p>In right ventricular failure (RVF), an interatrial shunt can relieve symptoms of severe pulmonary hypertension by reducing right ventricular preload and increasing systemic flow. Using a pig model to determine if a pulmonary artery - left atrium shunt (PA-LA) is better than a right atrial - left atrial shunt (RA-LA), we compared the hemodynamic effects and blood gases between the two shunts.</p> <p>Methods</p> <p>Thirty, male Large White pigs weighting in average 21.3 kg ± 0.7 (SEM) were divided into two groups (15 pigs per group): In group 1, banding of the pulmonary artery and a pulmonary artery to left atrium shunt with an 8 mm graft (PA-LA) was performed and in group 2 banding of the pulmonary artery and right atrial to left atrial shunt (RA-LA) with a similar graft was performed. Hemodynamic parameters and blood gases were measured from all cardiac chambers in 10 and 20 minutes, half and one hour interval from the baseline (30 min from the banding). Cardiac output and flow of at the left anterior descending artery was also monitored.</p> <p>Results</p> <p>In both groups, a stable RVF was generated. The PA-LA shunt compared to the RA-LA shunt has better hemodynamic performance concerning the decreased right ventricle afterload, the 4 fold higher mean pressure of the shunt, the better flow in left anterior descending artery and the decreased systemic vascular resistance. Favorable to the PA-LA shunt is also the tendency - although not statistically significant - in relation to central venous pressure, left atrial filling and cardiac output.</p> <p>Conclusion</p> <p>The PA-LA shunt can effectively reverse the catastrophic effects of acute RVF offering better hemodynamic characteristics than an interatrial shunt.</p

Value Iteration for Simple Stochastic Games: Stopping Criterion and Learning Algorithm

Simple stochastic games can be solved by value iteration (VI), which yields a sequence of under-approximations of the value of the game. This sequence is guaranteed to converge to the value only in the limit. Since no stopping criterion is known, this technique does not provide any guarantees on its results. We provide the first stopping criterion for VI on simple stochastic games. It is achieved by additionally computing a convergent sequence of over-approximations of the value, relying on an analysis of the game graph. Consequently, VI becomes an anytime algorithm returning the approximation of the value and the current error bound. As another consequence, we can provide a simulation-based asynchronous VI algorithm, which yields the same guarantees, but without necessarily exploring the whole game graph.Comment: CAV201

Notch signaling during human T cell development

Notch signaling is critical during multiple stages of T cell development in both mouse and human. Evidence has emerged in recent years that this pathway might regulate T-lineage differentiation differently between both species. Here, we review our current understanding of how Notch signaling is activated and used during human T cell development. First, we set the stage by describing the developmental steps that make up human T cell development before describing the expression profiles of Notch receptors, ligands, and target genes during this process. To delineate stage-specific roles for Notch signaling during human T cell development, we subsequently try to interpret the functional Notch studies that have been performed in light of these expression profiles and compare this to its suggested role in the mouse

Emergent Properties of Patch Shapes Affect Edge Permeability to Animals

Animal travel between habitat patches affects populations, communities and ecosystems. There are three levels of organization of edge properties, and each of these can affect animals. At the lowest level are the different habitats on each side of an edge, then there is the edge itself, and finally, at the highest level of organization, is the geometry or structure of the edge. This study used computer simulations to (1) find out whether effects of edge shapes on animal behavior can arise as emergent properties solely due to reactions to edges in general, without the animals reacting to the shapes of the edges, and to (2) generate predictions to allow field and experimental studies to test mechanisms of edge shape response. Individual animals were modeled traveling inside a habitat patch that had different kinds of edge shapes (convex, concave and straight). When animals responded edges of patches, this created an emergent property of responding to the shape of the edge. The response was mostly to absolute width of the shapes, and not the narrowness of them. When animals were attracted to edges, then they tended to collect in convexities and disperse from concavities, and the opposite happened when animals avoided edges. Most of the responses occurred within a distance of 40% of the perceptual range from the tip of the shapes. Predictions were produced for directionality at various locations and combinations of treatments, to be used for testing edge behavior mechanisms. These results suggest that edge shapes tend to either concentrate or disperse animals, simply because the animals are either attracted to or avoid edges, with an effect as great as 3 times the normal density. Thus edge shape could affect processes like pollination, seed predation and dispersal and predator abundance