Value Iteration for Simple Stochastic Games: Stopping Criterion and Learning Algorithm

Simple stochastic games can be solved by value iteration (VI), which yields a sequence of under-approximations of the value of the game. This sequence is guaranteed to converge to the value only in the limit. Since no stopping criterion is known, this technique does not provide any guarantees on its results. We provide the first stopping criterion for VI on simple stochastic games. It is achieved by additionally computing a convergent sequence of over-approximations of the value, relying on an analysis of the game graph. Consequently, VI becomes an anytime algorithm returning the approximation of the value and the current error bound. As another consequence, we can provide a simulation-based asynchronous VI algorithm, which yields the same guarantees, but without necessarily exploring the whole game graph.Comment: CAV201

LNCS

Discrete-time Markov Chains (MCs) and Markov Decision Processes (MDPs) are two standard formalisms in system analysis. Their main associated quantitative objectives are hitting probabilities, discounted sum, and mean payoff. Although there are many techniques for computing these objectives in general MCs/MDPs, they have not been thoroughly studied in terms of parameterized algorithms, particularly when treewidth is used as the parameter. This is in sharp contrast to qualitative objectives for MCs, MDPs and graph games, for which treewidth-based algorithms yield significant complexity improvements. In this work, we show that treewidth can also be used to obtain faster algorithms for the quantitative problems. For an MC with n states and m transitions, we show that each of the classical quantitative objectives can be computed in O((n+m)⋅t2) time, given a tree decomposition of the MC with width t. Our results also imply a bound of O(κ⋅(n+m)⋅t2) for each objective on MDPs, where κ is the number of strategy-iteration refinements required for the given input and objective. Finally, we make an experimental evaluation of our new algorithms on low-treewidth MCs and MDPs obtained from the DaCapo benchmark suite. Our experiments show that on low-treewidth MCs and MDPs, our algorithms outperform existing well-established methods by one or more orders of magnitude

The Hanoi Omega-Automata Format

We propose a flexible exchange format for ω-automata, as typically used in formal verification, and implement support for it in a range of established tools. Our aim is to simplify the interaction of tools, helping the research community to build upon other people’s work. A key feature of the format is the use of very generic acceptance conditions, specified by Boolean combinations of acceptance primitives, rather than being limited to common cases such as Büchi, Streett, or Rabin. Such flexibility in the choice of acceptance conditions can be exploited in applications, for example in probabilistic model checking, and furthermore encourages the development of acceptance-agnostic tools for automata manipulations. The format allows acceptance conditions that are either state-based or transition-based, and also supports alternating automata

Produção de pellets à base de hidroxiapatite contendo um fármaco modelo

Tese de mestrado, Farmacotecnia Avançada, Universidade de Lisboa, Faculdade de Farmácia, 2012Este trabalho foi realizado tendo como objectivo principal a produção de pellets à base de hidroxiapatite com diferentes dimensões e porosidades, uma vez que a utilização de pellets de dimensões variadas permite um melhor preenchimento do espaço/cavidades ósseas e que a porosidade é um factor promotor da osteointegração. Os pellets foram produzidos através da tecnologia de extrusão-esferonização utilizando fieiras de diferentes diâmetros. Como potencial promotor da formação dos poros foi incluído cloreto de sódio com diferentes granulometrias na estrutura dos pellets. Os pellets produzidos foram sinterizados e posteriormente mergulhados sob vácuo numa solução de ibuprofeno para incorporação desta substância activa na sua estrutura. Os resultados obtidos neste trabalho mostram que a tecnologia utilizada permitiu produzir pellets de diferentes dimensões por extrusão-esferonização embora não tenha sido possível optimizar a produção desses pellets com estruturas macroporosas. Foi possível verificar que o aumento da temperatura de sinterização conduziu a uma maior densificação do material com redução da porosidade e que o cloreto de sódio presente nos pellets conduziu a uma menor contracção dos mesmos quando comparados com os das formulações testadas contendo hidroxiapatite modificada sem cloreto de sódio. Adicionalmente os resultados obtidos no ensaio de dissolução indicam que o ibuprofeno deverá ter ficado apenas à superfície dos pellets. Ainda que não tenha sido possível obter as estruturas macroporosas propostas, os restantes objectivos deste trabalho foram atingidos, tendo sido possível produzir pellets de hidroxiapatite utilizando a tecnologia de extrusão-esferonização e sua caracterização.The purpose of this work was to produce pellets of hydroxyapatite with different sizes and porosity. This was done because the use of pellets with different dimensions allows a better filling of bone cavities and the porosity is an important variable of osteointegration. The pellets were produced by extrusion-spheronization using extrusion screens of different diameters. Sodium chloride with different particle sizes was added to the pellets formulation as a potencial porogenic component. The pellets were sintered and afterwards they were soaked under vacuum in an ibubrofen solution to promote its incorporation in the structure. The data produced revealed that the technology of extrusion-spheronization was adequate to manufacture pellets with different sizes although it was not possible to optimize macroporous structures. It was possible to verify that the sintering temperature led to a densification of material with porosity reduction and that the sodium chloride led to lower collapse of the pellets when compared with the tested formulations with modified hydroxyapatite without sodium chloride. Adicionally, the dissolution results indicate that ibuprofen should have stayed only on the pellets external surface. Although it was not possible to obtain the proposed macroporous structures, the other objectives of this study were attained, having been possible to produce hydroxyapatite pellets by the extrusion-spheronization technology and characterize them

PAC Statistical Model Checking of Mean Payoff in Discrete- and Continuous-Time MDP

Markov decision processes (MDP) and continuous-time MDP (CTMDP) are the fundamental models for non-deterministic systems with probabilistic uncertainty. Mean payoff (a.k.a. long-run average reward) is one of the most classic objectives considered in their context. We provide the first algorithm to compute mean payoff probably approximately correctly in unknown MDP; further, we extend it to unknown CTMDP. We do not require any knowledge of the state space, only a lower bound on the minimum transition probability, which has been advocated in literature. In addition to providing probably approximately correct (PAC) bounds for our algorithm, we also demonstrate its practical nature by running experiments on standard benchmarks.Comment: Full version of CAV 2022 paper, 57 page

Reachability of Hennessy–Milner properties for weakly extended PRS

We examine the problem whether a given weakly extended process rewrite system (wPRS) contains a reachable state satisfying a given formula of Hennessy–Milner logic. We show that this problem is decidable. As a corollary we observe that the problem of strong bisimilarity between wPRS and finite-state systems is decidable. Decidability of the same problem for wPRS subclasses, namely PAN and PRS, has been formulated as an open question, see e.g. [Srb02]. We also strengthen some related undecidability results on some PRS subclasses