Policy learning for time-bounded reachability in Continuous-Time Markov Decision Processes via doubly-stochastic gradient ascent

Continuous-time Markov decision processes are an important class of models in a wide range of applications, ranging from cyber-physical systems to synthetic biology. A central problem is how to devise a policy to control the system in order to maximise the probability of satisfying a set of temporal logic specifications. Here we present a novel approach based on statistical model checking and an unbiased estimation of a functional gradient in the space of possible policies. The statistical approach has several advantages over conventional approaches based on uniformisation, as it can also be applied when the model is replaced by a black box, and does not suffer from state-space explosion. The use of a stochastic gradient to guide our search considerably improves the efficiency of learning policies. We demonstrate the method on a proof-of-principle non-linear population model, showing strong performance in a non-trivial task

Performance Evaluation of Train Moving-Block Control

International audienceIn moving block systems for railway transportation a central controller periodically communicates to the train how far it can safely advance. On-board automatic protection mechanisms stop the train if no message is received during a given time window. In this paper we consider as reference a typical implementation of moving-block control for metro and quantify the rate of spurious Emergency Brakes (EBs), i.e. of train stops due to communication losses and not to an actual risk of collision. Such unexpected EBs can happen at any point on the track and are a major service disturbance. Our general formula for the EB rate requires a probabilistic characterization of losses and delays. Calculations are surprisingly simple in the case of homogeneous and independent packet losses. Our approach is computationally efficient even when emergency brakes are very rare (as they should be) and can no longer be estimated via discrete-event simulations

Fluid approximation of broadcasting systems

Nature-inspired paradigms have been proposed to design and forecast behaviour of open distributed systems, such as sensor networks and the internet of things. In these paradigms system behaviour emerges from (complex) interactions among a large number of agents. Modelling these interactions in terms of classical point-to-point communication is often not practical. This is due to the large scale and the open nature of the systems, which means that partners for point-to-point communication may not be available at any given time. Nevertheless the need for efficient formal verification of qualitative and quantitative properties of these systems is of utmost importance, especially given their proposed pervasive and transparent nature. CARMA is a recently proposed formal modelling language for open distributed systems, which is equipped with a broadcast communication in order to meet the communication challenges of such systems. The inclusion of quantitative information about the timing and probability of actions gives rise to models suitable for analysing questions such as the probability that information will achieve total coverage within a system, or the expected market share that might be gained by competing service providers relying on viral advertising. The ability to express models is not the only challenge, because the scale of the systems we are interested in often defies discrete state-based analysis techniques such as stochastic simulation. This is the problem that we address in this paper as we consider how to provide an efficient fluid approximation, supporting efficient and accurate quantitative analysis of large scale systems, for a language that incorporates broadcast communication

Stochastic modelling of spatial collective adaptive systems

Collective Adaptive Systems (CAS) are composed of individual agents with internal knowledge and rules which organize themselves into ensembles. These ensembles can often be observed to exhibit behaviour resembling that of a single entity with a clear goal and a consistent internal knowledge, even when the individual agents within the ensemble are not managed by any outside, globally-accessible entity. Because of their lack of a need for centralized control which results in high robustness, CAS are commonly observed in nature – and for similar reasons are often reflected in human engineered systems. Researching the patterns of operation observed in such systems provides meaningful insight into how to design and optimise stable multiagent systems capable of withstanding adverse conditions. Formal modelling provides valuable intellectual tools which can be applied to the problem of analysis of systems by means of modelling and simulation. In this thesis we explore the modelling of CAS in which space (topology and distances) plays a significant role. Working with CARMA (Collective Adaptive Resource-sharing Markovian Agents) a formal feature-rich language for modelling stochastic CAS, we investigate a number of spatial CAS scenarios from the realm of urban planning. When components operate in a spatial context, their behaviour can be affected by where they are located in that space. For example, their location can influence the speed at which they move, and their ability to communicate with other components. Components in CARMA have internal store, and behaviour expressed by Markov processes. They can communicate with each other through sending messages on state transitions in a unicast or broadcast fashion. Simulation with pseudo-random events can be used to obtain values of measures applied to CARMA models, providing a basis for analysis and optimisation. The CARMA models developed in the case studies are data-driven and the results of simulating these models are compared with real-world data. In particular, we explore two scenarios: crowd-routing and city transportation systems. Building on top of CARMA, we also introduce CGP (CARMA Graphical Plugin), a novel graphical software tool for graphically specifying spatial CAS systems with the feature of automatic translation into CARMA models. We also supply CARMA with additional syntax structures for expressing spatial constructs