Formal Controller Synthesis for Markov Jump Linear Systems with Uncertain Dynamics

Automated synthesis of provably correct controllers for cyber-physical systems is crucial for deployment in safety-critical scenarios. However, hybrid features and stochastic or unknown behaviours make this problem challenging. We propose a method for synthesising controllers for Markov jump linear systems (MJLSs), a class of discrete-time models for cyber-physical systems, so that they certifiably satisfy probabilistic computation tree logic (PCTL) formulae. An MJLS consists of a finite set of stochastic linear dynamics and discrete jumps between these dynamics that are governed by a Markov decision process (MDP). We consider the cases where the transition probabilities of this MDP are either known up to an interval or completely unknown. Our approach is based on a finite-state abstraction that captures both the discrete (mode-jumping) and continuous (stochastic linear) behaviour of the MJLS. We formalise this abstraction as an interval MDP (iMDP) for which we compute intervals of transition probabilities using sampling techniques from the so-called 'scenario approach', resulting in a probabilistically sound approximation. We apply our method to multiple realistic benchmark problems, in particular, a temperature control and an aerial vehicle delivery problem.Comment: 14 pages, 6 figures, under review at QES

Demand flexibility management for buildings-to-grid integration with uncertain generation

Contains fulltext : 228323.pdf (publisher's version ) (Open Access

Correct-by-construction reach-avoid control of partially observable linear stochastic systems

We study feedback controller synthesis for reach-avoid control of discrete-time, linear time-invariant (LTI) systems with Gaussian process and measurement noise. The problem is to compute a controller such that, with at least some required probability, the system reaches a desired goal state in finite time while avoiding unsafe states. Due to stochasticity and nonconvexity, this problem does not admit exact algorithmic or closed-form solutions in general. Our key contribution is a correct-by-construction controller synthesis scheme based on a finite-state abstraction of a Gaussian belief over the unmeasured state, obtained using a Kalman filter. We formalize this abstraction as a Markov decision process (MDP). To be robust against numerical imprecision in approximating transition probabilities, we use MDPs with intervals of transition probabilities. By construction, any policy on the abstraction can be refined into a piecewise linear feedback controller for the LTI system. We prove that the closed-loop LTI system under this controller satisfies the reach-avoid problem with at least the required probability. The numerical experiments show that our method is able to solve reach-avoid problems for systems with up to 6D state spaces, and with control input constraints that cannot be handled by methods such as the rapidly-exploring random belief trees (RRBT)

Cold-storage defects in butter and their relation to the autoxidation of unsaturated fatty acids

In this thesis investigations are described of the identification of aroma compounds which are formed as a result of oxidative deterioration of butter during cold storage, producing a typical trainy (fishy) off-flavour. As these flavour defects are caused chiefly by autoxidative breakdown of unsaturated fatty acids, it was also studied which fatty acids may act as precursors in the formation of these off-flavours. For this purpose the volatile odorous compounds formed in the autoxidation of pure unsaturated fatty acids were identified and compared with the compounds formed in butter with cold-storage defects.In the introductory chapter (I), a survey is given of results obtained in previous investigations concerning the aroma compounds in butter with coldstorage defects. Aliphatic carbonyl compounds in particular are considered to be responsible for these flavour defects, but these components were isolated only from washed cream and sweet-cream butter, and not from cold-stored butter made from soured cream. In addition, several classes of compounds such as carbonyl compounds with double bonds not in conjunction with the carbonyl group and cis/trans -isomers have been overlooked in previous investigations, because of inadequate analytical techniques.In Chapter II a survey is given of the mechanisms of autoxidation of fatty acids. The mechanisms are discussed of the radical chain reactions leading to the formation of hydroperoxides, and of the dismutation of these compounds to secondary autoxidation products. The latter reactions are of particular interest, as they result in the formation of aroma compounds which may cause flavour defects.In Table 2 a survey is given of the hydroperoxides and the secondary oxidation products that may be expected in the autoxidation of a number of unsaturated fatty acids, and of those which were actually found in previous investigations.Chapter III deals with flavour defects observed in autoxidized food lipids in general and cold-storage defects of butter in particular.First of all, the organoleptic properties and flavour threshold values are given of many secondary autoxidation products (see Table 3) which have very low flavour threshold values and highly repugnant odours.Secondly, autoxidation experiments are described which indicate the contribution of various unsaturated fatty acids to autoxidation off-flavours.Thirdly, the generation of oxidation defects in lipid-containing foods, and the effect of several factors on this type of deterioration, are discussed.Finally, special attention is paid to the development of cold-storage defects in butter. The main factors which influence the keeping quality of cold-stored butter are: the extent to which milk, cream and butter are contaminated with copper, and the pH of the butter serum.It is known that the oxidation processes in butter, which may cause inter alia a trainy flavour, start at the fat/serum interface. Experiments were carried out which provided further evidence that the oxidative deterioration of coldstored butter is due primarily to oxidation of the phospholipids. Further experiments show, however, that 'artificial butter' made from butter fat and milk serum (without fat globule membrane material or phospholipids) may also develop a trainy flavour, provided that a strong pro-oxidant (copper) is present at the lipid/water interface. It must therefore be concluded that it is uncertain whether the off-flavours are formed by oxidation of the unsaturated fatty acids of the phospholipids, or of the butter fat, or even of both.Chapter IV consists of a description of the techniques which were used for the isolation and identification of volatile flavour compounds in oxidized lipids. The special problems concerning the isolation and separation of volatile flavour compounds are outlined.For the isolation of volatile flavour compounds in butter fat, a semi-continuous high-vacuum distillation apparatus was designed. Aroma compounds were isolated from small samples of oxidized lipids by means of a batch-wise high-vacuum distillation technique.The aroma components were first separated by temperature-programmed gas chromatography, using a type of column which minimized the risk of artefact formation.For further separation and identification, the carbonyl compounds from the fractions obtained by gas chromatography were converted into their DNPH-derivatives, because the latter have favourable properties for liquid-chromatographic and spectrometric analyses. A special conversion reaction was used to avoid isomerization and artefact formation. A method of silver nitrate complex chromatography was used for the further separation of mixtures of DNPH's according to type and degree of unsaturation.After these manipulations, the DNPH's were in general separated into the individual compounds. For their further identification ultraviolet, infrared and mass spectrometry were used. Many reference DNPH's have enabled specific characteristics to be determined (absorption wavelengths in ultraviolet and infrared spectrometry, and m/e -values in mass spectrometry).In a few cases odorous compounds other than carbonyl compounds had to be identified. These compounds were separated by gas chromatography (using a second column, if necessary) and subsequently used for spectrometric analysis.Figure 9 is a diagram which visualizes the combination of techniques used for identification of volatile compounds obtained from oxidized lipids. It could be established that even very labile compounds such as 2 cis -enals, 3 cis -enals, etc. were not modified and could be identified correctly by means of this combination of techniques.In Chapter V the results are presented of investigations on the identification of volatile compounds contributing to the trainy flavour of cold-stored butter. For the sake of comparison, the aroma compounds from fresh butter without flavour defects have also been analysed.In Chapter VI the results are given of the identification of aroma compounds which are formed in the autoxidation of a number of unsaturated fatty acids.Many compounds have been identified, which have not been detected previously in butter with oxidation defects or in autoxidized fatty acids.In Chapter VII the results of the present investigations are discussed and summarized. From the analysis of trainy butter and butter without flavour defects it can be seen that the odorous fraction shows large changes as a result of the oxidative deterioration. Many of the aroma compounds in trainy butter are the same as those found in autoxidized fatty acids.By comparing the composition of the mixture of odorous compounds from trainy butter with that of compounds from autoxidized fatty acids (see Table 25 and Figure 10) it has been possible to determine that the autoxidation of unsaturated fatty acids and particularly the autoxidation of linolenic acid (and fatty acids with the same alkyl-terminal structure) chiefly contributes to the development of a trainy flavour. This conclusion was confirmed experimentally; the addition of autoxidized linolenic acid (with a small quantity of arachidonic acid) to butter without flavour defects caused a distinctly trainy flavour.The analyses of aroma compounds which result from the autoxidation of a number of pure unsaturated fatty acids have established that the aroma compounds formed are well in line with those that can be expected from the mechanisms of autoxidation of unsaturated fatty acids and the dismutation of the hydroperoxides (see Tables 21 to 24).It was established that the changes in flavour in the first stages of oxidative deterioration of cold-stored butter can be ascribed to the rapid oxidation of the highly unsaturated fatty acids, which play the main part in the development of a fatty-metallic off-flavour. The tallowy flavour in the last stage of oxidative deterioration of cold-stored butter can be ascribed to the continued oxidation of several fatty acids.This thesis, although dealing in particular with cold-storage defects in butter, also presents general information on the development of oxidation flavours in lipid-containing food products

Decision-Making Under Uncertainty: Beyond Probabilities

This position paper reflects on the state-of-the-art in decision-making under uncertainty. A classical assumption is that probabilities can sufficiently capture all uncertainty in a system. In this paper, the focus is on the uncertainty that goes beyond this classical interpretation, particularly by employing a clear distinction between aleatoric and epistemic uncertainty. The paper features an overview of Markov decision processes (MDPs) and extensions to account for partial observability and adversarial behavior. These models sufficiently capture aleatoric uncertainty but fail to account for epistemic uncertainty robustly. Consequently, we present a thorough overview of so-called uncertainty models that exhibit uncertainty in a more robust interpretation. We show several solution techniques for both discrete and continuous models, ranging from formal verification, over control-based abstractions, to reinforcement learning. As an integral part of this paper, we list and discuss several key challenges that arise when dealing with rich types of uncertainty in a model-based fashion

Efficient Sensitivity Analysis for Parametric Robust Markov Chains

We provide a novel method for sensitivity analysis of parametric robust Markov chains. These models incorporate parameters and sets of probability distributions to alleviate the often unrealistic assumption that precise probabilities are available. We measure sensitivity in terms of partial derivatives with respect to the uncertain transition probabilities regarding measures such as the expected reward. As our main contribution, we present an efficient method to compute these partial derivatives. To scale our approach to models with thousands of parameters, we present an extension of this method that selects the subset of $k$ parameters with the highest partial derivative. Our methods are based on linear programming and differentiating these programs around a given value for the parameters. The experiments show the applicability of our approach on models with over a million states and thousands of parameters. Moreover, we embed the results within an iterative learning scheme that profits from having access to a dedicated sensitivity analysis.Comment: To be presented at CAV 202

CTMCs with Imprecisely Timed Observations

Labeled continuous-time Markov chains (CTMCs) describe processes subject to random timing and partial observability. In applications such as runtime monitoring, we must incorporate past observations. The timing of these observations matters but may be uncertain. Thus, we consider a setting in which we are given a sequence of imprecisely timed labels called the evidence. The problem is to compute reachability probabilities, which we condition on this evidence. Our key contribution is a method that solves this problem by unfolding the CTMC states over all possible timings for the evidence. We formalize this unfolding as a Markov decision process (MDP) in which each timing for the evidence is reflected by a scheduler. This MDP has infinitely many states and actions in general, making a direct analysis infeasible. Thus, we abstract the continuous MDP into a finite interval MDP (iMDP) and develop an iterative refinement scheme to upper-bound conditional probabilities in the CTMC. We show the feasibility of our method on several numerical benchmarks and discuss key challenges to further enhance the performance

CTMCs with Imprecisely Timed Observations

Labeled continuous-time Markov chains (CTMCs) describe processes subject to random timing and partial observability. In applications such as runtime monitoring, we must incorporate past observations. The timing of these observations matters but may be uncertain. Thus, we consider a setting in which we are given a sequence of imprecisely timed labels called the evidence. The problem is to compute reachability probabilities, which we condition on this evidence. Our key contribution is a method that solves this problem by unfolding the CTMC states over all possible timings for the evidence. We formalize this unfolding as a Markov decision process (MDP) in which each timing for the evidence is reflected by a scheduler. This MDP has infinitely many states and actions in general, making a direct analysis infeasible. Thus, we abstract the continuous MDP into a finite interval MDP (iMDP) and develop an iterative refinement scheme to upper-bound conditional probabilities in the CTMC. We show the feasibility of our method on several numerical benchmarks and discuss key challenges to further enhance the performance

Efficient Sensitivity Analysis for Parametric Robust Markov Chains

We provide a novel method for sensitivity analysis of parametric robust Markov chains. These models incorporate parameters and sets of probability distributions to alleviate the often unrealistic assumption that precise probabilities are available. We measure sensitivity in terms of partial derivatives with respect to the uncertain transition probabilities regarding measures such as the expected reward. As our main contribution, we present an efficient method to compute these partial derivatives. To scale our approach to models with thousands of parameters, we present an extension of this method that selects the subset of $k$ parameters with the highest partial derivative. Our methods are based on linear programming and differentiating these programs around a given value for the parameters. The experiments show the applicability of our approach on models with over a million states and thousands of parameters. Moreover, we embed the results within an iterative learning scheme that profits from having access to a dedicated sensitivity analysis