Formalization of Discrete-time Markov Chains in HOL

Markov chains are extensively used in the modeling and analysis of engineering and scientific problems which can be expressed as random processes with the memoryless property. Usually, paper-and-pencil proofs, simulation or computer algebra software are used to analyze Markovian models. However, these techniques either are not scalable or do not guarantee accurate results, which are vital in safety-critical systems. To improve the accuracy of the analysis, probabilistic model checking has been recently proposed to formally analyze Markovian systems. However, model checking suffers from the inherent state-explosion problem and thus has a very limited scope in terms of analyzing Markovian models.\newline \indent In order to overcome the above mentioned limitations, this thesis advocates the usage of higher-order-logic theorem proving for conducting the analysis of Markov chains. We present the higher-order-logic formalization of Discrete-time Markov Chains with finite number of discrete states. We also verify some of their most widely used properties using a theorem prover. These foundations allow us to formally express and reason about Markov chains within the sound core of a theorem prover and thus attain precise results. Moreover, by building upon these foundational results, this thesis also presents the formalization of classified discrete-time Markov chains and hidden Markov chains in higher-order logic. These are widely used concepts in the analysis of Markovian models and thus allow us to tackle the formal analysis of a wide range of engineering and scientific systems. For illustration purposes, the thesis also presents some applications including a binary communication channel, the automatic mail quality measurement (AMQM) protocol, a DNA sequence, a least recently used (LRU) stack model and the birth-death process

Stochastic demographic dynamics and economic growth: an application and insights from the world data

'This research has two broad objectives: First, to model population growth in a stochastic framework such that the effects of possible non-mean convergent shocks could be studied theoretically on long-run economic growth and planning. Second, an empirical strategy for modelling stochastic population growth over time is provided. Forecasting exercise has been rigorously carried for population growth and income by embedding the stochastic growth feature of population. For modelling purpose, a long-memory mechanism for population growth is suggested so that the classical economic growth assumption of constant and/ or non-stochastic population growth in economic growth models appear as a limiting case. The analytical results show that embedding the stochastic features of population growth helps in explaining the economic growth volatility. In particular, it is found to be a formidable cause of the presence of long-memory in output. The empirical analysis shows that unless the stochastic feature of population growth is taken into empirical growth models, the author will not be able map out the significant effects of demographic variables consistently over time. It is also shown that how corroborating the information of stochastic shocks of population alters our forecast vision by impacting significantly on the precision of the estimates.' (author's abstract)

An object-oriented representation for efficient reinforcement learning

Rich representations in reinforcement learning have been studied for the purpose of enabling generalization and making learning feasible in large state spaces. We introduce Object-Oriented MDPs (OO-MDPs), a representation based on objects and their interactions, which is a natural way of modeling environments and offers important generalization opportunities. We introduce a learning algorithm for deterministic OO-MDPs and prove a polynomial bound on its sample complexity. We illustrate the performance gains of our representation and algorithm in the wellknown Taxi domain, plus a real-life videogame. 1

An Ontology-Based Approach To Concern-Specific Dynamic Software Structure Monitoring

Software reliability has not kept pace with computing hardware. Despite the use reliability improvement techniques and methods, faults remain that lead to software errors and failures. Runtime monitoring can improve software reliability by detecting certain errors before failures occur. Monitoring is also useful for online and electronic services, where resource management directly impacts reliability and quality. For example, resource ownership errors can accumulate over time (e. g. , as resource leaks) and result in software aging. Early detection of errors allows more time for corrective action before failures or service outages occur. In addition, the ability to monitor individual software concerns, such as application resource ownership structure, can help support autonomic computing for self-healing, self-adapting and self-optimizing software. This thesis introduces ResOwn - an application resource ownership ontology for interactive session-oriented services. ResOwn provides software monitoring with enriched concepts of application resource ownership borrowed from real-world legal and ownership ontologies. ResOwn is formally defined in OWL-DL (Web Ontology Language Description Logic), verified using an off-the-shelf reasoner, and tested using the call processing software for a small private branch exchange (PBX). The ResOwn Prime Directive states that every object in an operational software system is a resource, an owner, or both simultaneously. Resources produce benefits. Beneficiary owners may receive resource benefits. Nonbeneficiary owners may only manage resources. This approach distinguishes resource ownership use from management and supports the ability to detect when a resource's role-based runtime capacity has been exceeded. This thesis also presents a greybox approach to concern-specific, dynamic software structure monitoring including a monitor architecture, greybox interpreter, and algorithms for deriving monitoring model from a monitored target's formal specifications. The target's requirements and design are assumed to be specified in SDL, a formalism based on communicating extended finite state machines. Greybox abstraction, applicable to both behavior and structure, provides direction on what parts, and how much of the target to instrument, and what types of resource errors to detect. The approach was manually evaluated using a number of resource allocation and ownership scenarios. These scenarios were obtained by collecting actual call traces from an instrumented PBX. The results of an analytical evaluation of ResOwn and the monitoring approach are presented in a discussion of key advantages and known limitations. Conclusions and recommended future work are discussed at the end of the thesis

System Design for Digital Experimentation and Explanation Generation

Experimentation increasingly drives everyday decisions in modern life, as it is considered by some to be the gold standard for determining cause and effect within any system. Digital experiments have expanded the scope and frequency of experiments, which can range in complexity from classic A/B tests to contextual bandits experiments, which share features with reinforcement learning. Although there exists a large body of prior work on estimating treatment effects using experiments, this prior work did not anticipate the new challenges and opportu- nities introduced by digital experimentation. Novel errors and threats to validity arise at the intersection of software and experimentation, especially when experimentation is in service of understanding humans behavior or autonomous black-box agents. We present several novel tools for automating aspects of the experimentation- analysis pipeline. We propose new methods for evaluating online field experimentation, automatically generating corresponding analyses of treatment effects. We then draw the connection between software testing and experimental design and argue that applying software testing techniques to a kind of autonomous agent—a deep reinforcement learning agent—to demonstrate the need for novel testing paradigms when a software stack uses learned components that may have emergent behavior. We show how our system may be used to evaluate claims made about the behavior of autonomous agents and find that some claims do not hold up under test. Finally, we show how to produce explanations of the behavior of black-box software-defined agents interacting with white-box environments via automated experimentation. We show how an automated system can be used for exploratory data analysis, with a human in the loop, to investigate a large space of possible counterfactual explanations

Perspectives on adaptive dynamical systems

Adaptivity is a dynamical feature that is omnipresent in nature, socio-economics, and technology. For example, adaptive couplings appear in various real-world systems like the power grid, social, and neural networks, and they form the backbone of closed-loop control strategies and machine learning algorithms. In this article, we provide an interdisciplinary perspective on adaptive systems. We reflect on the notion and terminology of adaptivity in different disciplines and discuss which role adaptivity plays for various fields. We highlight common open challenges, and give perspectives on future research directions, looking to inspire interdisciplinary approaches.Comment: 46 pages, 9 figure

Making the Law More ABLE: Reforming Medicaid for Disability

Passed on the eve of Medicaid's fiftieth anniversary, the Achieving a Better Live Experience (ABLE) Act was a hard-fought victory for individuals with significant disabilities and their families. The law, which creates a new form of tax-preferred savings account, represents an invaluable work-around for highly restrictive Medicaid eligibility requirements. Medicaid eligibility is crucially important for individuals with intellectual, developmental, and other significant disabilities because it provides nearly exclusive access to government- coordinated habilitative care, such as in-home assistance, job supports, and adaptive equipment. These services are necessary to maintain a base-level quality of lfe, facilitate independent living, and preserve the dignity of individuals with disabilities. Despite their importance, they are difflicult to purchase and coordinate in the private market, and due to income and asset holding restrictions on eligibility, only the very poor can access them through Medicaid, even after passage of the Affordable Care Act. This Article argues that despite their facial neutrality, income and asset holding restrictions, commonly referred to as means testing, result in undue hardship when they are applied to the provision of government-coordinated habilitative care for individuals with significant disabilities. Congress's attempts to mitigate this hardship, including the recently passed ABLE Act, are important steps forward, but they also can impose economic, dignitary, and emotional harms on individuals with disabilities. Based on the distinctive needs of individuals with significant disabilities, this Article takes the counterintuitive position that these individuals should be afforded access to government-coordinated habilitative care through Medicaid without regard to income or wealth. Under current market conditions, non-means-tested access to habilitative care is a normatively superior solution because it preserves the autonomy and dignity of individuals with disabilities and may be simultaneously cost-neutral and utility-increasing. Granting unrestricted access to government-coordinated habilitative care to individuals with significant disabilities would eliminate perverse employment and financial planning incentives created by Congress's past attempts to broaden access. Finally, it would create parity among parents who plan for the future of children with disabilities and those whose children are typically-abled, as well as parity for retirement savings among workers with significant disabilities and those without. As a result, Congress should revisit and revise means-tested access to disability-related services through Medicaid