Certified Reinforcement Learning with Logic Guidance

This paper proposes the first model-free Reinforcement Learning (RL) framework to synthesise policies for unknown, and continuous-state Markov Decision Processes (MDPs), such that a given linear temporal property is satisfied. We convert the given property into a Limit Deterministic Buchi Automaton (LDBA), namely a finite-state machine expressing the property. Exploiting the structure of the LDBA, we shape a synchronous reward function on-the-fly, so that an RL algorithm can synthesise a policy resulting in traces that probabilistically satisfy the linear temporal property. This probability (certificate) is also calculated in parallel with policy learning when the state space of the MDP is finite: as such, the RL algorithm produces a policy that is certified with respect to the property. Under the assumption of finite state space, theoretical guarantees are provided on the convergence of the RL algorithm to an optimal policy, maximising the above probability. We also show that our method produces ''best available'' control policies when the logical property cannot be satisfied. In the general case of a continuous state space, we propose a neural network architecture for RL and we empirically show that the algorithm finds satisfying policies, if there exist such policies. The performance of the proposed framework is evaluated via a set of numerical examples and benchmarks, where we observe an improvement of one order of magnitude in the number of iterations required for the policy synthesis, compared to existing approaches whenever available.Comment: This article draws from arXiv:1801.08099, arXiv:1809.0782

A Review of System Development Systems

The requirements for a system development system are defined and used as guidelines to review six such systems: SAMM, SREM, SADT, ADS / SODA, PSL/PSA and Systematics. It is found that current system development systems emphasise only validation and user verification. They can perform relatively little on automatic file optimisation, process optimisation and maintenance.postprin

ConSus : a decision support tool for the procurement of sustainable social housing

Abstract unavailable please refer to PD

Subject agreement in control and modal constructions in Russian Sign Language:Implications for the hierarchy of person features

The present research combines three fields of inquiry in sign language linguistics: verbal agreement, person features, and syntactic complexity. These topics have previously been addressed in isolation, but little is known about their interaction. This study attempts to fill this gap by investigating subject agreement in complement clauses in Russian Sign Language. By means of corpus investigation and grammaticality judgments, I found that subject agreement in clausal complements of the control predicates try, love, want, begin, and modal can may be deficient – in particular, it can be reduced to the forms identical to first-person marking even in the case of a third-person subject controller. Deficient subject agreement in complement clauses is thus reminiscent of non-finite verbal forms in spoken languages. I further argue that the choice of first-person forms in deficient agreement reveals a default status of first person in sign languages, which is consistent with proposals regarding the modality-specific properties of first-person reference in these languages

Requirements modelling of real-time systems

Real-time systems are characterised by the critical nature of their missions, and the demanding environment with which they interact. Real-time systems are used for dedicated applications. Every application is the subject of special requirements enforced by the customer. Considering the vital role that these systems play, it is imperative that a systematic approach be adopted in modelling their unique requirements. In this thesis I propose such a treatment. Real-time systems are time critical. Temporal requirements are the timing restrictions imposed by the application environment. Previous studies in requirements modelling of real-time systems have focused on adding the notion of time to modelling techniques of traditional systems without regard to the realities of requirements modelling. The information should be presented in the way the user handles it, and not the way which is convenient to the software engineer. I attempt to understand the needs of the users better by modelling the real world as close to the user's perspective as possible, and propose the Real World Model (RWM). RWM is assumed to be developed by users, and requirements engineers. An engineering approach to building the model is provided. A real-time system has a well defined use to its community. A requirements model must rely on the user level activities, and aid the human understanding and communication. In the RWM, a real-time system is viewed as a set of concurrently acting automata, each representing a system entity. This model supports temporal reasoning in easily described ways, for all classes of timing properties. A generalised classification of timing constraints is provided. A requirements modelling language facilitates the description of requirements, and serves as a medium of communication among developers and stakeholders. Jarke et al [Jarke 94] observe that there is a need for a requirements language that manages the relationship between the meta-level domain scheme, and the scenarios that actually instantiate the scheme under development. Here I propose Timed Requirements Language (TRL) to bridge this gulf between the world of stakeholders, and the world of specifiers. TRL has natural looking expressions for formulating the needs. TRL has a number of novel features including the treatment of causality, and the description of static, and dynamic constraints all integrated into one uniform framework. TRL has been used with a number of systems. The generality of the language is validated through its application to specific systems

A Random Attention Model

This paper illustrates how one can deduce preference from observed choices when attention is not only limited but also random. In contrast to earlier approaches, we introduce a Random Attention Model (RAM) where we abstain from any particular attention formation, and instead consider a large class of nonparametric random attention rules. Our model imposes one intuitive condition, termed Monotonic Attention, which captures the idea that each consideration set competes for the decision-maker's attention. We then develop revealed preference theory within RAM and obtain precise testable implications for observable choice probabilities. Based on these theoretical findings, we propose econometric methods for identification, estimation, and inference of the decision maker's preferences. To illustrate the applicability of our results and their concrete empirical content in specific settings, we also develop revealed preference theory and accompanying econometric methods under additional nonparametric assumptions on the consideration set for binary choice problems. Finally, we provide general purpose software implementation of our estimation and inference results, and showcase their performance using simulations

Verification of Branching-Time and Alternating-Time Properties for Exogenous Coordination Models

Information and communication systems enter an increasing number of areas of daily lives. Our reliance and dependence on the functioning of such systems is rapidly growing together with the costs and the impact of system failures. At the same time the complexity of hardware and software systems extends to new limits as modern hardware architectures become more and more parallel, dynamic and heterogenous. These trends demand for a closer integration of formal methods and system engineering to show the correctness of complex systems within the design phase of large projects. The goal of this thesis is to introduce a formal holistic approach for modeling, analysis and synthesis of parallel systems that potentially addresses complex system behavior at any layer of the hardware/software stack. Due to the complexity of modern hardware and software systems, we aim to have a hierarchical modeling framework that allows to specify the behavior of a parallel system at various levels of abstraction and that facilitates designing complex systems in an iterative refinement procedure, in which more detailed behavior is added successively to the system description. In this context, the major challenge is to provide modeling formalisms that are expressive enough to address all of the above issues and are at the same time amenable to the application of formal methods for proving that the system behavior conforms to its specification. In particular, we are interested in specification formalisms that allow to apply formal verification techniques such that the underlying model checking problems are still decidable within reasonable time and space bounds. The presented work relies on an exogenous modeling approach that allows a clear separation of coordination and computation and provides an operational semantic model where formal methods such as model checking are well suited and applicable. The channel-based exogenous coordination language Reo is used as modeling formalism as it supports hierarchical modeling in an iterative top-down refinement procedure. It facilitates reusability, exchangeability, and heterogeneity of components and forms the basis to apply formal verification methods. At the same time Reo has a clear formal semantics based on automata, which serve as foundation to apply formal methods such as model checking. In this thesis new modeling languages are presented that allow specifying complex systems in terms of Reo and automata models which yield the basis for a holistic approach on modeling, verification and synthesis of parallel systems. The second main contribution of this thesis are tailored branching-time and alternating time temporal logics as well as corresponding model checking algorithms. The thesis includes results on the theoretical complexity of the underlying model checking problems as well as practical results. For the latter the presented approach has been implemented in the symbolic verification tool set Vereofy. The implementation within Vereofy and evaluation of the branching-time and alternating-time model checker is the third main contribution of this thesis

Formalisation and evaluation of focus theories for requirements elicitation dialogues in natural language

Requirements engineering is an important part of software engineering. It consists in defining the needs of users when building a new system. These needs may be functional, i.e., what service should the system be able to provide, as well as non-functional, i.e., under which constraints should the system operate. Errors in requirements may have disastrous effects in the rest of the software engineering process (Brooks 1995, p.199), since they would lead to the construction of a system of little interest to its users or would require expensive modifications to correct. Because requirements documents may be very large, errors are usually hard to detect manually. Computer support is therefore often beneficial for their analysis. This is made easier if requirements are expressed formally. However, this support must also be adapted to and be usable by people who are expressing their requirements. These people are usually not computer specialists and are not accustomed to use formal languages. It is therefore necessary to help them express their requirements. Numerous approaches, have been suggested as aids to the acquisition of requirements (Reubenstein 1990). Much less attention has been paid to the control of the dialogue taking place between the users and the system whilst using such frameworks (Bubenko et al. 1994). Frameworks for requirements acquisition are not normally accompanied by theories of the types of dialogue which they support. Our ability to develop sophisticated formal frameworks to analyse requirements makes this deficiency more acutely felt, since increases in formality are often accompanied by greater difficulty in understanding and using the frameworks (Robertson et al. 1989).Users write their requirements in more or less natural language. This is then translated into a formal language that can be interpreted by the elicitation module. This module works on the requirements and provide feedback. The translation process is then applied to convert feedback into more or less natural language. Different systems put different emphasis on the parts of that general architecture. Some are very good at natural language interpretation while others put more emphasis on analysing the requirements and providing feedback.Natural language approaches to requirements elicitation, put an emphasis on natural language interpretation (see section 1.2.1). In these approaches, users write their specifica¬ tion in a subset of natural language. The system then translates it into a formal notation. The main benefit provided by these approaches is the improvement in the ease of use of the system: natural language is the main means of communication for human beings and does not need to be learned. However, most of these approaches do not provide a dialogue well suited for the requirements elicitation process. Because they translate the natural lan¬ guage specification into a formal notation but do not provide guidance on how to write the specification in the first place, users are left in charge of writing correct requirements. If a mistake is made while writing the specification, it will simply be translated into the formal notation.In order to actively help users in the process of writing the requirements, the elicit¬ ation system must interact with them. The emphasis, here, is no longer on translating requirements, but on actively extracting them through a dialogue with users. This is useful, since the requirements elicitation process is complex, and offering guidance is a big help for users. Unfortunately, most of the approaches providing guidance expose their formal underlying frameworks directly to users (see section 1.2.2). In order to benefit from the guidance provided, users have to learn the idiosyncrasies of the system they use. The task of providing guidance is complicated by the fact that there are numerous ways of carrying out the requirements elicitation. Very little research has been done on how to organise best the elicitation process to provide effective guidance. An arbitrary choice could be made, but forcing users to adopt a predefined method is usually not possible as it would make the elicitation process very difficult to follow and understand. The system must therefore be able to adapt itself to various elicitation methods. On the other hand, it is necessary for the system to make choices in order to provide active guidance. A "least-commitment" strategy, such as asking users at every choice point what to do next, is not a useful approach (Ferguson et al. 1996).One way of offering guidance without restricting users too much is by communicating with them in natural language, and by using natural language constraints to inform the choices made by the system to select a guidance strategy. These constraints ensure that the system adopts a strategy that will guide users in a natural and understandable manner, by taking into account the current state of the dialogue. In other words, the system takes into account the current state of the specification to help users complete it, but the current state of the dialogue is the principal factor constraining what will be spoken about next. Using such an approach reduces some of the problems discussed above. The specification does not need to be immediately correct as it will be checked and reworked by the system. The formal framework is hidden from users but is still there to ensure the correctness of the specifications. Guidance is continuously offered through dialogue, which is influenced by but does not directly follow the steps of construction of the specification.The natural language constraints we use in this thesis are theories of dialogue coherence, called "focus" theories. They define what can be spoken about next in a dialogue based on what has already been discussed and the subject under discussion. The theories take into account what participants in a dialogue pay attention to and try to ensure that the rest of the dialogue is related to it. The systems tries to help its users define how a research group WWW site should look like. The way the dialogue evolves from discussing the research group, to discussing the site and its associated home page, to discussing the set of publication can quite easily be followed. The use of pronouns helps in making the text fell natural. It would have been difficult to achieve the same result without using focus rules.Other techniques for organising dialogues, such as those based on the intentions under¬ lying the dialogue (Cohen et al. 1990), would require the dialogue manager to know what the elicitation system is trying to achieve and what its plan is. For some elicitation systems, this knowledge may not be available. Similarly, techniques based on the content of the communications exchanged and how they relate, e.g., based on RST (Mann and Thompson 1987), usually require a lot of domain knowledge. They are therefore time-consumming to code. Focus theories require less information from the elicitation module while enabling the dialogue manager to structure the dialogue. However, in some cases, focus theories are not sufficient to organise a dialogue. We use a theory based on speech act (see section 3.4.1) and some ideas from Grice's work on conversation (see section 5.2.1) to deal with these cases. More generally, although we tried to minimise the impact of other theories to study in detail focus theories, it would be interesting to know whether and how we can integrate them with the work presented in this thesis. In particular, the notion of dialog act and its application to dialog grammar could be of interest. General frameworks developped to study various aspects of dialogue, including dialog acts and focus, have started to appear but work is still at an early stage (C-Star Consortium 1998; Allen and Core 1997).Organising a dialogue based on attention requires a lot of domain knowledge in order to know how things mentioned in the dialogue relate to each other. Therefore, the amount of knowledge engineering needed to build natural language applications is also an important issue. We have tried to limit the engineering difficulties by clearly separating the domain knowledge needed by our dialogue manager from its management capabilities, and by provid¬ ing a way of re-using the existing domain knowledge as far as possible. This is done by using rules which enable us to re-use part of the domain knowledge already used by the elicitation module.The contribution of this thesis is therefore the formalisation and evaluation of focus theories for requirements elicitation dialogues in natural language. The main questions we deal with are the following: • Which focus theories should we use? • What are the relations between the constraints imposed by the focus theories and the constraints inherent to the requirements elicitation process? • Does this approach improve the perceived quality of the dialogue between the elicita¬ tion tool and its users?A prototype system has been developed. This system mainly operates in the WWW site design domain. It has also been applied in other domains as an initial demonstration of the range of problems that can be tackled by our approach