Numeric Reward Machines

Reward machines inform reinforcement learning agents about the reward structure of the environment and often drastically speed up the learning process. However, reward machines only accept Boolean features such as robot-reached-gold. Consequently, many inherently numeric tasks cannot profit from the guidance offered by reward machines. To address this gap, we aim to extend reward machines with numeric features such as distance-to-gold. For this, we present two types of reward machines: numeric-Boolean and numeric. In a numeric-Boolean reward machine, distance-to-gold is emulated by two Boolean features distance-to-gold-decreased and robot-reached-gold. In a numeric reward machine, distance-to-gold is used directly alongside the Boolean feature robot-reached-gold. We compare our new approaches to a baseline reward machine in the Craft domain, where the numeric feature is the agent-to-target distance. We use cross-product Q-learning, Q-learning with counter-factual experiences, and the options framework for learning. Our experimental results show that our new approaches significantly outperform the baseline approach. Extending reward machines with numeric features opens up new possibilities of using reward machines in inherently numeric tasks.Comment: ICAPS 2024; Workshop on Bridging the Gap Between AI Planning and Reinforcement Learnin

Connecting ProCom and REMES

Abstrac

A Survey on the Integration of Generative AI for Critical Thinking in Mobile Networks

In the near future, mobile networks are expected to broaden their services and coverage to accommodate a larger user base and diverse user needs. Thus, they will increasingly rely on artificial intelligence (AI) to manage network operation and control costs, undertaking complex decision-making roles. This shift will necessitate the application of techniques that incorporate critical thinking abilities, including reasoning and planning. Symbolic AI techniques already facilitate critical thinking based on existing knowledge. Yet, their use in telecommunications is hindered by the high cost of mostly manual curation of this knowledge and high computational complexity of reasoning tasks. At the same time, there is a spurt of innovations in industries such as telecommunications due to Generative AI (GenAI) technologies, operating independently of human-curated knowledge. However, their capacity for critical thinking remains uncertain. This paper aims to address this gap by examining the current status of GenAI algorithms with critical thinking capabilities and investigating their potential applications in telecom networks. Specifically, the aim of this study is to offer an introduction to the potential utilization of GenAI for critical thinking techniques in mobile networks, while also establishing a foundation for future research.Comment: 14 pages, 3 figures, 4 table

A Resource-Aware Component Model for Embedded Systems

Embedded systems are microprocessor-based systems that cover a large range of computer systems from ultra small computer-based devices to large systems monitoring and controlling complex processes. The particular constraints that must be met by embedded systems, such as timeliness, resource-use efficiency, short time-to-market and low cost, coupled with the increasing complexity of embedded system software, demand technologies and processes that will tackle these issues. An attractive approach to manage the software complexity, increase productivity, reduce time to market and decrease development costs, lies in the adoption of the component based software engineering (CBSE) paradigm. The specific characteristics of embedded systems lead to important design issues that need to be addressed by a component model. Consequently, a component model for development of embedded systems needs to systematically address extra-functional system properties. The component model should support predictable system development and as such guarantee absence or presence of certain properties. Formal methods can be a suitable solution to guarantee the correctness and reliability of software systems.   Following the CBSE spirit, in this thesis we introduce the ProCom component model for development of distributed embedded systems. ProCom is structured in two layers, in order to support both a high-level view of loosely coupled subsystems encapsulating complex functionality, and a low-level view of control loops with restricted functionality. These layers differ from each other in terms of execution model, communication style, synchronization etc., but also in kind of analysis which are suitable. To describe the internal behavior of a component, in a structured way, in this thesis we propose REsource Model for Embedded Systems (REMES) that describes both functional and extra-functional behavior of interacting embedded components. We also formalize the resource-wise properties of interest and show how to analyze such behavioral models against them.PROGRES

A Resource-Aware Framework for Designing Predictable Component-Based Embedded Systems

Managing complexity is an increasing challenge in the development of embedded systems (ES). Some of the factors contributing to the increase in complexity are the growing complexity of hardware and software, and the increased pressure to deliver full-featured products with reduced time-to-market. An attractive approach to manage the software complexity, reduce time-to-market and decrease development costs lies in the adoption of component-based development that has been proven as a successful approach in other domains. Another raising challenge, due to complexity increase, in ES, is predictability, i.e., the ability to anticipate the behavior of a system at run-time. The particular predictability requirements of ES call for a development framework equipped with techniques and tools that can be applied to deal with requirements, such as timing, and resource utilization, already at early-stage of development. Modeling and formal analysis play increasingly important roles in achieving predictability, since they can help us to understand how systems function, validate the design and verify some important properties. In this thesis, we present a resource-aware framework for designing predictable component-based ES. The proposed framework consists of (i) the formally specified ProCom component model that takes into account the characteristics of control-intensive ES, and (ii) the resource-aware timed behavioral language - REMES for modeling and reasoning about components’ and systems’ functional and extra-functional behavior that includes relevant resource types for ES, associated analysis techniques for various resource-wise properties, and a set of associated tools. To demonstrate the potential application of our framework, we present a number of case studies, out of which one is an industrial research prototype, where ProCom and REMES are applied.PROGRES

Applying REMES behavioral modeling to PLC systems

Abstract—Programmable logic controllers (PLCs), as aspecialized type of embedded systems, have been introduced toincrease system flexibility and reliability, but at the same time togive faster response time and lower cost of implementation. Inthe beginning, their use brought a revolutionary change, but withthe constant growth of system complexity, it became harder toguarantee both functional and extra functional properties, asearly as possible in the development process. In this paper, weshow how formal methods can be applied to describe PLC-basedsystems and illustrate it on an example of a car wash system.First, we show how the existing behavioral modeling languageREMES (REsource Model for Embedded Systems) can beextended to model the behavior of such systems. Second, we showhow REMES can be translated into networks of timed automataand priced timed automata in order to support safety andresource-wise reasoning about PLC systems. The formalverification of PLC systems is carried out in the UPPAAL andUPPAAL CORA tools.ProgressQ-Impres