Control-Tutored Reinforcement Learning: Towards the Integration of Data-Driven and Model-Based Control

We present an architecture where a feedback controller derived on an approximate model of the environment assists the learning process to enhance its data efficiency. This architecture, which we term as Control-Tutored Q-learning (CTQL), is presented in two alternative flavours. The former is based on defining the reward function so that a Boolean condition can be used to determine when the control tutor policy is adopted, while the latter, termed as probabilistic CTQL (pCTQL), is instead based on executing calls to the tutor with a certain probability during learning. Both approaches are validated, and thoroughly benchmarked against Q-Learning, by considering the stabilization of an inverted pendulum as defined in OpenAI Gym as a representative problem

A Multi-One Instruction Set Computer for Microcontroller Applications

This work presents a simple integer-only instruction set architecture and microarchitecture derived from One Instruction Set Computers (OISCs) and embedding multiple execution modes ( ${m}$ OISC), capable of running at a reasonable performance level to enable basic usability in microcontroller applications. The purpose of ${m}$ OISC is to enable simple data transfer tasks and run small programs while maintaining ultimate simplicity. We present the internal organization for a computer architecture including an 8bit I/O register, and 64kB central Random Access Memory (RAM), organized in two-bytes words. The processor can run code generated assuming an OISC or a Complex Instruction Set Computer (CISC) scheme (op-code based), depending on the programmer’s demands and based on the initial setting of a register during start-up. To enable practical applications and demonstrate successful exploitation of ${m}$ OISC in view of integration in a compiler back-end, we designed a custom Proof-of-Concept (PoC) software design toolchain based on LLVM and clang. Although not targeting all the features of commercial ISA, the toolchain is capable of compiling C code from LLVM intermediate representation or generating ${m}$ OISC code translated from ARM, x86, RISC-V, and MIPS assembly. The toolchain also enables practical Value Change Dump (VCD) simulations output with graphical plots of the CPU state and associated symbols. A PoC microcontroller system has been synthesized in a low power Field Programmable Gate Array (FPGA) and verified in a basic wireless telemetry application including a Synchronous Peripheral Interface (SPI) RFM9x Long RAnge (LoRA) transceiver and a MAX30205 Inter Integrated Circuit (I2C) temperature sensor, using its 8bit I/O port, with software bus interface implementation. ${m}$ OISC occupies ~6% of resources on a Cyclone 10LP FPGA, for 1397 Adaptive Look-Up Tables (ALUTs) and 461 dedicated logic registers. The measured dynamic current consumption of the complete FPGA board with synthesized ${m}$ OISC is 12mA at 100MHz clock

CT-DQN: Control-Tutored Deep Reinforcement Learning

One of the major challenges in Deep Reinforcement Learning for control is the need for extensive training to learn the policy. Motivated by this, we present the design of the Control-Tutored Deep Q-Networks (CT-DQN) algorithm, a Deep Reinforcement Learning algorithm that leverages a control tutor, i.e., an exogenous control law, to reduce learning time. The tutor can be designed using an approximate model of the system, without any assumption about the knowledge of the system’s dynamics. There is no expectation that it will be able to achieve the control objective if used stand-alone. During learning, the tutor occasionally suggests an action, thus partially guiding exploration. We validate our approach on three scenarios from OpenAI Gym: the inverted pendulum, lunar lander, and car racing. We demonstrate that CT-DQN is able to achieve better or equivalent data efficiency with respect to the classic function approximation solutions

Learning Queuing Networks by Recurrent Neural Networks

It is well known that building analytical performance models in practice is difficult because it requires a considerable degree of proficiency in the underlying mathematics. In this paper, we propose a machine-learning approach to derive performance models from data. We focus on queuing networks, and crucially exploit a deterministic approximation of their average dynamics in terms of a compact system of ordinary differential equations. We encode these equations into a recurrent neural network whose weights can be directly related to model parameters. This allows for an interpretable structure of the neural network, which can be trained from system measurements to yield a white-box parameterized model that can be used for prediction purposes such as what-if analyses and capacity planning. Using synthetic models as well as a real case study of a load-balancing system, we show the effectiveness of our technique in yielding models with high predictive power

HnRNPK maintains single strand RNA through controlling double-strand RNA in mammalian cells

Although antisense transcription is a widespread event in the mammalian genome, double-stranded RNA (dsRNA) formation between sense and antisense transcripts is very rare and mechanisms that control dsRNA remain unknown. By characterizing the FGF-2 regulated transcriptome in normal and cancer cells, we identified sense and antisense transcripts IER3 and IER3-AS1 that play a critical role in FGF-2 controlled oncogenic pathways. We show that IER3 and IER3-AS1 regulate each other\u27s transcription through HnRNPK-mediated post-transcriptional regulation. HnRNPK controls the mRNA stability and colocalization of IER3 and IER3-AS1. HnRNPK interaction with IER3 and IER3-AS1 determines their oncogenic functions by maintaining them in a single-stranded form. hnRNPK depletion neutralizes their oncogenic functions through promoting dsRNA formation and cytoplasmic accumulation. Intriguingly, hnRNPK loss-of-function and gain-of-function experiments reveal its role in maintaining global single- and double-stranded RNA. Thus, our data unveil the critical role of HnRNPK in maintaining single-stranded RNAs and their physiological functions by blocking RNA-RNA interactions

FITFES: A Wearable Myoelectrically Controlled Functional Electrical Stimulator Designed Using a User-Centered Approach

Myoelectrically Controlled Functional Electrical Stimulation (MeCFES) has proven to be a useful tool in the rehabilitation of the hemiplegic arm. This paper reports the steps involved in the development of a wearable MeCFES device (FITFES) through a user-centered design. We defined the minimal viable features and functionalities requirements for the device design from a questionnaire-based survey among physiotherapists with experience in functional electrical stimulation. The result was a necklace layout that poses minimal hindrance to task-oriented movement therapy, the context in which it is aimed to be used. FITFES is battery-powered and embeds a standard low power Bluetooth module, enabling wireless control by using PC/Mobile devices vendor specific built-in libraries. It is designed to deliver a biphasic, charge-balanced stimulation current pulses of up to 113 mA with a maximum differential voltage of 300 V. The power consumption for typical clinical usage is 320 mW at 20mA stimulation current and of less than $10~\mu \text{W}$ in sleep mode, thus ensuring an estimated full day of FITFES therapy on a battery charge. We conclude that a multidisciplinary user-centered approach can be successfully applied to the design of a clinically and ergonomically viable prototype of a wearable myoelectrically controlled functional electrical stimulator to be used in rehabilitation

Quali strumenti giuridici statali e regionali per le comunit\ue0 patrimoniali?

The Framework Convention on the Value of Cultural Heritage for Society (Faro Convention, 2005) recognizes a central role to heritage communities in the process of identification, study, interpretation, protection, conservation and presentation of the cultural heritage. As a signatory State of the Convention (signed on 27th February 2013, still waiting for ratification), Italy has in any case to ensure its contribution to the safeguarding of the tangible and intangible cultural heritage by adequate policies. Currently, a State law providing a general regulation of the participation of civil society to the protection and the enhancement of cultural heritage in the Italian legal system has not been adopted yet. Nevertheless, communities, groups and individuals have a wide range of instruments available, which can be drawn by an accurate interpretation of the Constitution and of many State and regional laws. In the long run, the persistent lack of common rules on this subject may be a source of uncertainty, capable of weakening, instead of strengthening, the role of heritage communities, in contrast with the principles of the Faro Convention

Citizens of Europe

Il titolo della collana esprime la volont\ue0 di approfondire i profili legati al processo di integrazione europeo, non ignorandone i risvolti pi\uf9 discutibili e burocratici ma sapendo guardare al di l\ue0 di essi, nella logica che traspare dal gioco di assonanze indicato dal titolo. In questo terzo volume Citizens of Europe, dedicato ai temi delle identit\ue0 e della cittadinanza culturale, viene in rilievo la tensione tra i limiti delle politiche, culturali e di cittadinanza, perseguite dalla UE e l\u2019imporsi progressivo \u2013 malgrado le cupe ombre proiettate dalla drammatica attualit\ue0 \u2013 di una pi\uf9 ampia nozione di \u2018cittadinanza d\u2019Europa\u2019, scandita in particolare da quei recenti strumenti giuridici del Consiglio d\u2019Europa attraverso i quali l\u2019afflato europeo, non imprigionato nelle pastoie dei meccanismi della EU citizenship, si sviluppa pi\uf9 significativamente