Chance-Constrained Control with Lexicographic Deep Reinforcement Learning

This paper proposes a lexicographic Deep Reinforcement Learning (DeepRL)-based approach to chance-constrained Markov Decision Processes, in which the controller seeks to ensure that the probability of satisfying the constraint is above a given threshold. Standard DeepRL approaches require i) the constraints to be included as additional weighted terms in the cost function, in a multi-objective fashion, and ii) the tuning of the introduced weights during the training phase of the Deep Neural Network (DNN) according to the probability thresholds. The proposed approach, instead, requires to separately train one constraint-free DNN and one DNN associated to each constraint and then, at each time-step, to select which DNN to use depending on the system observed state. The presented solution does not require any hyper-parameter tuning besides the standard DNN ones, even if the probability thresholds changes. A lexicographic version of the well-known DeepRL algorithm DQN is also proposed and validated via simulations

Bellman's principle of optimality and deep reinforcement learning for time-varying tasks

This paper presents the first framework (up to the authors' knowledge) to address time-varying objectives in finite-horizon Deep Reinforcement Learning (DeepRL), based on a switching control solution developed on the ground of Bellman's principle of optimality. By augmenting the state space of the system with information on its visit time, the DeepRL agent is able to solve problems in which its task dynamically changes within the same episode. To address the scalability problems caused by the state space augmentation, we propose a procedure to partition the episode length to define separate sub-problems that are then solved by specialised DeepRL agents. Contrary to standard solutions, with the proposed approach the DeepRL agents correctly estimate the value function at each time-step and are hence able to solve time-varying tasks. Numerical simulations validate the approach in a classic RL environment

Controlled optimal black start procedures in smart grids for service restoration in presence of electrical storage systems

This paper presents an optimisation problem to determine the optimal reclosure order of remotely operable switches deployed in a smart grid consisting in a distribution network equipped with one or more Energy Storage Systems (ESS). The proposed solution integrates nonlinear real and reactive power flow equations, by reconducting them to a set of conic constraints, together with several network operator requirements, such as network radiality and ampacity limits. A numerical simulation validates the approach and concludes the work

Attack-Surface Metrics, OSSTMM and Common Criteria Based Approach to “Composable Security” in Complex Systems

In recent studies on Complex Systems and Systems-of-Systems theory, a huge effort has been put to cope with behavioral problems, i.e. the possibility of controlling a desired overall or end-to-end behavior by acting on the individual elements that constitute the system itself. This problem is particularly important in the “SMART” environments, where the huge number of devices, their significant computational capabilities as well as their tight interconnection produce a complex architecture for which it is difficult to predict (and control) a desired behavior; furthermore, if the scenario is allowed to dynamically evolve through the modification of both topology and subsystems composition, then the control problem becomes a real challenge. In this perspective, the purpose of this paper is to cope with a specific class of control problems in complex systems, the “composability of security functionalities”, recently introduced by the European Funded research through the pSHIELD and nSHIELD projects (ARTEMIS-JU programme). In a nutshell, the objective of this research is to define a control framework that, given a target security level for a specific application scenario, is able to i) discover the system elements, ii) quantify the security level of each element as well as its contribution to the security of the overall system, and iii) compute the control action to be applied on such elements to reach the security target. The main innovations proposed by the authors are: i) the definition of a comprehensive methodology to quantify the security of a generic system independently from the technology and the environment and ii) the integration of the derived metrics into a closed-loop scheme that allows real-time control of the system. The solution described in this work moves from the proof-of-concepts performed in the early phase of the pSHIELD research and enrich es it through an innovative metric with a sound foundation, able to potentially cope with any kind of pplication scenarios (railways, automotive, manufacturing, ...)

Joint Model Predictive Control of Electric and Heating Resources in a Smart Building

The new challenge in power systems design and operation is to organize and control smart micro grids supplying aggregation of users and special loads as electric vehicles charging stations. The presence of renewable and storage can help the optimal operation only if a good control manages all the elements of the grid. New models of green buildings and energy communities are proposed. For a real application they need an appropriate and advanced power system equipped with a building automation control system. This article presents an economic model predictive control approach to the problem of managing the electric and heating resources in a smart building in a coordinated way, for the purpose of achieving in real time nearly zero energy consumption and automated participation to demand response programs. The proposed control, leveraging a mixed integer quadratic programming problem, allows to meet manifold thermal and electric users' requirements and react to inbound demand response signals, while still guaranteeing stable operation of the building's electric and thermal storage equipment. The simulation results, performed for a real case study in Italy, highlight the peculiarities of the proposed approach in the joint handling of electric and thermal building flexibility

Magnetic carbon nanotubes: a new tool for shepherding mesenchymal stem cells by magnetic fields

We investigated the interaction between magnetic carbon nanotubes (CNTs) and mesenchymal stem cells (MSCs), and their ability to guide these intravenously injected cells in living rats by using an external magnetic field. MATERIALS & METHODS: Multiwalled CNTs were used to treat MSCs derived from rat bone marrow. Cytotoxicity induced by nanotubes was studied using the WST-1 proliferation and Hoechest 33258 apoptosis assays. The effects of nanotubes on MSCs were evaluated by monitoring the effects on cellular growth rates, immunophenotyping and differentiation, and on the arrangement of cytoskeletal actin. MSCs loaded with nanotubes were injected in vivo in the portal vein of rats driving their localization in the liver by magnetic field. An histological analysis was performed on the liver, lungs and kidneys of all animals. RESULTS: CNTs did not affect cell viability and their ability to differentiate in osteocytes and adipocytes. Both the CNTs and the magnetic field did not alter the cell growth rate, phenotype and cytoskeletal conformation. CNTs, when exposed to magnetic fields, are able to shepherd MSCs towards the magnetic source in vitro. Moreover, the application of a magnetic field alters the biodistribution of CNT-labelled MSCs after intravenous injection into rats, increasing the accumulation of cells into the target organ (liver). CONCLUSION: Multiwalled CNTs hold the potential for use as nanodevices to improve therapeutic protocols for transplantation and homing of stem cells in vivo. This could pave the way for the development of new strategies for the manipulation/guidance of MSCs in regenerative medicine and cell transplantation

Investigation of interactions between poly-l-lysine-coated boron nitride nanotubes and C2C12 cells: up-take, cytocompatibility, and differentiation

Boron nitride nanotubes (BNNTs) have generated considerable interest within the scientific community by virtue of their unique physical properties, which can be exploited in the biomedical field. In the present in vitro study, we investigated the interactions of poly-l-lysine-coated BNNTs with C2C12 cells, as a model of muscle cells, in terms of cytocompatibility and BNNT internalization. The latter was performed using both confocal and transmission electron microscopy. Finally, we investigated myoblast differentiation in the presence of BNNTs, evaluating the protein synthesis of differentiating cells, myotube formation, and expression of some constitutive myoblastic markers, such as MyoD and Cx43, by reverse transcription – polymerase chain reaction and Western blot analysis. We demonstrated that BNNTs are highly internalized by C2C12 cells, with neither adversely affecting C2C12 myoblast viability nor significantly interfering with myotube formation

Value of multidetector computed tomography image segmentation for preoperative planning in general surgery

Using practical examples, this report aims to highlight the clinical value of patient-specific three-dimensional (3D) models, obtained segmenting multidetector computed tomography (MDCT) images, for preoperative planning in general surgery.In this study, segmentation and 3D model generation were performed using a semiautomatic tool developed in the authors' laboratory. Their segmentation procedure is based on the neighborhood connected region-growing algorithm that, appropriately parameterized for the anatomy of interest and combined with the optimal segmentation sequence, generates good-quality 3D images coupled with facility of use. Using a touch screen monitor, manual refining can be added to segment structures unsuitable for automatic reconstruction. Three-dimensional models of 10 candidates for major general surgery procedures were presented to the operating surgeons for evaluation. A questionnaire then was administered after surgery to assess the perceived added value of the new technology.The questionnaire results were very positive. The authors recorded the diffuse opinion that planning the procedure using a segmented data set allows the surgeon to plan critical interventions with better awareness of the specific patient anatomy and consequently facilitates choosing the best surgical approach.The benefit shown in this report supports a wider use of segmentation software in clinical practice, even taking into account the extra time and effort required to learn and use these systems

Control architecture to provide E2E security in interconnected systems: the (new) SHIELD approach

Modern Systems are usually obtained as incremental composition of proper (smaller and SMART) subsystems interacting through communication interfaces. Such flexible architecture allows the pervasive provisioning of a wide class of services, ranging from multimedia contents delivery, through monitoring data collection, to command and control functionalities. All these services requires that the adequate level of robustness and security is assured at End-to- End (E2E) level, according to user requirements that may vary depending on the specific context or the involved technologies. A flexible methodology to dynamically control the security level of the service being offered is then needed. In this perspective, the authors propose an innovative control architecture able to assure E2E security potentially in any application, by dynamically adapting to the underlying systems and using its resources to “build the security”. In particular, the main novelties of this solution are: i) the possibility of dynamically discovering and composing the available functionalities offered by the environment to satisfy the security needs and ii) the possibility of modelling and measuring the security through innovative technology-independent metrics. The results presented in this paper moves from the solutions identified in the pSHIELD project and enrich them with the innovative advances achieved through the nSHIELD research, still ongoing. Both projects have been funded by ARTEMIS-JU