Online Safety Property Collection and Refinement for Safe Deep Reinforcement Learning in Mapless Navigation

Safety is essential for deploying Deep Reinforcement Learning (DRL) algorithms in real-world scenarios. Recently, verification approaches have been proposed to allow quantifying the number of violations of a DRL policy over input-output relationships, called properties. However, such properties are hard-coded and require task-level knowledge, making their application intractable in challenging safety-critical tasks. To this end, we introduce the Collection and Refinement of Online Properties (CROP) framework to design properties at training time. CROP employs a cost signal to identify unsafe interactions and use them to shape safety properties. Hence, we propose a refinement strategy to combine properties that model similar unsafe interactions. Our evaluation compares the benefits of computing the number of violations using standard hard-coded properties and the ones generated with CROP. We evaluate our approach in several robotic mapless navigation tasks and demonstrate that the violation metric computed with CROP allows higher returns and lower violations over previous Safe DRL approaches.Comment: Accepted at the 2023 IEEE International Conference on Robotics and Automation (ICRA). Marzari and Marchesini contributed equall

The #DNN-Verification Problem: Counting Unsafe Inputs for Deep Neural Networks

Deep Neural Networks are increasingly adopted in critical tasks that require a high level of safety, e.g., autonomous driving. While state-of-the-art verifiers can be employed to check whether a DNN is unsafe w.r.t. some given property (i.e., whether there is at least one unsafe input configuration), their yes/no output is not informative enough for other purposes, such as shielding, model selection, or training improvements. In this paper, we introduce the #DNN-Verification problem, which involves counting the number of input configurations of a DNN that result in a violation of a particular safety property. We analyze the complexity of this problem and propose a novel approach that returns the exact count of violations. Due to the #P-completeness of the problem, we also propose a randomized, approximate method that provides a provable probabilistic bound of the correct count while significantly reducing computational requirements. We present experimental results on a set of safety-critical benchmarks that demonstrate the effectiveness of our approximate method and evaluate the tightness of the bound.Comment: Accepted in the International Joint Conference on Artificial Intelligence (IJCAI), 2023. [Marzari and Corsi contributed equally

Towards Hierarchical Task Decomposition using Deep Reinforcement Learning for Pick and Place Subtasks

Deep Reinforcement Learning (DRL) is emerging as a promising approach to generate adaptive behaviors for robotic platforms. However, a major drawback of using DRL is the data-hungry training regime that requires millions of trial and error attempts, which is impractical when running experiments on robotic systems. Learning from Demonstrations (LfD) has been introduced to solve this issue by cloning the behavior of expert demonstrations. However, LfD requires a large number of demonstrations that are difficult to be acquired since dedicated complex setups are required. To overcome these limitations, we propose a multi-subtask reinforcement learning methodology where complex pick and place tasks can be decomposed into low-level subtasks. These subtasks are parametrized as expert networks and learned via DRL methods. Trained subtasks are then combined by a high-level choreographer to accomplish the intended pick and place task considering different initial configurations. As a testbed, we use a pick and place robotic simulator to demonstrate our methodology and show that our method outperforms a benchmark methodology based on LfD in terms of sample-efficiency. We transfer the learned policy to the real robotic system and demonstrate robust grasping using various geometric-shaped objects.Comment: This work has been accepted to the IEEE International Conference on Advanced Robotics (ICAR) 202

Constrained Reinforcement Learning and Formal Verification for Safe Colonoscopy Navigation

The field of robotic Flexible Endoscopes (FEs) has progressed significantly, offering a promising solution to reduce patient discomfort. However, the limited autonomy of most robotic FEs results in non-intuitive and challenging manoeuvres, constraining their application in clinical settings. While previous studies have employed lumen tracking for autonomous navigation, they fail to adapt to the presence of obstructions and sharp turns when the endoscope faces the colon wall. In this work, we propose a Deep Reinforcement Learning (DRL)-based navigation strategy that eliminates the need for lumen tracking. However, the use of DRL methods poses safety risks as they do not account for potential hazards associated with the actions taken. To ensure safety, we exploit a Constrained Reinforcement Learning (CRL) method to restrict the policy in a predefined safety regime. Moreover, we present a model selection strategy that utilises Formal Verification (FV) to choose a policy that is entirely safe before deployment. We validate our approach in a virtual colonoscopy environment and report that out of the 300 trained policies, we could identify three policies that are entirely safe. Our work demonstrates that CRL, combined with model selection through FV, can improve the robustness and safety of robotic behaviour in surgical applications.Comment: Accepted in the IEEE International Conference on Intelligent Robots and Systems (IROS), 2023. [Corsi, Marzari and Pore contributed equally

Fe-chitosan complexes for oxidative degradation of emerging contaminants in water: Structure, activity, and reaction mechanism

Versatile and ecofriendly methods to perform oxidations at near-neutral pH are of crucial importance for processes aimed at purifying water. Chitosan, a deacetylated form of chitin, is a promising starting material owing to its biocompatibility and ability to form stable films and complexes with metals. Here, we report a novel chitosan-based organometallic complex that was tested both as homogeneous and heterogeneous catalyst in the degradation of contaminants of emerging concern in water. The stoichiometry of the complex was experimentally verified with different metals, namely, Cu(II), Fe(III), Fe(II), Co(II), Pd(II), and Mn(II), and we identified the chitosan-Fe(III) complex as the most efficient catalyst. This complex effectively degraded phenol, triclosan, and 3-chlorophenol in the presence of hydrogen peroxide. A putative ferryl-mediated reaction mechanism is proposed based on experimental data, density functional theory calculations, and kinetic modeling. Finally, a film of the chitosan-Fe(III) complex was synthesized and proven a promising supported heterogeneous catalyst for water purification

Long-Term Survival and Predictors of Failure of Opening Wedge High Tibial Osteotomy

Objective: High tibial valgus osteotomy (HTO) is a widely accepted procedure indicated for varus knee with symptomatic osteoarthritis of the medial compartment. However, there is a lack of studies evaluating long term results of this procedure. The primary aim of this study was to evaluate the long-term survival of opening wedge high tibial osteotomy (HTO) for isolated osteoarthritis in the medial compartment of the knee. The secondary objective was to identify independent predictors of conversion to total knee arthroplasty (TKA). Methods: This is a long term retrospective study of 296 cases of open wedge HTOs performed at a single center (level of evidence IV) between January 2005 and August 2015. Opening wedge medial HTO was always performed after diagnostic arthroscopy. Eighty-three percent of the population (233 patients, 247 procedures) was followed up at a mean 11.6 years (6-17) by telephone interview, to evaluate the possible conversion to TKA. Mean age at the index operation was 42.8 years (range 15-70) and most patients were male (70%). Associated procedures (e.g., platelet rich plasma supplementation, microfractures, meniscectomy, etc.) were carried out at the time of the HTO in 80 (32%) cases. Survival of HTO and its association with age, sex, body mass index, smoking habit, preoperative severity of varus deformity, cartilage status at surgery, and associated procedures were evaluated. Kaplan-Meier and Cox regression analyses were performed. Results: Thirty-three of the 247 HTOs (13.4%) were converted to knee replacement, with 86.6% of the original procedures surviving at a mean 12-year follow-up. Kaplan-Meier survival estimates at 17 years for HTO were 75.5% (95% confidence interval [CI] 66.7-84.3). There was significant difference (P < 0.001) in the 17-year survival rate between obese (55.5%; 95% CI 35.3-75.6) and non-obese (79.7%; 95% CI 70.1-89.2) patients. The determinants of conversion to knee arthroplasty detected at multivariate Cox regression analysis were body mass index, severity of cartilage degeneration in the medial compartment (Outerbridge grade), and age. Conclusion: The long-term survival of open wedge HTO for osteoarthritis in the medial compartment of the knee is satisfactory. The risk of conversion to TKA is significantly increased in obese patients. Advanced age and severity of pre-existing cartilage damage may also contribute to the risk of conversion to TKA

Safe and Efficient Reinforcement Learning for Environmental Monitoring

This paper discusses the challenges of applying reinforcement techniques to real-world environmental monitoring problems and proposes innovative solutions to overcome them. In particular, we focus on safety, a fundamental problem in RL that arises when it is applied to domains involving humans or hazardous uncertain situations. We propose to use deep neural networks, formal verification, and online refinement of domain knowledge to improve the transparency and efficiency of the learning process, as well as the quality of the final policies. We present two case studies, specifically (i) autonomous water monitoring and (ii) smart control of air quality indoors. In particular, we discuss the challenges and solutions to these problems, addressing crucial issues such as anomaly detection and prevention, real-time control, and online learning. We believe that the proposed techniques can be used to overcome some limitations of RL, providing safe and efficient solutions to complex and urgent problems

Heterogeneous Feature State Estimation with Rao-Blackwellized Particle Filters

In this paper we present a novel technique to estimate the state of heterogeneous features from inaccurate sensors. The proposed approach exploits the reliability of the feature extraction process in the sensor model and uses a RaoBlackwellized particle filter to address the data association problem. Experimental results show that the use of reliability improves performance by allowing the approach to perform better data association among detected features. Moreover, the method has been tested on a real robot during an exploration task in a non-planar environment. This last experiment shows an improvement in correctly detecting and classifying interesting features for navigation purpose. © 2007 IEEE

Evaluation of GeneXpert® system for detection of methicillin-resistant Staphyloccocus aureus in clinical samples

Infections caused by methicillin-resistant Staphyloccocus aureus strains (MRSA) have reached epidemic proportions globally, being the major cause of nosocomial infections. Rapid identification of MRSA in nasal swabs or in clinical samples is considered a useful strategy for control and treatment of these infections. GeneXpert system (Cepheid Europe,Vira-Solelch, Maurence-Scopont-France) can detect by real-time PCR in approximately one hour methicillin-resistant S. aureus or coagulase-negative staphylococci (CoNS) in clinical samples, in comparison with 24 hours for the culture or 48 hours for the antimicrobial susceptibility testing. In this study GeneXpert system was compared with traditional tests for MRSA detection in nasal swabs, bloodcultures and surgical wound swabs. Materials and methods. Eighteen nasal swabs, 23 blood-cultures and 13 surgical wound swabs were tested. The samples were cultured on blood-agar and mannitol-salt agar. Identification of isolates was carried out with traditional tests (Gram staining, catalase, coagulase) and automatic Phoenix system. Methicillin-susceptibility was evaluated according to 2010 CLSI guidelines. GeneXpert system was performed according to manufacturers instructions, by using the specific kits and methicillin-resistance was detected by amplification of the genic sequences spa, SCC e mecA. Results. The results showed a 100% accordance between GeneXpert system and traditional tests for detection of methicillin-resistant staphylococci. In particular, among 18 nasal swabs, no MRSA was detected, while 1 bloodculture (4.3%) and 4 surgical wound swabs (30.7%) were positive for MRSA. Conclusions. GeneXpert system allows a rapid detection of MRSA in clinical samples and shows the same sensitivity and specificity as traditional tests. Therefore, it represents a further effective diagnostic method for prevention and treatment of nosocomial infections due to methicillin-resistant staphylococci

First detection of X-ray polarization from the accreting neutron star 4U 1820-303

This paper reports the first detection of polarization in the X-rays for atoll-source 4U 1820-303, obtained with the Imaging X-ray Polarimetry Explorer (IXPE) at 99.999% confidence level (CL). Simultaneous polarimetric measurements were also performed in the radio with the Australia Telescope Compact Array (ATCA). The IXPE observations of 4U 1820-303 were coordinated with Swift-XRT, NICER, and NuSTAR aiming to obtain an accurate X-ray spectral model covering a broad energy interval. The source shows a significant polarization above 4 keV, with a polarization degree of 2.0(0.5)% and a polarization angle of -55(7) deg in the 4-7 keV energy range, and a polarization degree of 10(2)% and a polarization angle of -67(7) deg in the 7-8 keV energy bin. This polarization also shows a clear energy trend with polarization degree increasing with energy and a hint for a position-angle change of about 90 deg at 96% CL around 4 keV. The spectro-polarimetric fit indicates that the accretion disk is polarized orthogonally to the hard spectral component, which is presumably produced in the boundary/spreading layer. We do not detect linear polarization from the radio counterpart, with a 99.97% upper limit of 50% at 7.25 GHz