Approximate Shielding of Atari Agents for Safe Exploration

Balancing exploration and conservatism in the constrained setting is an important problem if we are to use reinforcement learning for meaningful tasks in the real world. In this paper, we propose a principled algorithm for safe exploration based on the concept of shielding. Previous approaches to shielding assume access to a safety-relevant abstraction of the environment or a high-fidelity simulator. Instead, our work is based on latent shielding - another approach that leverages world models to verify policy roll-outs in the latent space of a learned dynamics model. Our novel algorithm builds on this previous work, using safety critics and other additional features to improve the stability and farsightedness of the algorithm. We demonstrate the effectiveness of our approach by running experiments on a small set of Atari games with state dependent safety labels. We present preliminary results that show our approximate shielding algorithm effectively reduces the rate of safety violations, and in some cases improves the speed of convergence and quality of the final agent.Comment: Accepted for presentation at the ALA workshop as part of AAMAS 202

Approximate Model-Based Shielding for Safe Reinforcement Learning

Reinforcement learning (RL) has shown great potential for solving complex tasks in a variety of domains. However, applying RL to safety-critical systems in the real-world is not easy as many algorithms are sample-inefficient and maximising the standard RL objective comes with no guarantees on worst-case performance. In this paper we propose approximate model-based shielding (AMBS), a principled look-ahead shielding algorithm for verifying the performance of learned RL policies w.r.t. a set of given safety constraints. Our algorithm differs from other shielding approaches in that it does not require prior knowledge of the safety-relevant dynamics of the system. We provide a strong theoretical justification for AMBS and demonstrate superior performance to other safety-aware approaches on a set of Atari games with state-dependent safety-labels.Comment: Accepted at ECAI 2023 (main technical track

Enabling 3D magnetic circuits by the additive manufacturing of soft magnetic material

Additive manufacturing has been revolutionary in enabling complex structural components such as lattice structures and topologically optimised parts, however the utilisation of functional materials such as soft magnetic materials is only just being realised. 3D magnetic flux pathways in electrical machines have been illusive due to the high eddy current losses caused by thick cross-sections, and the inability to process electrical steel laminations into 3D structures. By processing soft magnetic materials with additive manufacturing, geometry can be tailored to avoid large bulk cross sections and reduce eddy currents whilst maintaining a 3D flux pathway, enabling the creation of new electrical machine architectures in the pursuit of higher power density and efficiency, which may enable the decarbonisation of the transport sectors including commercial aircraft. This thesis demonstrates the processing of high silicon electrical steel (Fe-6.5 wt%Si) using laser powder bed fusion and characterises the magnetic properties of this material. The importance of surface roughness on the magnetic susceptibility is investigated, showing that contours may be used to improve the as-built surface finish, but post-processing methods such as polishing are required to obtain the best magnetic properties. The samples in this study exhibited a weak crystallographic texture and the orientation of the samples in the build chamber displayed little impact on the magnetic susceptibility. Two methods are used in this study to reduce eddy currents and enable components with 3D magnetic flux pathways to be manufactured. The first is by designing thin-walled cross-sections, which use air as an insulating medium to reduce the thick cross section of the material. This method is demonstrated in lab experiments showing a reduction of the eddy current loss coefficient to 0.0005 using a novel hexagonal cross section. A Hilbert pattern was implemented into an axial flux electrical machine, demonstrating loss performance comparable to thick electrical steel laminations below 500 Hz, increasing torque density by 13% by achieving a reduction in volume of magnetic material of 33%. The second method uses process control to create stochastic cracking within the material, demonstrating excellent loss behaviour of 2.2 W/kg (50 Hz, 1T) with stacking factors >97%. The mechanical integrity was confirmed to be adequate for implementation into the axial flux machine tested with a UTS of 25 MPa when embedded with epoxy resin. These methods can be implemented into electrical machines enabling the creation of new architectures, with the hope to increase power density and efficiency. This is the first time that additively manufactured soft magnetic material has been characterised in an electrical machine, overcoming the issues of large cross sections. Although the soft magnetic material has not displayed loss behaviour as good as electrical steel laminations, it does enable 3D magnetic circuits within electrical machines which may be exploited to improve the performance of the machine. Further optimisation of the stochastic cracking method of eddy current management by aligning the cracks with the flux direction will yield further improvements, and may compete with the thinnest laminations. Due to the current cost and limitations of additive manufacturing, this technology is only likely to be implemented into the highest value electrical machines, such as those in top end automotive and aerospace, where benefits in performance are of upmost importance. This development in processing of soft magnetic material is the missing piece to enable fully additively manufactured motors, which could revolutionise electrical machine architecture

The inverse-Compton ghost HDF 130 and the giant radio galaxy 6C 0905+3955: matching an analytic model for double radio source evolution

We present new GMRT observations of HDF 130, an inverse-Compton (IC) ghost of a giant radio source that is no longer being powered by jets. We compare the properties of HDF 130 with the new and important constraint of the upper limit of the radio flux density at 240 MHz to an analytic model. We learn what values of physical parameters in the model for the dynamics and evolution of the radio luminosity and X-ray luminosity (due to IC scattering of the cosmic microwave background (CMB)) of a Fanaroff-Riley II (FR II) source are able to describe a source with features (lobe length, axial ratio, X-ray luminosity, photon index and upper limit of radio luminosity) similar to the observations. HDF 130 is found to agree with the interpretation that it is an IC ghost of a powerful double-lobed radio source, and we are observing it at least a few Myr after jet activity (which lasted 5--100 Myr) has ceased. The minimum Lorentz factor of injected particles into the lobes from the hotspot is preferred to be $\gamma\sim10^3$ for the model to describe the observed quantities well, assuming that the magnetic energy density, electron energy density, and lobe pressure at time of injection into the lobe are linked by constant factors according to a minimum energy argument, so that the minimum Lorentz factor is constrained by the lobe pressure. We also apply the model to match the features of 6C 0905+3955, a classical double FR II galaxy thought to have a low-energy cutoff of $\gamma\sim10^4$ in the hotspot due to a lack of hotspot inverse-Compton X-ray emission. The models suggest that the low-energy cutoff in the hotspots of 6C 0905+3955 is $\gamma\gtrsim 10^3$, just slightly above the particles required for X-ray emission.Comment: 9 pages, 3 figure

Novel Cold-Adapted Lipase from Marine Plankton, Salpa thompsoni

Please click Download on the upper right corner to see the full description

Trieste: Efficiently Exploring The Depths of Black-box Functions with TensorFlow

We present Trieste, an open-source Python package for Bayesian optimization and active learning benefiting from the scalability and efficiency of TensorFlow. Our library enables the plug-and-play of popular TensorFlow-based models within sequential decision-making loops, e.g. Gaussian processes from GPflow or GPflux, or neural networks from Keras. This modular mindset is central to the package and extends to our acquisition functions and the internal dynamics of the decision-making loop, both of which can be tailored and extended by researchers or engineers when tackling custom use cases. Trieste is a research-friendly and production-ready toolkit backed by a comprehensive test suite, extensive documentation, and available at https://github.com/secondmind-labs/trieste