Towards real-time reinforcement learning control of a wave energy converter

The levellised cost of energy of wave energy converters (WECs) is not competitive with fossil fuel-powered stations yet. To improve the feasibility of wave energy, it is necessary to develop effective control strategies that maximise energy absorption in mild sea states, whilst limiting motions in high waves. Due to their model-based nature, state-of-the-art control schemes struggle to deal with model uncertainties, adapt to changes in the system dynamics with time, and provide real-time centralised control for large arrays of WECs. Here, an alternative solution is introduced to address these challenges, applying deep reinforcement learning (DRL) to the control of WECs for the first time. A DRL agent is initialised from data collected in multiple sea states under linear model predictive control in a linear simulation environment. The agent outperforms model predictive control for high wave heights and periods, but suffers close to the resonant period of the WEC. The computational cost at deployment time of DRL is also much lower by diverting the computational effort from deployment time to training. This provides confidence in the application of DRL to large arrays of WECs, enabling economies of scale. Additionally, model-free reinforcement learning can autonomously adapt to changes in the system dynamics, enabling fault-tolerant control

Exploiting biological and physical determinants of radiotherapy toxicity to individualise treatment.

This is the final version of the article. It first appeared from the British Institute of Radiology via http://dx.doi.org/10.1259/bjr.20150172The recent advances in radiation delivery can improve tumour control probability and reduce treatment related toxicity. The use of intensity-modulated radiotherapy (IMRT) in particular can reduce normal tissue toxicity, an objective in its own right, and can allow safe dose escalation in selected cases. Ideally IMRT should be combined with image guidance to verify the position of the target, since patients, target and organs at risk can move day-to-day. Daily image guidance scans can be used to identify the position of normal tissue structures, and potentially to compute the daily delivered dose. Fundamentally, it is still the tolerance of the normal tissues which limits radiotherapy dose and therefore tumour control. However, the dose response relationships for both tumour and normal tissues are relatively steep, meaning that small dose differences can translate into clinically relevant improvements. Differences exist between individuals in the severity of toxicity experienced for a given dose of radiotherapy. Some of this difference may be the result of differences between the planned dose and the accumulated dose (DA). However, some may be due to intrinsic differences in radiosensitivity of the normal tissues between individuals. This field has been developing rapidly, with the demonstration of definite associations between genetic polymorphisms and variation in toxicity recently described. It might be possible to identify more resistant patients who would be suitable for dose escalation, as well as more sensitive patients for whom toxicity could be reduced or avoided. Daily differences in delivered dose have been investigated within the VoxTox research programme, using the rectum as an example organ at risk. In prostate cancer patients receiving curative radiotherapy, considerable daily variation in rectal position and dose can be demonstrated, although the median position matches the planning scan well. Overall, in 10 patients, the mean difference between planned and accumulated rectal equivalent uniform doses (EUDs) was -2.7 Gy (5%), and a dose reduction was seen in 7/10 cases. If dose escalation were performed to take rectal dose back to the planned level, this should increase the mean tumour control probability (TCP) (as biochemical progression-free survival) by 5%. Combining radiogenomics with individual estimates of DA might identify almost half of patients undergoing radical radiotherapy who might benefit from either dose escalation, suggesting improved tumour cure, or reduced toxicity, or both.JS is supported by Cancer Research UK through the Cambridge Cancer Centre. NGB is supported by the NIHR Cambridge Biomedical Research Centre. The VoxTox Research Programme is funded by Cancer Research UK

Docking Control of an Autonomous Underwater Vehicle Using Reinforcement Learning

To achieve persistent systems in the future, autonomous underwater vehicles (AUVs) will need to autonomously dock onto a charging station. Here, reinforcement learning strategies were applied for the first time to control the docking of an AUV onto a fixed platform in a simulation environment. Two reinforcement learning schemes were investigated: one with continuous state and action spaces, deep deterministic policy gradient (DDPG), and one with continuous state but discrete action spaces, deep Q network (DQN). For DQN, the discrete actions were selected as step changes in the control input signals. The performance of the reinforcement learning strategies was compared with classical and optimal control techniques. The control actions selected by DDPG suffer from chattering effects due to a hyperbolic tangent layer in the actor. Conversely, DQN presents the best compromise between short docking time and low control effort, whilst meeting the docking requirements. Whereas the reinforcement learning algorithms present a very high computational cost at training time, they are five orders of magnitude faster than optimal control at deployment time, thus enabling an on-line implementation. Therefore, reinforcement learning achieves a performance similar to optimal control at a much lower computational cost at deployment, whilst also presenting a more general framework

Platelet-Induced Clumping of Plasmodium falciparum–Infected Erythrocytes from Malawian Patients with Cerebral Malaria—Possible Modulation In Vivo by Thrombocytopenia

Platelets may play a role in the pathogenesis of human cerebral malaria (CM), and they have been shown to induce clumping of Plasmodium falciparum–parasitized red blood cells (PRBCs) in vitro. Both thrombocytopenia and platelet-inducedPRBCclumping are associated with severe malaria and, especially, withCM.In the present study, we investigated the occurrence of the clumping phenomenon in patients with CM by isolating and coincubating their plasma and PRBCs ex vivo. Malawian children with CM all had low platelet counts, with the degree of thrombocytopenia directly proportional to the density of parasitemia. Plasma samples obtained from these patients subsequently induced weak PRBC clumping. When the assays were repeated, with the plasma platelet concentrations adjusted to within the physiological range considered to be normal, massive clumping occurred. The results of this study suggest that thrombocytopenia may, through reduction of platelet-mediated clumping of PRBCs, provide a protective mechanism for the host during CM

Forces between clustered stereocilia minimize friction in the ear on a subnanometre scale

The detection of sound begins when energy derived from acoustic stimuli deflects the hair bundles atop hair cells. As hair bundles move, the viscous friction between stereocilia and the surrounding liquid poses a fundamental challenge to the ear's high sensitivity and sharp frequency selectivity. Part of the solution to this problem lies in the active process that uses energy for frequency-selective sound amplification. Here we demonstrate that a complementary part involves the fluid-structure interaction between the liquid within the hair bundle and the stereocilia. Using force measurement on a dynamically scaled model, finite-element analysis, analytical estimation of hydrodynamic forces, stochastic simulation and high-resolution interferometric measurement of hair bundles, we characterize the origin and magnitude of the forces between individual stereocilia during small hair-bundle deflections. We find that the close apposition of stereocilia effectively immobilizes the liquid between them, which reduces the drag and suppresses the relative squeezing but not the sliding mode of stereociliary motion. The obliquely oriented tip links couple the mechanotransduction channels to this least dissipative coherent mode, whereas the elastic horizontal top connectors stabilize the structure, further reducing the drag. As measured from the distortion products associated with channel gating at physiological stimulation amplitudes of tens of nanometres, the balance of forces in a hair bundle permits a relative mode of motion between adjacent stereocilia that encompasses only a fraction of a nanometre. A combination of high-resolution experiments and detailed numerical modelling of fluid-structure interactions reveals the physical principles behind the basic structural features of hair bundles and shows quantitatively how these organelles are adapted to the needs of sensitive mechanotransduction.Comment: 21 pages, including 3 figures. For supplementary information, please see the online version of the article at http://www.nature.com/natur

Towards Real-Time Reinforcement Learning Control of a Wave Energy Converter

This is the final version. Available on open access from MDPI via the DOI in this recordThe levellised cost of energy of wave energy converters (WECs) is not competitive with fossil-fuel powered stations yet. To improve the feasibility of wave energy, it is necessary to develop effective control strategies that maximise energy absorption in mild sea states, whilst limiting motions in high waves. Due to their model-based nature, state-of-the-art control schemes struggle to deal with model uncertainties, adapt to changes in the system dynamics with time, and provide real-time centralised control for large arrays of WECs. Here, an alternative solution is introduced to address these challenges, applying deep reinforcement learning (DRL) to the control of WECs for the first time. A DRL agent is initialised from data collected in multiple sea states under linear model predictive control in a linear simulation environment. The agent outperforms model predictive control for high wave heights and periods, but suffers close to the resonant period of the WEC. The computational cost at deployment time of DRL is also much lower by diverting the computational effort from deployment time to training. This provides confidence in the application of DRL to large arrays of WECs, enabling economies of scale. Additionally, model-free reinforcement learning can autonomously adapt to changes in the system dynamics, enabling fault-tolerant control

The mental health system in Brazil: Policies and future challenges

This is an Open Access article distributed under the terms of the Creative Commons Attribution Licens