Trajectory-Based Off-Policy Deep Reinforcement Learning

Policy gradient methods are powerful reinforcement learning algorithms and have been demonstrated to solve many complex tasks. However, these methods are also data-inefficient, afflicted with high variance gradient estimates, and frequently get stuck in local optima. This work addresses these weaknesses by combining recent improvements in the reuse of off-policy data and exploration in parameter space with deterministic behavioral policies. The resulting objective is amenable to standard neural network optimization strategies like stochastic gradient descent or stochastic gradient Hamiltonian Monte Carlo. Incorporation of previous rollouts via importance sampling greatly improves data-efficiency, whilst stochastic optimization schemes facilitate the escape from local optima. We evaluate the proposed approach on a series of continuous control benchmark tasks. The results show that the proposed algorithm is able to successfully and reliably learn solutions using fewer system interactions than standard policy gradient methods.Comment: Includes appendix. Accepted for ICML 201

Probabilistic Recurrent State-Space Models

State-space models (SSMs) are a highly expressive model class for learning patterns in time series data and for system identification. Deterministic versions of SSMs (e.g. LSTMs) proved extremely successful in modeling complex time series data. Fully probabilistic SSMs, however, are often found hard to train, even for smaller problems. To overcome this limitation, we propose a novel model formulation and a scalable training algorithm based on doubly stochastic variational inference and Gaussian processes. In contrast to existing work, the proposed variational approximation allows one to fully capture the latent state temporal correlations. These correlations are the key to robust training. The effectiveness of the proposed PR-SSM is evaluated on a set of real-world benchmark datasets in comparison to state-of-the-art probabilistic model learning methods. Scalability and robustness are demonstrated on a high dimensional problem

Results of the 1st external quality assurance for SARS new coronavirus diagnostic PCR and serology : talk

Background The detection of the new Coranavirus (CoV) causing agent of the severe acute respiratory syndrome (SARS) for diagnostic purposes is still a critical step in prevention of secondary hospital infections. In this respect the PCR for SARS diagnostic is the fastest and most sensitive method and was published very early after the description of the new pathogen by different groups. To evaluate the quality and sensitivity of the SARS PCR performed in diagnostic laboratories all over the world an external quality assurance (EQA) for SARS PCR was initiated by the WHO, the European Network for Diagnostics of "Imported" Viral Diseases (ENIVD) and the Robert Koch-Institut. Methods Therefore 10 samples of inactivated SARS CoV strains isolated in Frankfurt and Hong Kong in different dilutions and negative controls were prepared. The freeze dried samples were send by mail to 62 different laboratories, in 37 countries in Europe and Israel (35), Asia (11), The Americas (11), Australia and New Zealand (4) and Africa (1). The results were returned by email or fax 1 week (13), 2 weeks (14), 3 weeks (6) and later (29) after receiving the material which does not mimic at all the possible speed of this fast method. But this was not considered in the evaluation of these first SARS EQA. Results 44 laboratories showed good or excellent results (26 = 100%, 18 = 90%) and even the 14 laboratories which archived only 80% (10) or 70% (4) correct results are mostly lacking sensitivity. The results of the other 4 laboratories show basic problems in regard to sensitivity, specificity and consistency of results and must be overcome as soon as possible. 4 laboratories seem to have problems with the specificity finding a positive signal in negative samples. The different methods used for preparation of the SARS CoV genome and diagnostic PCR test procedure used by the participating laboratories will be discussed in more detail in the presentation. Conclusion However, in contrast to previous EQAs for Ebola, Lassa and Orthopoxviruses the quality of PCR results was rather good which might be caused by the early publication and distribution of well developed PCR methods. An EQA for evaluation of SARS specific serology is still ongoing, first results will be available beginning of April 2004

ERASURE : early autologous blood pleurodesis for postoperative air leaks : a randomized, controlled trial comparing prophylactic autologous blood pleurodesis versus standard watch and wait treatment for postoperative air leaks following thoracoscopic anatomic lung resections

Background: The prolonged air leak is probably the most common complication following lung resections. Around 10–20% of the patients who undergo a lung resection will eventually develop a prolonged air leak. The definition of a prolonged air leak varies between an air leak, which is evident after the fifth, seventh or even tenth postoperative day to every air leak that prolongs the hospital stay. However, the postoperative hospital stay following a thoracoscopic lobectomy can be as short as 2 days, making the above definitions sound outdated. The treatment of these air leaks is also very versatile. One of the broadly accepted treatment options is the autologous blood pleurodesis or “blood patch”. The purpose of this trial is to investigate the impact of a prophylactic autologous blood pleurodesis on reducing the duration of the postoperative air leak and therefore prevent the air leak from becoming prolonged. Methods: Patients undergoing an elective thoracoscopic anatomic lung resection for primary lung cancer or metastatic disease will be eligible for recruitment. Patients with an air leak of > 100 ml/min within 6 h prior to the morning round on the second postoperative day will be eligible for inclusion in the study and randomization. Patients will be randomized to either blood pleurodesis or watchful waiting. The primary endpoint is the time to drain removal measured in full days. The trial ends on the seventh postoperative day. Discussion: The early autologous blood pleurodesis could lead to a faster cessation of the air leak and therefore to a faster removal of the drain. A faster removal of the drain would relieve the patient from all the well-known drain-associated complications (longer hospital stay, stronger postoperative pain, risk of drain-associated infection, etc.). From the economical point of view, faster drain removal would reduce the hospital costs as well as the costs associated with the care of a patient with a chest drain in an outpatient setting

On the impact of the cutoff time on the performance of algorithm configurators

Algorithm conigurators are automated methods to optimise the parameters of an algorithm for a class of problems. We evaluate the performance of a simple random local search conigurator (Param- RLS) for tuning the neighbourhood size k of the RLS k algorithm. We measure performance as the expected number of coniguration evaluations required to identify the optimal value for the parameter. We analyse the impact of the cutof time κ (the time spent evaluat- ing a coniguration for a problem instance) on the expected number of coniguration evaluations required to ind the optimal parameter value, where we compare conigurations using either best found itness values (ParamRLS-F) or optimisation times (ParamRLS-T). We consider tuning RLS k for a variant of the Ridge function class ( Ridge* ), where the performance of each parameter value does not change during the run, and for the OneMax function class, where longer runs favour smaller k . We rigorously prove that ParamRLS- F eiciently tunes RLS k for Ridge* for any κ while ParamRLS-T requires at least quadratic κ . For OneMax ParamRLS-F identiies k = 1 as optimal with linear κ while ParamRLS-T requires a κ of at least Ω ( n log n ) . For smaller κ ParamRLS-F identiies that k > 1 performs better while ParamRLS-T returns k chosen uniformly at random

Scientists' warning on extreme wildfire risks to water supply

2020 is the year of wildfire records. California experienced its three largest fires early in its fire season. The Pantanal, the largest wetland on the planet, burned over 20% of its surface. More than 18 million hectares of forest and bushland burned during the 2019–2020 fire season in Australia, killing 33 people, destroying nearly 2500 homes, and endangering many endemic species. The direct cost of damages is being counted in dozens of billion dollars, but the indirect costs on water‐related ecosystem services and benefits could be equally expensive, with impacts lasting for decades. In Australia, the extreme precipitation (“200 mm day −1 in several location”) that interrupted the catastrophic wildfire season triggered a series of watershed effects from headwaters to areas downstream. The increased runoff and erosion from burned areas disrupted water supplies in several locations. These post‐fire watershed hazards via source water contamination, flash floods, and mudslides can represent substantial, systemic long‐term risks to drinking water production, aquatic life, and socio‐economic activity. Scenarios similar to the recent event in Australia are now predicted to unfold in the Western USA. This is a new reality that societies will have to live with as uncharted fire activity, water crises, and widespread human footprint collide all‐around of the world. Therefore, we advocate for a more proactive approach to wildfire‐watershed risk governance in an effort to advance and protect water security. We also argue that there is no easy solution to reducing this risk and that investments in both green (i.e., natural) and grey (i.e., built) infrastructure will be necessary. Further, we propose strategies to combine modern data analytics with existing tools for use by water and land managers worldwide to leverage several decades worth of data and knowledge on post‐fire hydrology

Identification of a novel coronavirus in patients with severe acute respiratory syndrome

BACKGROUND: The severe acute respiratory syndrome (SARS) has recently been identified as a new clinical entity. SARS is thought to be caused by an unknown infectious agent. METHODS: Clinical specimens from patients with SARS were searched for unknown viruses with the use of cell cultures and molecular techniques. RESULTS: A novel coronavirus was identified in patients with SARS. The virus was isolated in cell culture, and a sequence 300 nucleotides in length was obtained by a polymerase-chain-reaction (PCR)-based random-amplification procedure. Genetic characterization indicated that the virus is only distantly related to known coronaviruses (identical in 50 to 60 percent of the nucleotide sequence). On the basis of the obtained sequence, conventional and real-time PCR assays for specific and sensitive detection of the novel virus were established. Virus was detected in a variety of clinical specimens from patients with SARS but not in controls. High concentrations of viral RNA of up to 100 million molecules per milliliter were found in sputum. Viral RNA was also detected at extremely low concentrations in plasma during the acute phase and in feces during the late convalescent phase. Infected patients showed seroconversion on the Vero cells in which the virus was isolated. CONCLUSIONS: The novel coronavirus might have a role

Single-cell transcriptomic atlas-guided development of CAR-T cells for the treatment of acute myeloid leukemia

A single-cell screening approach identifies targets for CAR-T cells in acute myeloid leukemia. Chimeric antigen receptor T cells (CAR-T cells) have emerged as a powerful treatment option for individuals with B cell malignancies but have yet to achieve success in treating acute myeloid leukemia (AML) due to a lack of safe targets. Here we leveraged an atlas of publicly available RNA-sequencing data of over 500,000 single cells from 15 individuals with AML and tissue from 9 healthy individuals for prediction of target antigens that are expressed on malignant cells but lacking on healthy cells, including T cells. Aided by this high-resolution, single-cell expression approach, we computationally identify colony-stimulating factor 1 receptor and cluster of differentiation 86 as targets for CAR-T cell therapy in AML. Functional validation of these established CAR-T cells shows robust in vitro and in vivo efficacy in cell line- and human-derived AML models with minimal off-target toxicity toward relevant healthy human tissues. This provides a strong rationale for further clinical development