Deep Reinforcement Learning with Feedback-based Exploration

Deep Reinforcement Learning has enabled the control of increasingly complex and high-dimensional problems. However, the need of vast amounts of data before reasonable performance is attained prevents its widespread application. We employ binary corrective feedback as a general and intuitive manner to incorporate human intuition and domain knowledge in model-free machine learning. The uncertainty in the policy and the corrective feedback is combined directly in the action space as probabilistic conditional exploration. As a result, the greatest part of the otherwise ignorant learning process can be avoided. We demonstrate the proposed method, Predictive Probabilistic Merging of Policies (PPMP), in combination with DDPG. In experiments on continuous control problems of the OpenAI Gym, we achieve drastic improvements in sample efficiency, final performance, and robustness to erroneous feedback, both for human and synthetic feedback. Additionally, we show solutions beyond the demonstrated knowledge.Comment: 6 page

Learning Sequential Force Interaction Skills

Learning skills from kinesthetic demonstrations is a promising way of minimizing the gap between human manipulation abilities and those of robots. We propose an approach to learn sequential force interaction skills from such demonstrations. The demonstrations are decomposed into a set of movement primitives by inferring the underlying sequential structure of the task. The decomposition is based on a novel probability distribution which we call Directional Normal Distribution. The distribution allows infering the movement primitive’s composition, i.e., its coordinate frames, control variables and target coordinates from the demonstrations. In addition, it permits determining an appropriate number of movement primitives for a task via model selection. After finding the task’s composition, the system learns to sequence the resulting movement primitives in order to be able to reproduce the task on a real robot. We evaluate the approach on three different tasks, unscrewing a light bulb, box stacking and box flipping. All tasks are kinesthetically demonstrated and then reproduced on a Barrett WAM robot

Non-high risk PE in the patients with acute or exacerbated respiratory disease: the value of the algorithm based on D-dimer evaluation and Revised Geneva Score

Introduction: The diagnostic algorithm of non-high risk pulmonary embolism (PE) is based on probability scoring systems and plasma D-dimer (DD) assessment. The aim of the present study was to investigate the efficacy of Revised Geneva Scoring (RGS) and DD testing for the excluding of non-high risk PE, in the patients admitted to the hospital due to acute respiratory diseases. Material and methods: The consecutive patients, above 18 years of age, referred to the department of lung diseases, entered the study. The exclusion criteria were: the pregnancy and the suspicion of high risk PE. Plasma DD was measured with quick ELISA test, VIDAS D-dimer New, bioMerieux, France. Multislice computed tomography angiography was performed in all of the patients. Results: 153 patients, median age 65 (19−88) years entered the study. The probability of PE was: low — in 58 patients (38%), intermediate — in 90 (59%), high — in 5 (3%). DD < 500 ng/ml was found in 12% of patients with low and intermediate probability of PE. PE was recognized in 10 out of 153 patients (7%). None of the patients with DD < 500 ng/ml was diagnosed with PE (NPV 100%). Median DD value was significantly higher in PE patients comparing to non-PE (4500 ng/ml and 1356 ng/ml respectively, p = 0.006). Conclusion: In the group of the patients with acute respiratory symptoms, low or intermediate clinical probability scoring combined with normal DD had a high NPV in excluding PE. Nevertheless, such approach was not very effective, as the increased DD was noted in 88% of the examined population.Introduction: The diagnostic algorithm of non-high risk pulmonary embolism (PE) is based on probability scoring systems and plasma D-dimer (DD) assessment. The aim of the present study was to investigate the efficacy of Revised Geneva Scoring (RGS) and DD testing for the excluding of non-high risk PE, in the patients admitted to the hospital due to acute respiratory diseases. Material and methods: The consecutive patients, above 18 years of age, referred to the department of lung diseases, entered the study. The exclusion criteria were: the pregnancy and the suspicion of high risk PE. Plasma DD was measured with quick ELISA test, VIDAS D-dimer New, bioMerieux, France. Multislice computed tomography angiography was performed in all of the patients. Results: 153 patients, median age 65 (19−88) years entered the study. The probability of PE was: low — in 58 patients (38%), intermediate — in 90 (59%), high — in 5 (3%). DD < 500 ng/ml was found in 12% of patients with low and intermediate probability of PE. PE was recognized in 10 out of 153 patients (7%). None of the patients with DD < 500 ng/ml was diagnosed with PE (NPV 100%). Median DD value was significantly higher in PE patients comparing to non-PE (4500 ng/ml and 1356 ng/ml respectively, p = 0.006). Conclusion: In the group of the patients with acute respiratory symptoms, low or intermediate clinical probability scoring combined with normal DD had a high NPV in excluding PE. Nevertheless, such approach was not very effective, as the increased DD was noted in 88% of the examined population

Learning modular policies for robotics

A promising idea for scaling robot learning to more complex tasks is to use elemental behaviors as building blocks to compose more complex behavior. Ideally, such building blocks are used in combination with a learning algorithm that is able to learn to select, adapt, sequence and co-activate the building blocks. While there has been a lot of work on approaches that support one of these requirements, no learning algorithm exists that unifies all these properties in one framework. In this paper we present our work on a unified approach for learning such a modular control architecture. We introduce new policy search algorithms that are based on information-theoretic principles and are able to learn to select, adapt and sequence the building blocks. Furthermore, we developed a new representation for the individual building block that supports co-activation and principled ways for adapting the movement. Finally, we summarize our experiments for learning modular control architectures in simulation and with real robots

Nadciśnienie płucne w przebiegu sarkoidozy leczone sildenafilem

Rozwój nadciśnienia płucnego w przebiegu sarkoidozy (SAPH) istotnie pogarsza rokowanie. Niestety, nie ma metody leczenia o udokumentowanym korzystnym działaniu w tej postaci nadciśnienia płucnego. Duże zainteresowanie w tym zakresie budzą leki stosowane w terapii tętniczego nadciśnienia płucnego (PAH). W prezentowanej pracy przedstawiono opis przypadku chorej na sarkoidozę powikłaną ciężkim nadciśnieniem płucnym, która była leczona sildenafilem. Obserwowano istotną, choć zaledwie przejściową poprawę stanu czynnościowego. Pacjentka zmarła z powodu stopniowo postępującej niewydolności krążenia i oddychania, w trakcie oczekiwania na przeszczepienie płuc.Rozwój nadciśnienia płucnego w przebiegu sarkoidozy (SAPH) istotnie pogarsza rokowanie. Niestety, nie ma metody leczenia o udokumentowanym korzystnym działaniu w tej postaci nadciśnienia płucnego. Duże zainteresowanie w tym zakresie budzą leki stosowane w terapii tętniczego nadciśnienia płucnego (PAH). W prezentowanej pracy przedstawiono opis przypadku chorej na sarkoidozę powikłaną ciężkim nadciśnieniem płucnym, która była leczona sildenafilem. Obserwowano istotną, choć zaledwie przejściową poprawę stanu czynnościowego. Pacjentka zmarła z powodu stopniowo postępującej niewydolności krążenia i oddychania, w trakcie oczekiwania na przeszczepienie płuc

Dapagliflozin and Diuretic Use in Patients With Heart Failure and Reduced Ejection Fraction in DAPA-HF

Background: In the DAPA-HF trial (Dapagliflozin and Prevention of Adverse-Outcomes in Heart Failure), the sodium-glucose cotransporter 2 inhibitor dapagliflozin reduced the risk of worsening heart failure and death in patients with heart failure and reduced ejection fraction. We examined the efficacy and tolerability of dapagliflozin in relation to background diuretic treatment and change in diuretic therapy after randomization to dapagliflozin or placebo. Methods: We examined the effects of study treatment in the following subgroups: No diuretic and diuretic dose equivalent to furosemide 40 mg daily at baseline. We examined the primary composite end point of cardiovascular death or a worsening heart failure event and its components, all-cause death and symptoms. Results: Of 4616 analyzable patients, 736 (15.9%) were on no diuretic, 1311 (28.4%) were on 40 mg. Compared with placebo, dapagliflozin reduced the risk of the primary end point across each of these subgroups: Hazard ratios were 0.57 (95% CI, 0.36-0.92), 0.83 (95% CI, 0.63-1.10), 0.77 (95% CI, 0.60-0.99), and 0.78 (95% CI, 0.63-0.97), respectively (P for interaction=0.61). The hazard ratio in patients taking any diuretic was 0.78 (95% CI, 0.68-0.90). Improvements in symptoms and treatment toleration were consistent across the diuretic subgroups. Diuretic dose did not change in most patients during follow-up, and mean diuretic dose did not differ between the dapagliflozin and placebo groups after randomization. Conclusions: The efficacy and safety of dapagliflozin were consistent across the diuretic subgroups examined in DAPA-HF

The "smoker's paradox" in patients with acute coronary syndrome: a systematic review

<p>Abstract</p> <p>Background</p> <p>Smokers have been shown to have lower mortality after acute coronary syndrome than non-smokers. This has been attributed to the younger age, lower co-morbidity, more aggressive treatment and lower risk profile of the smoker. Some studies, however, have used multivariate analyses to show a residual survival benefit for smokers; that is, the "smoker's paradox". The aim of this study was, therefore, to perform a systematic review of the literature and evidence surrounding the existence of the "smoker's paradox".</p> <p>Methods</p> <p>Relevant studies published by September 2010 were identified through literature searches using EMBASE (from 1980), MEDLINE (from 1963) and the Cochrane Central Register of Controlled Trials, with a combination of text words and subject headings used. English-language original articles were included if they presented data on hospitalised patients with defined acute coronary syndrome, reported at least in-hospital mortality, had a clear definition of smoking status (including ex-smokers), presented crude and adjusted mortality data with effect estimates, and had a study sample of > 100 smokers and > 100 non-smokers. Two investigators independently reviewed all titles and abstracts in order to identify potentially relevant articles, with any discrepancies resolved by repeated review and discussion.</p> <p>Results</p> <p>A total of 978 citations were identified, with 18 citations from 17 studies included thereafter. Six studies (one observational study, three registries and two randomised controlled trials on thrombolytic treatment) observed a "smoker's paradox". Between the 1980s and 1990s these studies enrolled patients with acute myocardial infarction (AMI) according to criteria similar to the World Health Organisation criteria from 1979. Among the remaining 11 studies not supporting the existence of the paradox, five studies represented patients undergoing contemporary management.</p> <p>Conclusion</p> <p>The "smoker's paradox" was observed in some studies of AMI patients in the pre-thrombolytic and thrombolytic era, whereas no studies of a contemporary population with acute coronary syndrome have found evidence for such a paradox.</p

Inhibition of Firefly Luciferase by General Anesthetics: Effect on In Vitro and In Vivo Bioluminescence Imaging

<div><h3></h3><p>Bioluminescence imaging is routinely performed in anesthetized mice. Often isoflurane anesthesia is used because of its ease of use and fast induction/recovery. However, general anesthetics have been described as important inhibitors of the luciferase enzyme reaction.</p> <h3>Aim</h3><p>To investigate frequently used mouse anesthetics for their direct effect on the luciferase reaction, both in vitro and in vivo.</p> <h3>Materials and Methods</h3><p>isoflurane, sevoflurane, desflurane, ketamine, xylazine, medetomidine, pentobarbital and avertin were tested in vitro on luciferase-expressing intact cells, and for non-volatile anesthetics on intact cells and cell lysates. In vivo, isoflurane was compared to unanesthetized animals and different anesthetics. Differences in maximal photon emission and time-to-peak photon emission were analyzed.</p> <h3>Results</h3><p>All volatile anesthetics showed a clear inhibitory effect on the luciferase activity of 50% at physiological concentrations. Avertin had a stronger inhibitory effect of 80%. For ketamine and xylazine, increased photon emission was observed in intact cells, but this was not present in cell lysate assays, and was most likely due to cell toxicity and increased cell membrane permeability. In vivo, the highest signal intensities were measured in unanesthetized mice and pentobarbital anesthetized mice, followed by avertin. Isoflurane and ketamine/medetomidine anesthetized mice showed the lowest photon emission (40% of unanesthetized), with significantly longer time-to-peak than unanesthetized, pentobarbital or avertin-anesthetized mice. We conclude that, although strong inhibitory effects of anesthetics are present in vitro, their effect on in vivo BLI quantification is mainly due to their hemodynamic effects on mice and only to a lesser extent due to the direct inhibitory effect.</p> </div