Scaling Reinforcement Learning Paradigms for Motor Control

Reinforcement learning offers a general framework to explain reward related learning in artificial and biological motor control. However, current reinforcement learning methods rarely scale to high dimensional movement systems and mainly operate in discrete, low dimensional domains like game-playing, artificial toy problems, etc. This drawback makes them unsuitable for application to human or bio-mimetic motor control. In this poster, we look at promising approaches that can potentially scale and suggest a novel formulation of the actor-critic algorithm which takes steps towards alleviating the current shortcomings. We argue that methods based on greedy policies are not likely to scale into high-dimensional domains as they are problematic when used with function approximation – a must when dealing with continuous domains. We adopt the path of direct policy gradient based policy improvements since they avoid the problems of unstabilizing dynamics encountered in traditional value iteration based updates. While regular policy gradient methods have demonstrated promising results in the domain of humanoid notor control, we demonstrate that these methods can be significantly improved using the natural policy gradient instead of the regular policy gradient. Based on this, it is proved that Kakade’s ‘average natural policy gradient’ is indeed the true natural gradient. A general algorithm for estimating the natural gradient, the Natural Actor-Critic algorithm, is introduced. This algorithm converges with probability one to the nearest local minimum in Riemannian space of the cost function. The algorithm outperforms nonnatural policy gradients by far in a cart-pole balancing evaluation, and offers a promising route for the development of reinforcement learning for truly high-dimensionally continuous state-action systems. Keywords: Reinforcement learning, neurodynamic programming, actorcritic methods, policy gradient methods, natural policy gradien

Relationship between antihypertensive medications and cognitive impairment: Part II. Review of Physiology and animal studies

Purpose of Review There is an established association between hypertension and increased risk of poor cognitive performance and dementia including Alzheimer’s disease; however, associations between antihypertensive medications (AHM) and dementia risk are less clear. An increased interest in AHM has resulted in expanding publications; however, none of the recent reviews provide comprehensive review. Our extensive review includes 24 mechanistic animal and human studies published over the last 5 years assessing relationship between AHM and cognitive function. Recent Findings All classes of AHM showed similar result patterns in animal studies. The mechanism by which AHM exert their effect was extensively studied by evaluating well-established pathways of AD disease process, including amyloid beta (Aβ), vascular, oxidative stress and inflammation pathways, but only few studies evaluated the blood pressure lowering effect on the AD disease process. Summary Methodological limitations of the studies prevent comprehensive conclusions prior to further work evaluating AHM in animals and larger human observational studies, and selecting those with promising results for future RCTs

Relationship between antihypertensive medications and cognitive impairment: Part I. review of human studies and clinical trials

Purpose of review: There is an established association between hypertension and increased risk of poor cognitive performance and dementia including Alzheimer’s disease; however, associations between antihypertensive medications (AHMs) and dementia risk are less consistent. An increased interest in AHM has resulted in expanding publications; however, none of the recent reviews are comprehensive. Our extensive review includes 15 observational and randomized controlled trials (RCTs) published over the last 5 years, assessing the relationship between AHM and cognitive impairment. Recent findings: All classes of AHM showed similar result patterns in human studies with the majority of study results reporting point estimates below one and only a small number of studies (N = 15) reporting statistically significant results in favor of a specific class. Summary: Only a small number of studies reported statistically significant results in favor of a specific class of AHM. Methodological limitations of the studies prevent definitive conclusions. Further work is now needed to evaluate the class of AHM and cognitive outcomes in future RCTs, with a particular focus on the drugs with the promising results in both animals and human observational studies

Computing Functions of Random Variables via Reproducing Kernel Hilbert Space Representations

We describe a method to perform functional operations on probability distributions of random variables. The method uses reproducing kernel Hilbert space representations of probability distributions, and it is applicable to all operations which can be applied to points drawn from the respective distributions. We refer to our approach as {\em kernel probabilistic programming}. We illustrate it on synthetic data, and show how it can be used for nonparametric structural equation models, with an application to causal inference

The clinical effectiveness and cost-effectiveness of inhaler devices used in the routine management of chronic asthma in older children: a systematic review and economic evaluation

Background: This review examines the clinical effectiveness and cost-effectiveness of hand-held inhalers to deliver medication for the routine management of chronic asthma in children aged between 5 and 15 years. Asthma is a common disease of the airways, with a prevalence of treated asthma in 5–15-year-olds of around 12% and an actual prevalence in the community as high as 23%. Treatment for the condition is predominantly by inhalation of medication. There are three main types of inhaler device, pressurised metered dose, breath actuated, and dry powder, with the option of the attachment of a spacer to the first two devices under some prescribed circumstances. Two recent reviews have examined the clinical and cost-effectiveness evidence on inhaler devices, but one was for children aged under 5 years and the comparison in the second was made between pressurised metered dose inhalers and other types only. Objectives: This review examines the clinical effectiveness and cost-effectiveness of manual pressurised metered dose inhalers, breath-actuated metered dose inhalers, and breath-actuated dry powder inhalers, with and without spacers as appropriate, to deliver medication for the routine management of chronic asthma in children aged between 5 and 15 years. Methods: Two previous HTA reviews have compared the effectiveness of inhaler devices, one focusing on asthma in children aged under 5 years and the other on asthma and chronic obstructive airways disease in all age groups. For the current review, a literature search was carried out to identify all evidence relating to the use of inhalers in older children with chronic asthma. A search of in-vitro studies undertaken for one of the previous reviews was also updated. The data sources used were: 15 electronic bibliographic databases; the reference lists of one of the previous HTA reports and other relevant articles; health services research-related internet resources; and all sponsor submissions. Studies were selected according to strict inclusion and exclusion criteria, and relevant information concerning effectiveness and patient compliance and preference was extracted directly on to an extraction/evidence table. Quality assurance was monitored. Economic evaluation was undertaken by reviewing existing cost-effective evidence. Further economic modelling was carried out, and tables constructed to determine device cost-minimisation and incremental quality-adjusted life-year (QALY) thresholds between devices. Results: Number and quality of studies, and direction of evidence: Fourteen randomised controlled studies were identified relating to the clinical effectiveness of inhaler devices for delivering β2-agonists. A further five were on devices delivering corticosteroids and one concerned the delivery of cromoglicate. Overall, there were no differences in clinical efficacy between inhaler devices, but a pressurised metered dose inhaler with a spacer would appear to be more effective than one without. These findings endorse those of a previous HTA review but extend them to other inhaler devices. Seven randomised controlled trials examined the impact on clinical effectiveness of using a nonchlorofluorocarbon (CFC) propellant in place of a CFC propellant in metered dose inhalers, both pressurised and breath activated, although only one study considered the latter type. No differences were found between inhalers containing either propellant. A further 30 studies of varying quality, from 12 randomised controlled trials to non-controlled studies, were identified that concerned the impact of use by, and preference for, inhaler type, and treatment adherence in children. Differences between the studies, and limitations in comparative data between various inhaler device types, make it difficult to draw any firm conclusions from this evidence. Summary of benefits: No obvious benefits for one inhaler device type over another for use in children aged 5–15 years were identified. Costs and cost per quality-adjusted life-year: Two approaches have been taken: cost-minimisation and QALY threshold. In the QALY threshold approach, additional QALYs that each device must produce compared with a cheaper device to achieve an acceptable cost per QALY were calculated. Using the cheapest and most expensive devices for delivering 200 μg of beclometasone per day, assuming no cost offset for any device, and a threshold of £5000, the largest QALY needed was 0.00807. With such a small QALY increase, no intervention can be categorically rejected as not cost-effective. Conclusions: Generalisability of findings: On the available evidence there are no obvious benefits for one inhaler device over another when used by children aged 5–15 years with chronic asthma. However, the evidence, in the majority of cases, was compiled on children with mild to moderate asthma and restricted to a limited number of drugs. Therefore the findings may not be generalisable to those at the more severe end of the spectrum of the disease or to inhaler devices delivering some of the drugs used in the management of asthma. Need for further research: Many of the previous studies are likely to have been underpowered. Further clinical trials with a robust methodology, sufficient power and qualitative components are needed to demonstrate any differences in clinical resource use and patients’ asthma symptoms. Further studies should also include the behavioural aspects of patients towards their medication and its delivery mechanisms. It is acknowledged that sufficient power may prove impractical owing to the large numbers of patients required

Spatial electron density and electric field strength measurements in microwave cavity experiments

Measurements of electron density and electric field strength have been made in an argon plasma contained in a resonant microwave cavity at 2.45 GHz. Spatial measurements of electron density, n sub e, are correlated with fluorescence observations of the discharge. Measurements of n sub e were made with Stark broadening and compared with n sub 3 calculated from measured plasma conductivity. Additional measurements of n sub 3 as a function of pressure and in mixtures of argon and oxygen are presented for pressures from 10 Torr to 1 atm. Measurements in flowing gases and in static systems are presented. In addition, limitations of these measurements are identified