Regret Bounds for Reinforcement Learning with Policy Advice

In some reinforcement learning problems an agent may be provided with a set of input policies, perhaps learned from prior experience or provided by advisors. We present a reinforcement learning with policy advice (RLPA) algorithm which leverages this input set and learns to use the best policy in the set for the reinforcement learning task at hand. We prove that RLPA has a sub-linear regret of \tilde O(\sqrt{T}) relative to the best input policy, and that both this regret and its computational complexity are independent of the size of the state and action space. Our empirical simulations support our theoretical analysis. This suggests RLPA may offer significant advantages in large domains where some prior good policies are provided

Extreme State Aggregation Beyond MDPs

We consider a Reinforcement Learning setup where an agent interacts with an environment in observation-reward-action cycles without any (esp.\ MDP) assumptions on the environment. State aggregation and more generally feature reinforcement learning is concerned with mapping histories/raw-states to reduced/aggregated states. The idea behind both is that the resulting reduced process (approximately) forms a small stationary finite-state MDP, which can then be efficiently solved or learnt. We considerably generalize existing aggregation results by showing that even if the reduced process is not an MDP, the (q-)value functions and (optimal) policies of an associated MDP with same state-space size solve the original problem, as long as the solution can approximately be represented as a function of the reduced states. This implies an upper bound on the required state space size that holds uniformly for all RL problems. It may also explain why RL algorithms designed for MDPs sometimes perform well beyond MDPs.Comment: 28 LaTeX pages. 8 Theorem

Learning Mazes with Aliasing States: An LCS Algorithm with Associative Perception

Learning classifier systems (LCSs) belong to a class of algorithms based on the principle of self-organization and have frequently been applied to the task of solving mazes, an important type of reinforcement learning (RL) problem. Maze problems represent a simplified virtual model of real environments that can be used for developing core algorithms of many real-world applications related to the problem of navigation. However, the best achievements of LCSs in maze problems are still mostly bounded to non-aliasing environments, while LCS complexity seems to obstruct a proper analysis of the reasons of failure. We construct a new LCS agent that has a simpler and more transparent performance mechanism, but that can still solve mazes better than existing algorithms. We use the structure of a predictive LCS model, strip out the evolutionary mechanism, simplify the reinforcement learning procedure and equip the agent with the ability of associative perception, adopted from psychology. To improve our understanding of the nature and structure of maze environments, we analyze mazes used in research for the last two decades, introduce a set of maze complexity characteristics, and develop a set of new maze environments. We then run our new LCS with associative perception through the old and new aliasing mazes, which represent partially observable Markov decision problems (POMDP) and demonstrate that it performs at least as well as, and in some cases better than, other published systems

Clinical delineation and natural history of the PIK3CA-related overgrowth spectrum.

Somatic mutations in the phosphatidylinositol/AKT/mTOR pathway cause segmental overgrowth disorders. Diagnostic descriptors associated with PIK3CA mutations include fibroadipose overgrowth (FAO), Hemihyperplasia multiple Lipomatosis (HHML), Congenital Lipomatous Overgrowth, Vascular malformations, Epidermal nevi, Scoliosis/skeletal and spinal (CLOVES) syndrome, macrodactyly, and the megalencephaly syndrome, Megalencephaly-Capillary malformation (MCAP) syndrome. We set out to refine the understanding of the clinical spectrum and natural history of these phenotypes, and now describe 35 patients with segmental overgrowth and somatic PIK3CA mutations. The phenotypic data show that these previously described disease entities have considerable overlap, and represent a spectrum. While this spectrum overlaps with Proteus syndrome (sporadic, mosaic, and progressive) it can be distinguished by the absence of cerebriform connective tissue nevi and a distinct natural history. Vascular malformations were found in 15/35 (43%) and epidermal nevi in 4/35 (11%) patients, lower than in Proteus syndrome. Unlike Proteus syndrome, 31/35 (89%) patients with PIK3CA mutations had congenital overgrowth, and in 35/35 patients this was asymmetric and disproportionate. Overgrowth was mild with little postnatal progression in most, while in others it was severe and progressive requiring multiple surgeries. Novel findings include: adipose dysregulation present in all patients, unilateral overgrowth that is predominantly left-sided, overgrowth that affects the lower extremities more than the upper extremities and progresses in a distal to proximal pattern, and in the most severely affected patients is associated with marked paucity of adipose tissue in unaffected areas. While the current data are consistent with some genotype-phenotype correlation, this cannot yet be confirmed

Star Formation and Dynamics in the Galactic Centre

The centre of our Galaxy is one of the most studied and yet enigmatic places in the Universe. At a distance of about 8 kpc from our Sun, the Galactic centre (GC) is the ideal environment to study the extreme processes that take place in the vicinity of a supermassive black hole (SMBH). Despite the hostile environment, several tens of early-type stars populate the central parsec of our Galaxy. A fraction of them lie in a thin ring with mild eccentricity and inner radius ~0.04 pc, while the S-stars, i.e. the ~30 stars closest to the SMBH (<0.04 pc), have randomly oriented and highly eccentric orbits. The formation of such early-type stars has been a puzzle for a long time: molecular clouds should be tidally disrupted by the SMBH before they can fragment into stars. We review the main scenarios proposed to explain the formation and the dynamical evolution of the early-type stars in the GC. In particular, we discuss the most popular in situ scenarios (accretion disc fragmentation and molecular cloud disruption) and migration scenarios (star cluster inspiral and Hills mechanism). We focus on the most pressing challenges that must be faced to shed light on the process of star formation in the vicinity of a SMBH.Comment: 68 pages, 35 figures; invited review chapter, to be published in expanded form in Haardt, F., Gorini, V., Moschella, U. and Treves, A., 'Astrophysical Black Holes'. Lecture Notes in Physics. Springer 201

Search for direct production of charginos and neutralinos in events with three leptons and missing transverse momentum in √s = 7 TeV pp collisions with the ATLAS detector

A search for the direct production of charginos and neutralinos in final states with three electrons or muons and missing transverse momentum is presented. The analysis is based on 4.7 fb−1 of proton–proton collision data delivered by the Large Hadron Collider and recorded with the ATLAS detector. Observations are consistent with Standard Model expectations in three signal regions that are either depleted or enriched in Z-boson decays. Upper limits at 95% confidence level are set in R-parity conserving phenomenological minimal supersymmetric models and in simplified models, significantly extending previous results

D* Production in Deep Inelastic Scattering at HERA

This paper presents measurements of D^{*\pm} production in deep inelastic scattering from collisions between 27.5 GeV positrons and 820 GeV protons. The data have been taken with the ZEUS detector at HERA. The decay channel $D^{*+}\to (D^0 \to K^- \pi^+) \pi^+$ (+ c.c.) has been used in the study. The $e^+p$ cross section for inclusive D^{*\pm} production with $5<Q^2<100 GeV^2$ and $y<0.7$ is 5.3 \pms 1.0 \pms 0.8 nb in the kinematic region {$1.3<p_T(D^{*\pm})<9.0$ GeV and $| \eta(D^{*\pm}) |<1.5$}. Differential cross sections as functions of p_T(D^{*\pm}), $\eta(D^{*\pm}), W$ and $Q^2$ are compared with next-to-leading order QCD calculations based on the photon-gluon fusion production mechanism. After an extrapolation of the cross section to the full kinematic region in p_T(D^{*\pm}) and $\eta$(D^{*\pm}), the charm contribution $F_2^{c\bar{c}}(x,Q^2)$ to the proton structure function is determined for Bjorken $x$ between 2 $\cdot$ 10$^{-4}$ and 5 $\cdot$ 10$^{-3}$.Comment: 17 pages including 4 figure

Measurement of D*+/- meson production in jets from pp collisions at sqrt(s) = 7 TeV with the ATLAS detector

This paper reports a measurement of D*+/- meson production in jets from proton-proton collisions at a center-of-mass energy of sqrt(s) = 7 TeV at the CERN Large Hadron Collider. The measurement is based on a data sample recorded with the ATLAS detector with an integrated luminosity of 0.30 pb^-1 for jets with transverse momentum between 25 and 70 GeV in the pseudorapidity range |eta| < 2.5. D*+/- mesons found in jets are fully reconstructed in the decay chain: D*+ -> D0pi+, D0 -> K-pi+, and its charge conjugate. The production rate is found to be N(D*+/-)/N(jet) = 0.025 +/- 0.001(stat.) +/- 0.004(syst.) for D*+/- mesons that carry a fraction z of the jet momentum in the range 0.3 < z < 1. Monte Carlo predictions fail to describe the data at small values of z, and this is most marked at low jet transverse momentum.Comment: 10 pages plus author list (22 pages total), 5 figures, 1 table, matches published version in Physical Review

Physical and emotional nourishment: Food as the embodied component of loving care of elderly family relatives

Purpose This purpose of this study is to examine the fluidity of family life which continues to attract attention. This is increasingly significant for the intergenerational relationship between adult children and their elderly parents. Using practice theory, the aims are to understand the role of food in elderly families and explore how family practices are maintained when elderly transition into care. Design/methodology/approach A phenomenological research approach was used as the authors sought to build an understanding of the social interactions between family and their lifeworld. Findings This study extends theory on the relationship between the elderly parent and their family and explores through practice theory how families performed their love, how altered routines and long standing rituals provided structure to the elderly relatives and how care practices were negotiated as the elderly relatives transitioned from independence to dependence and towards care. A theoretical framework is introduced that provides guidance for the transition stages and the areas for negotiation. Research limitations/implications This research has implications for food manufacturers and marketers, as the demand for healthy food for the elderly is made more widely available, healthy and easy to prepare. The limitations of the research are due to the sample located in East Yorkshire only. Practical implications This research has implications for brand managers of food manufacturers and supermarkets that need to create product lines that target this segment by producing healthy, convenience food. Social implications It is also important for health and social care policy as the authors seek to understand the role of food, family and community and how policy can be devised to provide stability in this transitional and uncertain lifestage. Originality/value This research extends the body of literature on food and the family by focussing on the elderly cared for and their family. The authors show how food can be construed as loving care, and using practice theory, a theoretical framework is developed that can explain the transitions and how the family negotiates the stages from independence to dependence

Physical and emotional nourishment: Food as the embodied component of loving care of elderly family relatives

Purpose This purpose of this study is to examine the fluidity of family life which continues to attract attention. This is increasingly significant for the intergenerational relationship between adult children and their elderly parents. Using practice theory, the aims are to understand the role of food in elderly families and explore how family practices are maintained when elderly transition into care. Design/methodology/approach A phenomenological research approach was used as the authors sought to build an understanding of the social interactions between family and their lifeworld. Findings This study extends theory on the relationship between the elderly parent and their family and explores through practice theory how families performed their love, how altered routines and long standing rituals provided structure to the elderly relatives and how care practices were negotiated as the elderly relatives transitioned from independence to dependence and towards care. A theoretical framework is introduced that provides guidance for the transition stages and the areas for negotiation. Research limitations/implications This research has implications for food manufacturers and marketers, as the demand for healthy food for the elderly is made more widely available, healthy and easy to prepare. The limitations of the research are due to the sample located in East Yorkshire only. Practical implications This research has implications for brand managers of food manufacturers and supermarkets that need to create product lines that target this segment by producing healthy, convenience food. Social implications It is also important for health and social care policy as the authors seek to understand the role of food, family and community and how policy can be devised to provide stability in this transitional and uncertain lifestage. Originality/value This research extends the body of literature on food and the family by focussing on the elderly cared for and their family. The authors show how food can be construed as loving care, and using practice theory, a theoretical framework is developed that can explain the transitions and how the family negotiates the stages from independence to dependence