Reinforcement Learning: A Survey

This paper surveys the field of reinforcement learning from a computer-science perspective. It is written to be accessible to researchers familiar with machine learning. Both the historical basis of the field and a broad selection of current work are summarized. Reinforcement learning is the problem faced by an agent that learns behavior through trial-and-error interactions with a dynamic environment. The work described here has a resemblance to work in psychology, but differs considerably in the details and in the use of the word ``reinforcement.'' The paper discusses central issues of reinforcement learning, including trading off exploration and exploitation, establishing the foundations of the field via Markov decision theory, learning from delayed reinforcement, constructing empirical models to accelerate learning, making use of generalization and hierarchy, and coping with hidden state. It concludes with a survey of some implemented systems and an assessment of the practical utility of current methods for reinforcement learning.Comment: See http://www.jair.org/ for any accompanying file

Targeting B-Raf inhibitor resistant melanoma with novel cell penetrating peptide disrupters of PDE8A – C-Raf

Background: Recent advances in the treatment of melanoma that involve immunotherapy and B-Raf inhibition have revolutionised cancer care for this disease. However, an un-met clinical need remains in B-Raf inhibitor resistant patients where first-generation B-Raf inhibitors provide only short-term disease control. In these cases, B-Raf inhibition leads to paradoxical activation of the C-Raf – MEK – ERK signalling pathway, followed by metastasis. PDE8A has been shown to directly interact with and modulate the cAMP microdomain in the vicinity of C-Raf. This interaction promotes C-Raf activation by attenuating the PKA-mediated inhibitory phosphorylation of the kinase. Methods: We have used a novel cell-penetrating peptide agent (PPL-008) that inhibits the PDE8A – C-Raf complex in a human malignant MM415 melanoma cell line and MM415 melanoma xenograft mouse model to investigate ERK MAP kinase signalling. Results: We have demonstrated that the PDE8A – C-Raf complex disruptor PPL-008 increased inhibitory C-Raf-S259 phosphorylation and significantly reduced phospho-ERK signalling. We have also discovered that the ability of PPL-008 to dampen ERK signalling can be used to counter B-Raf inhibitor-driven paradoxical activation of phospho-ERK in MM415 cells treated with PLX4032 (Vemurafenib). PPL-008 treatment also significantly retarded the growth of these cells. When applied to a MM415 melanoma xenograft mouse model, PPL-008C penetrated tumour tissue and significantly reduced phospho-ERK signalling in that domain. Conclusion: Our data suggests that the PDE8A-C-Raf complex is a promising therapeutic treatment for B-Raf inhibitor resistant melanoma

On the discrete spectrum of spin-orbit Hamiltonians with singular interactions

We give a variational proof of the existence of infinitely many bound states below the continuous spectrum for spin-orbit Hamiltonians (including the Rashba and Dresselhaus cases) perturbed by measure potentials thus extending the results of J.Bruening, V.Geyler, K.Pankrashkin: J. Phys. A 40 (2007) F113--F117.Comment: 10 pages; to appear in Russian Journal of Mathematical Physics (memorial volume in honor of Vladimir Geyler). Results improved in this versio

Next nearest neighbour Ising models on random graphs

This paper develops results for the next nearest neighbour Ising model on random graphs. Besides being an essential ingredient in classic models for frustrated systems, second neighbour interactions interactions arise naturally in several applications such as the colour diversity problem and graphical games. We demonstrate ensembles of random graphs, including regular connectivity graphs, that have a periodic variation of free energy, with either the ratio of nearest to next nearest couplings, or the mean number of nearest neighbours. When the coupling ratio is integer paramagnetic phases can be found at zero temperature. This is shown to be related to the locked or unlocked nature of the interactions. For anti-ferromagnetic couplings, spin glass phases are demonstrated at low temperature. The interaction structure is formulated as a factor graph, the solution on a tree is developed. The replica symmetric and energetic one-step replica symmetry breaking solution is developed using the cavity method. We calculate within these frameworks the phase diagram and demonstrate the existence of dynamical transitions at zero temperature for cases of anti-ferromagnetic coupling on regular and inhomogeneous random graphs.Comment: 55 pages, 15 figures, version 2 with minor revisions, to be published J. Stat. Mec

Vortex Rings in Fast Rotating Bose-Einstein Condensates

When Bose-Eintein condensates are rotated sufficiently fast, a giant vortex phase appears, that is the condensate becomes annular with no vortices in the bulk but a macroscopic phase circulation around the central hole. In a former paper [M. Correggi, N. Rougerie, J. Yngvason, {\it arXiv:1005.0686}] we have studied this phenomenon by minimizing the two dimensional Gross-Pitaevskii energy on the unit disc. In particular we computed an upper bound to the critical speed for the transition to the giant vortex phase. In this paper we confirm that this upper bound is optimal by proving that if the rotation speed is taken slightly below the threshold there are vortices in the condensate. We prove that they gather along a particular circle on which they are evenly distributed. This is done by providing new upper and lower bounds to the GP energy.Comment: to appear in Archive of Rational Mechanics and Analysi

Green's functions for parabolic systems of second order in time-varying domains

We construct Green's functions for divergence form, second order parabolic systems in non-smooth time-varying domains whose boundaries are locally represented as graph of functions that are Lipschitz continuous in the spatial variables and 1/2-H\"older continuous in the time variable, under the assumption that weak solutions of the system satisfy an interior H\"older continuity estimate. We also derive global pointwise estimates for Green's function in such time-varying domains under the assumption that weak solutions of the system vanishing on a portion of the boundary satisfy a certain local boundedness estimate and a local H\"older continuity estimate. In particular, our results apply to complex perturbations of a single real equation.Comment: 25 pages, 0 figur

Atomic Resonance and Scattering

Contains research objectives, summary of research and reports on four research projects.Joint Services Electronics Programs (U. S. Army, U. S. Navy, and U. S. Air Force) under Contract DAAB07-71-C-0300National Science Foundation (Grant GP-28679)National Bureau of Standards (Grant NBS2-9011)U. S. Air Force - Office of Scientific Research (Contract F44620-72-C-0057

A limit model for thermoelectric equations

We analyze the asymptotic behavior corresponding to the arbitrary high conductivity of the heat in the thermoelectric devices. This work deals with a steady-state multidimensional thermistor problem, considering the Joule effect and both spatial and temperature dependent transport coefficients under some real boundary conditions in accordance with the Seebeck-Peltier-Thomson cross-effects. Our first purpose is that the existence of a weak solution holds true under minimal assumptions on the data, as in particular nonsmooth domains. Two existence results are studied under different assumptions on the electrical conductivity. Their proofs are based on a fixed point argument, compactness methods, and existence and regularity theory for elliptic scalar equations. The second purpose is to show the existence of a limit model illustrating the asymptotic situation.Comment: 20 page

The mixed problem for the Laplacian in Lipschitz domains

We consider the mixed boundary value problem or Zaremba's problem for the Laplacian in a bounded Lipschitz domain in R^n. We specify Dirichlet data on part of the boundary and Neumann data on the remainder of the boundary. We assume that the boundary between the sets where we specify Dirichlet and Neumann data is a Lipschitz surface. We require that the Neumann data is in L^p and the Dirichlet data is in the Sobolev space of functions having one derivative in L^p for some p near 1. Under these conditions, there is a unique solution to the mixed problem with the non-tangential maximal function of the gradient of the solution in L^p of the boundary. We also obtain results with data from Hardy spaces when p=1.Comment: Version 5 includes a correction to one step of the main proof. Since the paper appeared long ago, this submission includes the complete paper, followed by a short section that gives the correction to one step in the proo

Anti-HIV-1 activity of cellulose acetate phthalate: Synergy with soluble CD4 and induction of "dead-end" gp41 six-helix bundles

BACKGROUND: Cellulose acetate phthalate (CAP), a promising candidate microbicide for prevention of sexual transmission of the human immunodeficiency virus type 1 (HIV-1) and other sexually transmitted disease (STD) pathogens, was shown to inactivate HIV-1 and to block the coreceptor binding site on the virus envelope glycoprotein gp120. It did not interfere with virus binding to CD4. Since CD4 is the primary cellular receptor for HIV-1, it was of interest to study CAP binding to HIV-1 complexes with soluble CD4 (sCD4) and its consequences, including changes in the conformation of the envelope glycoprotein gp41 within virus particles. METHODS: Enzyme-linked immunosorbent assays (ELISA) were used to study CAP binding to HIV-1-sCD4 complexes and to detect gp41 six-helix bundles accessible on virus particles using antibodies specific for the α-helical core domain of gp41. RESULTS: 1) Pretreatment of HIV-1 with sCD4 augments subsequent binding of CAP; 2) there is synergism between CAP and sCD4 for inhibition of HIV-1 infection; 3) treatment of HIV-1 with CAP induced the formation of gp41 six-helix bundles. CONCLUSIONS: CAP and sCD4 bind to distinct sites on HIV-1 IIIB and BaL virions and their simultaneous binding has profound effects on virus structure and infectivity. The formation of gp41 six-helical bundles, induced by CAP, is known to render the virus incompetent for fusion with target cells thus preventing infection