Sample Efficient Deep Reinforcement Learning for Dialogue Systems with Large Action Spaces

In spoken dialogue systems, we aim to deploy artificial intelligence to build automated dialogue agents that can converse with humans. A part of this effort is the policy optimisation task, which attempts to find a policy describing how to respond to humans, in the form of a function taking the current state of the dialogue and returning the response of the system. In this paper, we investigate deep reinforcement learning approaches to solve this problem. Particular attention is given to actor-critic methods, off-policy reinforcement learning with experience replay, and various methods aimed at reducing the bias and variance of estimators. When combined, these methods result in the previously proposed ACER algorithm that gave competitive results in gaming environments. These environments however are fully observable and have a relatively small action set so in this paper we examine the application of ACER to dialogue policy optimisation. We show that this method beats the current state-of-the-art in deep learning approaches for spoken dialogue systems. This not only leads to a more sample efficient algorithm that can train faster, but also allows us to apply the algorithm in more difficult environments than before. We thus experiment with learning in a very large action space, which has two orders of magnitude more actions than previously considered. We find that ACER trains significantly faster than the current state-of-the-art.Toshiba Research Europe Ltd, Cambridge Research Laboratory - RG85875 EPSRC Research Council - RG8079

Growing Older in Hong Kong, New York and London

published_or_final_versio

Asymptotic scaling of the gluon propagtor on the lattice

We pursue the study of the high energy behaviour of the gluon propagator on the lattice in the Landau gauge in the flavorless case (n_f=0). It was shown in a precedin g paper that the gluon propagator did not reach three-loop asymptotic scaling at an energy scale as high as 5 GeV. Our present high statistics analysis includes also a simulation at $\beta=6.8$ ($a\simeq 0.03$ fm), which allows to reach $\mu \simeq 10$ GeV. Special care has been devoted to the finite lattice-spacing artifacts as well as to the finite volume effects, the latter being acute at $\beta=6.8$ where the volume is bounded by technical limits. Our main conclusion is a strong evidence that the gluon propagator has reached three-loop asymptotic scaling, at $\mu$ ranging from 5.6 GeV to 9.5 GeV. We buttress up this conclusion on several demanding criteria of asymptoticity, including scheme independence. Our fit in the 5.6 GeV to 9.5 GeV window yields $\Lambda^{\bar{{\rm MS}}} = 319 \pm 14 ^{+10}_{-20}$ MeV, in good agreement with our previous result, $\Lambda^{\bar{{\rm MS}}} = 295 \pm 20$ MeV, obtained from the three gluon vertex, but it is significantly above the Schr\"odinger functional method estimate : $238 \pm 19$ MeV. The latter difference is not understood. Confirming our previous paper, we show that a fourth loop is necessary to fit the whole ($2.8 \div 9.5$) GeV energy window.Comment: latex-file, 19 pgs., 6 fig

A tale of three world cities: health care policy implications

Conference Theme: Managing World Citiespublished_or_final_versio

Pair formation in two electron correlated chains

We study two correlated electrons in a nearest neighbour tight- binding chain, with both on site and nearest neighbour interaction. Both the cases of parallel and antiparallel spins are considered. In addition to the free electron band for two electrons, there are correlated bands with positive or negative energy, depending on wheather the interaction parameters are repulsive or attractive. Electrons form bound states, with amplitudes that decay exponentially with separation. Conditions for such states to be filled at low temperatures are discussed.Comment: To appear in J. Phys: Condens. Matter 15 (2003

Revisiting soliton contributions to perturbative amplitudes

Open Access funded by SCOAP3. CP is a Royal Society Research Fellow and partly supported by the U.S. Department of Energy under grants DOE-SC0010008, DOE-ARRA-SC0003883 and DOE-DE-SC0007897. ABR is supported by the Mitchell Family Foundation. We would like to thank the Mitchell Institute at Texas A&M and the NHETC at Rutgers University respectively for hospitality during the course of this work. We would also like to acknowledge the Aspen Center for Physics and NSF grant 1066293 for a stimulating research environment

Renormalization group improved action on anisotropic lattices

We study a block spin transformation in the SU(3) lattice gauge theory on anisotropic lattices to obtain Iwasaki's renormalization group improved action for anisotropic cases. For the class of actions with plaquette and $1\times2$ rectangular terms, we determine the improvement parameters as functions of the anisotropy $\xi= a_s/a_t$. We find that the program of improvement works well also on anisotropic lattices. From a study of an indicator which estimates the distance to the renormalized trajectory, we show that, for the range of the anisotropy $\xi \approx 1$--4, the coupling parameters previously determined for isotropic lattices improve the theory considerably.Comment: 15 pages, 10 figure

Age-related changes in neural functional connectivity and its behavioral relevance

<p>Abstract</p> <p>Background</p> <p>Resting-state recordings are characterized by widely distributed networks of coherent brain activations. Disturbances of the default network - a set of regions that are deactivated by cognitive tasks and activated during passive states - have been detected in age-related disorders such as Alzheimer's or Parkinson's disease but alterations in the course of healthy aging still need to be explored.</p> <p>Results</p> <p>Using magnetoencephalography (MEG), the present study investigated how age-related functional resting-state brain connectivity links to cognitive performance in healthy aging in fifty-three participants ranging in age from 18 to 89 years. A beamforming technique was used to reconstruct the brain activity in source space and the interregional coupling was investigated using partial directed coherence (PDC). We found significant age-related alterations of functional resting-state connectivity. These are mainly characterized by reduced information input into the posterior cingulum/precuneus region together with an enhanced information flow to the medial temporal lobe. Furthermore, higher inflow in the medial temporal lobe subsystem was associated with weaker cognitive performance whereas stronger inflow in the posterior cluster was related to better cognitive performance.</p> <p>Conclusion</p> <p>This is the first study to show age-related alterations in subsystems of the resting state network that are furthermore associated with cognitive performance.</p

Histone H1 Subtypes Differentially Modulate Chromatin Condensation without Preventing ATP-Dependent Remodeling by SWI/SNF or NURF

Although ubiquitously present in chromatin, the function of the linker histone subtypes is partly unknown and contradictory studies on their properties have been published. To explore whether the various H1 subtypes have a differential role in the organization and dynamics of chromatin we have incorporated all of the somatic human H1 subtypes into minichromosomes and compared their influence on nucleosome spacing, chromatin compaction and ATP-dependent remodeling. H1 subtypes exhibit different affinities for chromatin and different abilities to promote chromatin condensation, as studied with the Atomic Force Microscope. According to this criterion, H1 subtypes can be classified as weak condensers (H1.1 and H1.2), intermediate condensers (H1.3) and strong condensers (H1.0, H1.4, H1.5 and H1x). The variable C-terminal domain is required for nucleosome spacing by H1.4 and is likely responsible for the chromatin condensation properties of the various subtypes, as shown using chimeras between H1.4 and H1.2. In contrast to previous reports with isolated nucleosomes or linear nucleosomal arrays, linker histones at a ratio of one per nucleosome do not preclude remodeling of minichromosomes by yeast SWI/SNF or Drosophila NURF. We hypothesize that the linker histone subtypes are differential organizers of chromatin, rather than general repressors