The Dreaming Variational Autoencoder for Reinforcement Learning Environments

Reinforcement learning has shown great potential in generalizing over raw sensory data using only a single neural network for value optimization. There are several challenges in the current state-of-the-art reinforcement learning algorithms that prevent them from converging towards the global optima. It is likely that the solution to these problems lies in short- and long-term planning, exploration and memory management for reinforcement learning algorithms. Games are often used to benchmark reinforcement learning algorithms as they provide a flexible, reproducible, and easy to control environment. Regardless, few games feature a state-space where results in exploration, memory, and planning are easily perceived. This paper presents The Dreaming Variational Autoencoder (DVAE), a neural network based generative modeling architecture for exploration in environments with sparse feedback. We further present Deep Maze, a novel and flexible maze engine that challenges DVAE in partial and fully-observable state-spaces, long-horizon tasks, and deterministic and stochastic problems. We show initial findings and encourage further work in reinforcement learning driven by generative exploration.Comment: Best Student Paper Award, Proceedings of the 38th SGAI International Conference on Artificial Intelligence, Cambridge, UK, 2018, Artificial Intelligence XXXV, 201

Simple threshold rules solve explore/exploit trade‐offs in a resource accumulation search task

How, and how well, do people switch between exploration and exploitation to search for and accumulate resources? We study the decision processes underlying such exploration/exploitation trade‐offs using a novel card selection task that captures the common situation of searching among multiple resources (e.g., jobs) that can be exploited without depleting. With experience, participants learn to switch appropriately between exploration and exploitation and approach optimal performance. We model participants' behavior on this task with random, threshold, and sampling strategies, and find that a linear decreasing threshold rule best fits participants' results. Further evidence that participants use decreasing threshold‐based strategies comes from reaction time differences between exploration and exploitation; however, participants themselves report non‐decreasing thresholds. Decreasing threshold strategies that “front‐load” exploration and switch quickly to exploitation are particularly effective in resource accumulation tasks, in contrast to optimal stopping problems like the Secretary Problem requiring longer exploration

Evolution of the Pairwise Peculiar Velocity Distribution Function in Lagrangian Perturbation Theory

The statistical distribution of the radial pairwise peculiar velocity of galaxies is known to have an exponential form as implied by observations and explicitly shown in N-body simulations. Here we calculate its statistical distribution function using the Zel'dovich approximation assuming that the primordial density fluctuations are Gaussian distributed. We show that the exponential distribution is realized as a transient phenomena on megaparsec scales in the standard cold-dark-matter model.Comment: 19 pages, 8 Postscript figures, AAS LaTe

The Effects of Electrical Degradation on the Microstructure of Metal Oxide Varistor

In this paper, the findings from systematic experiments with the purpose to determine the degradation effect of single and multiple current pulses on the microstructure of the Metal Oxide (MO) varistor are described. Six distribution class varistors from one manufacturer were used in these experiments. The first part of the paper describes the electrical condition after application of single and multiple lightning current pulses. The results (before and after each elevated current impulse test) of 1mA AC reference voltage and residual voltage are presented in this section. We have also investigated a new technique called ‘Return Voltage Measurement’ for monitoring of the degradation in MO varistors. This is described in detail. The second part deals with the microstructure observations of MO varistors. The results of microstructural examination of impulse current on MO varsitors are examined in detail. This section also explains the relationship between the microstructural changes and electrical degradation of MO varistors

Temperature dependence of surface reconstructions of Au on Pd(110)

Surface reconstructions of Au film on Pd(110) substrate are studied using a local Einstein approximation to quasiharmonic theory with the Sutton-Chen interatomic potential. Temperature dependent surface free energies for different coverages and surface structures are calculated. Experimentally observed transformations from $(1\times1)$ to $(1 \times 2)$ and $(1 \times 3)$ structures can be explained in the framework of this model. Also conditions for Stranski-Krastanov growth mode are found to comply with experiments. The domain of validity of the model neglecting mixing entropy is analyzed.Comment: 7 pages, REVTeX two-column format, 3 postscript figures available on request from [email protected] To appear in Phys. Rev. Letter

Damping of Electron Density Structures and Implications for Interstellar Scintillation

The forms of electron density structures in kinetic Alfven wave turbulence are studied in connection with scintillation. The focus is on small scales $L \sim 10^8-10^{10}$ cm where the Kinetic Alfv\'en wave (KAW) regime is active in the interstellar medium. MHD turbulence converts to a KAW cascade, starting at 10 times the ion gyroradius and continuing to smaller scales. These scales are inferred to dominate scintillation in the theory of Boldyrev et al. From numerical solutions of a decaying kinetic Alfv\'en wave turbulence model, structure morphology reveals two types of localized structures, filaments and sheets, and shows that they arise in different regimes of resistive and diffusive damping. Minimal resistive damping yields localized current filaments that form out of Gaussian-distributed initial conditions. When resistive damping is large relative to diffusive damping, sheet-like structures form. In the filamentary regime, each filament is associated with a non-localized magnetic and density structure, circularly symmetric in cross section. Density and magnetic fields have Gaussian statistics (as inferred from Gaussian-valued kurtosis) while density gradients are strongly non-Gaussian, more so than current. This enhancement of non-Gaussian statistics in a derivative field is expected since gradient operations enhance small-scale fluctuations. The enhancement of density gradient kurtosis over current kurtosis is not obvious, yet it suggests that modest fluctuation levels in electron density may yield large scintillation events during pulsar signal propagation in the interstellar medium. In the sheet regime the same statistical observations hold, despite the absence of localized filamentary structures. Probability density functions are constructed from statistical ensembles in both regimes, showing clear formation of long, highly non-Gaussian tails

Estimating the cost-effectiveness of detecting cases of chronic hepatitis C infection on reception into prison

Background In England and Wales where less than 1% of the population are Injecting drug users (IDUs), 97% of HCV reports are attributed to injecting drug use. As over 60% of the IDU population will have been imprisoned by the age of 30 years, prison may provide a good location in which to offer HCV screening and treatment. The aim of this work is to examine the cost effectiveness of a number of alternative HCV case-finding strategies on prison reception Methods A decision analysis model embedded in a model of the flow of IDUs through prison was used to estimate the cost effectiveness of a number of alternative case-finding strategies. The model estimates the average cost of identifying a new case of HCV from the perspective of the health care provider and how these estimates may evolve over time. Results The results suggest that administering verbal screening for a past positive HCV test and for ever having engaged in illicit drug use prior to the administering of ELISA and PCR tests can have a significant impact on the cost effectiveness of HCV case-finding strategies on prison reception; the discounted cost in 2017 being £2,102 per new HCV case detected compared to £3,107 when no verbal screening is employed. Conclusion The work here demonstrates the importance of targeting those individuals that have ever engaged in illicit drug use for HCV testing in prisons, these individuals can then be targeted for future intervention measures such as treatment or monitored to prevent future transmission

Large-Scale Image Processing with the ROTSE Pipeline for Follow-Up of Gravitational Wave Events

Electromagnetic (EM) observations of gravitational-wave (GW) sources would bring unique insights into a source which are not available from either channel alone. However EM follow-up of GW events presents new challenges. GW events will have large sky error regions, on the order of 10-100 square degrees, which can be made up of many disjoint patches. When searching such large areas there is potential contamination by EM transients unrelated to the GW event. Furthermore, the characteristics of possible EM counterparts to GW events are also uncertain. It is therefore desirable to be able to assess the statistical significance of a candidate EM counterpart, which can only be done by performing background studies of large data sets. Current image processing pipelines such as that used by ROTSE are not usually optimised for large-scale processing. We have automated the ROTSE image analysis, and supplemented it with a post-processing unit for candidate validation and classification. We also propose a simple ad hoc statistic for ranking candidates as more likely to be associated with the GW trigger. We demonstrate the performance of the automated pipeline and ranking statistic using archival ROTSE data. EM candidates from a randomly selected set of images are compared to a background estimated from the analysis of 102 additional sets of archival images. The pipeline's detection efficiency is computed empirically by re-analysis of the images after adding simulated optical transients that follow typical light curves for gamma-ray burst afterglows and kilonovae. We show that the automated pipeline rejects most background events and is sensitive to simulated transients to limiting magnitudes consistent with the limiting magnitude of the images

Investigation of Diagnostic Techniques for Metal Oxide Surge Arresters

Gapless metal oxide surge arresters(MOSA)have been available in the market for many years since they were first introduced in the 1970's. The aim of this study is to investigate some reliable diagnostic techniques to assess the condition of a metal oxide surge arrester when subjected to severe lightning strikes in the field. A number of non-destructive and destructive diagnostic techniques for Metal Oxide Surge Arrester (MOSA) are discussed in this paper. The non-destructive techniques include the standard 1 mA reference voltage, lightning impulse discharge residual voltage and a number of modern diagnostics based on polarization methods: Return voltage and polarization/depolarization current measurements. In order to observe, analyze and correctly explain the degradation phenomena, a number of destructive techniques based on microstructure observation are also conducted. The techniques include optical microscopy, scanning electron microscopy, X-Ray diffraction and energy dispersive spectrometry. The single and multi-pulse currents of 8r20 s wave shape were used to artificially degrade the MOSA. The before and after diagnostic results of the non-destructive and destructive techniques are presented and interpreted to understand the aging mechanism in MOSA. The importance of modern non-destructive electrical diagnostics based on polarization methods is validated by test results and is highlighted in detail in this paper. Finally the correlation of the results of different diagnostic techniques with each other and with the results of standard techniques is discussed