Lenient multi-agent deep reinforcement learning

Much of the success of single agent deep reinforcement learning (DRL) in recent years can be attributed to the use of experience replay memories (ERM), which allow Deep Q-Networks (DQNs) to be trained efficiently through sampling stored state transitions. However, care is required when using ERMs for multi-agent deep reinforcement learning (MA-DRL), as stored transitions can become outdated because agents update their policies in parallel [11]. In this work we apply leniency [23] to MA-DRL. Lenient agents map state-action pairs to decaying temperature values that control the amount of leniency applied towards negative policy updates that are sampled from the ERM. This introduces optimism in the value-function update, and has been shown to facilitate cooperation in tabular fully-cooperative multi-agent reinforcement learning problems. We evaluate our Lenient-DQN (LDQN) empirically against the related Hysteretic-DQN (HDQN) algorithm [22] as well as a modified version we call scheduled-HDQN, that uses average reward learning near terminal states. Evaluations take place in extended variations of the Coordinated Multi-Agent Object Transportation Problem (CMOTP) [8] which include fully-cooperative sub-tasks and stochastic rewards. We find that LDQN agents are more likely to converge to the optimal policy in a stochastic reward CMOTP compared to standard and scheduled-HDQN agents

Reconstructing CMEs with Coordinated Imaging and In Situ Observations: Global Structure, Kinematics, and Implications for Space Weather Forecasting

See the pdf for detailsComment: 45 pages, 16 figures, ApJ, in pres

The Longitudinal Properties of a Solar Energetic Particle Event Investigated Using Modern Solar Imaging

We use combined high-cadence, high-resolution, and multi-point imaging by the Solar-Terrestrial Relations Observatory (STEREO) and the Solar and Heliospheric Observatory to investigate the hour-long eruption of a fast and wide coronal mass ejection (CME) on 2011 March 21 when the twin STEREO spacecraft were located beyond the solar limbs. We analyze the relation between the eruption of the CME, the evolution of an Extreme Ultraviolet (EUV) wave, and the onset of a solar energetic particle (SEP) event measured in situ by the STEREO and near-Earth orbiting spacecraft. Combined ultraviolet and white-light images of the lower corona reveal that in an initial CME lateral "expansion phase," the EUV disturbance tracks the laterally expanding flanks of the CME, both moving parallel to the solar surface with speeds of ~450 km s^(–1). When the lateral expansion of the ejecta ceases, the EUV disturbance carries on propagating parallel to the solar surface but devolves rapidly into a less coherent structure. Multi-point tracking of the CME leading edge and the effects of the launched compression waves (e.g., pushed streamers) give anti-sunward speeds that initially exceed 900 km s^(–1) at all measured position angles. We combine our analysis of ultraviolet and white-light images with a comprehensive study of the velocity dispersion of energetic particles measured in situ by particle detectors located at STEREO-A (STA) and first Lagrange point (L1), to demonstrate that the delayed solar particle release times at STA and L1 are consistent with the time required (30-40 minutes) for the CME to perturb the corona over a wide range of longitudes. This study finds an association between the longitudinal extent of the perturbed corona (in EUV and white light) and the longitudinal extent of the SEP event in the heliosphere

A Plasma {\beta} Transition Within a Propagating Flux Rope

We present a 2.5D MHD simulation of a magnetic flux rope (FR) propagating in the heliosphere and investigate the cause of the observed sharp plasma beta transition. Specifically, we consider a strong internal magnetic field and an explosive fast start, such that the plasma beta is significantly lower in the FR than the sheath region that is formed ahead. This leads to an unusual FR morphology in the first stage of propagation, while the more traditional view (e.g. from space weather simulations like Enlil) of a `pancake' shaped FR is observed as it approaches 1 AU. We investigate how an equipartition line, defined by a magnetic Weber number, surrounding a core region of a propagating FR can demarcate a boundary layer where there is a sharp transition in the plasma beta. The substructure affects the distribution of toroidal flux, with the majority of the flux remaining in a small core region which maintains a quasi-cylindrical structure. Quantitatively, we investigate a locus of points where the kinetic energy density of the relative inflow field is equal to the energy density of the transverse magnetic field (i.e. effective tension force). The simulation provides compelling evidence that at all heliocentric distances the distribution of toroidal magnetic flux away from the FR axis is not linear; with 80% of the toroidal flux occurring within 40% of the distance from the FR axis. Thus our simulation displays evidence that the competing ideas of a pancaking structure observed remotely can coexist with a quasi-cylindrical magnetic structure seen in situ.Comment: 11 pages of text + 6 figures. Accepted to ApJ on 16 Oct 201

Traumatic brain injury leads to alterations in contusional cortical miRNAs involved in dementia

There is compelling evidence that head injury is a significant environmental risk factor for Alzheimer's disease (AD) and that a history of traumatic brain injury (TBI) accelerates the onset of AD. Amyloid-β plaques and tau aggregates have been observed in the post-mortem brains of TBI patients; however, the mechanisms leading to AD neuropathology in TBI are still unknown. In this study, we hypothesized that focal TBI induces changes in miRNA expression in and around affected areas, resulting in the altered expression of genes involved in neurodegeneration and AD pathology. For this purpose, we performed a miRNA array in extracts from rats subjected to experimental TBI, using the controlled cortical impact (CCI) model. In and around the contusion, we observed alterations of miRNAs associated with dementia/AD, compared to the contralateral side. Specifically, the expression of miR-9 was significantly upregulated, while miR-29b, miR-34a, miR-106b, miR-181a and miR-107 were downregulated. Via qPCR, we confirmed these results in an additional group of injured rats when compared to naïve animals. Interestingly, the changes in those miRNAs were concomitant with alterations in the gene expression of mRNAs involved in amyloid generation and tau pathology, such as β-APP cleaving enzyme (BACE1) and Glycogen synthase-3-β (GSK3β). In addition increased levels of neuroinflammatory markers (TNF-α), glial activation, neuronal loss, and tau phosphorylation were observed in pericontusional areas. Therefore, our results suggest that the secondary injury cascade in TBI affects miRNAs regulating the expression of genes involved in AD dementia

GANGs: Generative Adversarial Network Games

Generative Adversarial Networks (GAN) have become one of the most successful frameworks for unsupervised generative modeling. As GANs are difficult to train much research has focused on this. However, very little of this research has directly exploited game-theoretic techniques. We introduce Generative Adversarial Network Games (GANGs), which explicitly model a finite zero-sum game between a generator ($G$) and classifier ($C$) that use mixed strategies. The size of these games precludes exact solution methods, therefore we define resource-bounded best responses (RBBRs), and a resource-bounded Nash Equilibrium (RB-NE) as a pair of mixed strategies such that neither $G$ or $C$ can find a better RBBR. The RB-NE solution concept is richer than the notion of `local Nash equilibria' in that it captures not only failures of escaping local optima of gradient descent, but applies to any approximate best response computations, including methods with random restarts. To validate our approach, we solve GANGs with the Parallel Nash Memory algorithm, which provably monotonically converges to an RB-NE. We compare our results to standard GAN setups, and demonstrate that our method deals well with typical GAN problems such as mode collapse, partial mode coverage and forgetting

LiftUpp: Support to Develop Learner Performance

Various motivations exist to move away from the simple assessment of knowledge towards the more complex assessment and development of competence. However, to accommodate such a change, high demands are put on the supporting e-infrastructure in terms of intelligently collecting and analysing data. In this paper, we discuss these challenges and how they are being addressed by LiftUpp, a system that is now used in 70% of UK dental schools, and is finding wider applications in physiotherapy, medicine and veterinary science. We describe how data is collected for workplace-based development in dentistry using a dedicated iPad app, which enables an integrated approach to linking and assessing work flows, skills and learning outcomes. Furthermore, we detail how the various forms of collected data can be fused, visualized and integrated with conventional forms of assessment. This enables curriculum integration, improved real-time student feedback, support for administration, and informed instructional planning. Together these facets contribute to better support for the development of learners' competence in situated learning setting, as well as an improved experience. Finally, we discuss several directions for future research on intelligent teaching systems that are afforded by using the design present within LiftUpp.Comment: Short 4-page version to appear at AIED 201

Recommendations for Next‐Generation Ground Magnetic Perturbation Validation

Data‐model validation of ground magnetic perturbation forecasts, specifically of the time rate of change of surface magnetic field, dB/dt, is a critical task for model development and for mitigation of geomagnetically induced current effects. While a current, community‐accepted standard for dB/dt validation exists (Pulkkinen et al., 2013), it has several limitations that prevent more complete understanding of model capability. This work presents recommendations from the International Forum for Space Weather Capabilities Assessment Ground Magnetic Perturbation Working Team for creating a next‐generation validation suite. Four recommendations are made to address the existing suite: greatly expand the number of ground observatories used, expand the number of events included in the suite from six to eight, generate metrics as a function of magnetic local time, and generate metrics as a function of activity type. For each of these, implementation details are explored. Limitations and future considerations are also discussed.Plain Language SummarySpace weather forecast models of magnetic field perturbations are important for protecting the power grid and other vulnerable infrastructure. These models must be validated by comparing their predictions to observations. This paper makes recommendations for how future models should be validated in order to best test their capabilities.Key PointsWe present a new validation suite for models of ground magnetic perturbations, dB/dt, of interest for geomagnetically induced currentsThe existing standard remains useful but provides limited information, so an expanded set of metrics is defined hereThis work is a result of the International Forum for Space Weather Capabilities Assessment and represents a new community consensusPeer Reviewedhttps://deepblue.lib.umich.edu/bitstream/2027.42/147786/1/swe20777_am.pdfhttps://deepblue.lib.umich.edu/bitstream/2027.42/147786/2/swe20777.pd

Intermittent release of transients in the slow solar wind: 2. In situ evidence

In paper 1, we showed that the Heliospheric Imager (HI) instruments on the pair of NASA STEREO spacecraft can be used to image the streamer belt and, in particular, the variability of the slow solar wind which originates near helmet streamers. The observation of intense intermittent transient outflow by HI implies that the corresponding in situ observations of the slow solar wind and corotating interaction regions (CIRs) should contain many signatures of transients. In the present paper, we compare the HI observations with in situ measurements from the STEREO and ACE spacecraft. Analysis of the solar wind ion, magnetic field, and suprathermal electron flux measurements from the STEREO spacecraft reveals the presence of both closed and partially disconnected interplanetary magnetic field lines permeating the slow solar wind. We predict that one of the transients embedded within the second CIR (CIR‐D in paper 1) should impact the near‐Earth ACE spacecraft. ACE measurements confirm the presence of a transient at the time of CIR passage; the transient signature includes helical magnetic fields and bidirectional suprathermal electrons. On the same day, a strahl electron dropout is observed at STEREO‐B, correlated with the passage of a high plasma beta structure. Unlike ACE, STEREO‐B observes the transient a few hours ahead of the CIR. STEREO‐A, STEREO‐B, and ACE spacecraft observe very different slow solar wind properties ahead of and during the CIR analyzed in this paper, which we associate with the intermittent release of transients