Learning with Opponent-Learning Awareness

Multi-agent settings are quickly gathering importance in machine learning. This includes a plethora of recent work on deep multi-agent reinforcement learning, but also can be extended to hierarchical RL, generative adversarial networks and decentralised optimisation. In all these settings the presence of multiple learning agents renders the training problem non-stationary and often leads to unstable training or undesired final results. We present Learning with Opponent-Learning Awareness (LOLA), a method in which each agent shapes the anticipated learning of the other agents in the environment. The LOLA learning rule includes a term that accounts for the impact of one agent's policy on the anticipated parameter update of the other agents. Results show that the encounter of two LOLA agents leads to the emergence of tit-for-tat and therefore cooperation in the iterated prisoners' dilemma, while independent learning does not. In this domain, LOLA also receives higher payouts compared to a naive learner, and is robust against exploitation by higher order gradient-based methods. Applied to repeated matching pennies, LOLA agents converge to the Nash equilibrium. In a round robin tournament we show that LOLA agents successfully shape the learning of a range of multi-agent learning algorithms from literature, resulting in the highest average returns on the IPD. We also show that the LOLA update rule can be efficiently calculated using an extension of the policy gradient estimator, making the method suitable for model-free RL. The method thus scales to large parameter and input spaces and nonlinear function approximators. We apply LOLA to a grid world task with an embedded social dilemma using recurrent policies and opponent modelling. By explicitly considering the learning of the other agent, LOLA agents learn to cooperate out of self-interest. The code is at github.com/alshedivat/lola

Absence of remote earthquake triggering within the Coso and Salton Sea geothermal production fields

Geothermal areas are long recognized to be susceptible to remote earthquake triggering, probably due to the high seismicity rates and presence of geothermal fluids. However, anthropogenic injection and extraction activity may alter the stress state and fluid flow within the geothermal fields. Here we examine the remote triggering phenomena in the Coso geothermal field and its surrounding areas to assess possible anthropogenic effects. We find that triggered earthquakes are absent within the geothermal field but occur in the surrounding areas. Similar observation is also found in the Salton Sea geothermal field. We hypothesize that continuous geothermal operation has eliminated any significant differential pore pressure between fractures inside the geothermal field through flushing geothermal precipitations and sediments out of clogged fractures. To test this hypothesis, we analyze the pore-pressure-driven earthquake swarms, and they are found to occur outside or on the periphery of the geothermal production field. Therefore, our results suggest that the geothermal operation has changed the subsurface fracture network, and differential pore pressure is the primary controlling factor of remote triggering in geothermal fields

Optimization of second-harmonic generation from touching plasmonic wires

We employ transformation optics to optimize the generic nonlinear wave interaction of second-harmonic generation from a pair of touching metallic wires. We demonstrate a 10 orders of magnitude increase in the second-harmonic scattering cross-section by increasing the background permittivity and a 5 orders of magnitude increase in efficiency with respect to a single wire. These results have clear implications for the design of nanostructured metallic frequency-conversion devices. Finally, we exploit our analytic solution of a non-trivial nanophotonic geometry as a platform for performing a critical comparison of the strengths, weaknesses and validity of other prevailing theoretical approaches previously employed for nonlinear wave interactions at the nanoscale

On The Problem of Vacuum energy in Brane Theories

We point out that modern brane theories suffer from a severe vacuum energy problem. To be specific, the Casimir energy associated with the matter fields confined to the brane, is stemming from the one and the same localization mechanism which forms the brane itself, and is thus generically unavoidable. Possible practical solutions are discussed, including in particular spontaneously broken supersymmetry, and quantum mechanically induced brane tension.Comment: 9 pages, 1 figure, to be published in Phys. Lett.

Reference Architecture for Collaborative Design

Issues and themes of Collaborative Design (CD) addressed by research done so far are so extensive that when running a project of collaborative design, people may lack directions or guidelines to support the whole picture. Hence, developing reference architecture for CD is important and necessary in the academic and the empirical fields. Reference architecture provides the systematic, elementary skeleton and can be extended and adapted to diverse, changing environments. It also provides a comprehensive framework and enables practices implemented more thoroughly and easily. The reference architecture developed in this re-search is formed along three dimensions: decision aspect, design stage, and collaboration scope. There are five elements in the dimension of decision aspect: (1) participant, (2) product, (3) process, (4) organization, and (5) information. The dimension of design stage includes three stages: (1) planning and concepting, (2) system-level design and detail design, and (3) testing and prototyping. The dimension of collaboration scope includes three types of collaboration: (1) cross-functional, (2) cross-company, and (3) cross-industry. Because of the three reference dimensions, a cubic architecture is developed. The cubic reference architecture helps decision-makers in dealing with implementing a CD project or activity. It also serves as a guideline for CD system developers or people involved in the design collaboration to figure out their own responsibility functions and their relations with other members. Demonstration of how to use the reference architecture in developing design collaboration activities and specifying the details for cross-company CD is also provided in this research

On prisms, M\"obius ladders and the cycle space of dense graphs

For a graph X, let f_0(X) denote its number of vertices, d(X) its minimum degree and Z_1(X;Z/2) its cycle space in the standard graph-theoretical sense (i.e. 1-dimensional cycle group in the sense of simplicial homology theory with Z/2-coefficients). Call a graph Hamilton-generated if and only if the set of all Hamilton circuits is a Z/2-generating system for Z_1(X;Z/2). The main purpose of this paper is to prove the following: for every s > 0 there exists n_0 such that for every graph X with f_0(X) >= n_0 vertices, (1) if d(X) >= (1/2 + s) f_0(X) and f_0(X) is odd, then X is Hamilton-generated, (2) if d(X) >= (1/2 + s) f_0(X) and f_0(X) is even, then the set of all Hamilton circuits of X generates a codimension-one subspace of Z_1(X;Z/2), and the set of all circuits of X having length either f_0(X)-1 or f_0(X) generates all of Z_1(X;Z/2), (3) if d(X) >= (1/4 + s) f_0(X) and X is square bipartite, then X is Hamilton-generated. All these degree-conditions are essentially best-possible. The implications in (1) and (2) give an asymptotic affirmative answer to a special case of an open conjecture which according to [European J. Combin. 4 (1983), no. 3, p. 246] originates with A. Bondy.Comment: 33 pages; 5 figure

Learning from Demonstration in the Wild

Learning from demonstration (LfD) is useful in settings where hand-coding behaviour or a reward function is impractical. It has succeeded in a wide range of problems but typically relies on manually generated demonstrations or specially deployed sensors and has not generally been able to leverage the copious demonstrations available in the wild: those that capture behaviours that were occurring anyway using sensors that were already deployed for another purpose, e.g., traffic camera footage capturing demonstrations of natural behaviour of vehicles, cyclists, and pedestrians. We propose Video to Behaviour (ViBe), a new approach to learn models of behaviour from unlabelled raw video data of a traffic scene collected from a single, monocular, initially uncalibrated camera with ordinary resolution. Our approach calibrates the camera, detects relevant objects, tracks them through time, and uses the resulting trajectories to perform LfD, yielding models of naturalistic behaviour. We apply ViBe to raw videos of a traffic intersection and show that it can learn purely from videos, without additional expert knowledge.Comment: Accepted to the IEEE International Conference on Robotics and Automation (ICRA) 2019; extended version with appendi

The missing link in gravitational-wave astronomy: A summary of discoveries waiting in the decihertz range

Since 2015 the gravitational-wave observations of LIGO and Virgo have transformed our understanding of compact-object binaries. In the years to come, ground-based gravitational-wave observatories such as LIGO, Virgo, and their successors will increase in sensitivity, discovering thousands of stellar-mass binaries. In the 2030s, the space-based LISA will provide gravitational-wave observations of massive black holes binaries. Between the ∼10–103 Hz band of ground-based observatories and the ∼10−4–10− 1 Hz band of LISA lies the uncharted decihertz gravitational-wave band. We propose a Decihertz Observatory to study this frequency range, and to complement observations made by other detectors. Decihertz observatories are well suited to observation of intermediate-mass (∼102–104M⊙) black holes; they will be able to detect stellar-mass binaries days to years before they merge, providing early warning of nearby binary neutron star mergers and measurements of the eccentricity of binary black holes, and they will enable new tests of general relativity and the Standard Model of particle physics. Here we summarise how a Decihertz Observatory could provide unique insights into how black holes form and evolve across cosmic time, improve prospects for both multimessenger astronomy and multiband gravitational-wave astronomy, and enable new probes of gravity, particle physics and cosmology.publishedVersio

Synaptic dynamics contribute to long-term single neuron response fluctuations

Firing rate variability at the single neuron level is characterized by long-memory processes and complex statistics over a wide range of time scales (from milliseconds up to several hours). Here, we focus on the contribution of non-stationary efficacy of the ensemble of synapses-activated in response to a given stimulus-on single neuron response variability. We present and validate a method tailored for controlled and specific long-term activation of a single cortical neuron in vitro via synaptic or antidromic stimulation, enabling a clear separation between two determinants of neuronal response variability: membrane excitability dynamics vs. synaptic dynamics. Applying this method we show that, within the range of physiological activation frequencies, the synaptic ensemble of a given neuron is a key contributor to the neuronal response variability, long-memory processes and complex statistics observed over extended time scales. Synaptic transmission dynamics impact on response variability in stimulation rates that are substantially lower compared to stimulation rates that drive excitability resources to fluctuate. Implications to network embedded neurons are discussed. \ua9 2014 Reinartz, Biro, Gal, Giugliano and Marom