QDQD-Learning: A Collaborative Distributed Strategy for Multi-Agent Reinforcement Learning Through Consensus + Innovations

The paper considers a class of multi-agent Markov decision processes (MDPs), in which the network agents respond differently (as manifested by the instantaneous one-stage random costs) to a global controlled state and the control actions of a remote controller. The paper investigates a distributed reinforcement learning setup with no prior information on the global state transition and local agent cost statistics. Specifically, with the agents' objective consisting of minimizing a network-averaged infinite horizon discounted cost, the paper proposes a distributed version of $Q$-learning, $\mathcal{QD}$-learning, in which the network agents collaborate by means of local processing and mutual information exchange over a sparse (possibly stochastic) communication network to achieve the network goal. Under the assumption that each agent is only aware of its local online cost data and the inter-agent communication network is \emph{weakly} connected, the proposed distributed scheme is almost surely (a.s.) shown to yield asymptotically the desired value function and the optimal stationary control policy at each network agent. The analytical techniques developed in the paper to address the mixed time-scale stochastic dynamics of the \emph{consensus + innovations} form, which arise as a result of the proposed interactive distributed scheme, are of independent interest.Comment: Submitted to the IEEE Transactions on Signal Processing, 33 page

Distributed Linear Parameter Estimation: Asymptotically Efficient Adaptive Strategies

The paper considers the problem of distributed adaptive linear parameter estimation in multi-agent inference networks. Local sensing model information is only partially available at the agents and inter-agent communication is assumed to be unpredictable. The paper develops a generic mixed time-scale stochastic procedure consisting of simultaneous distributed learning and estimation, in which the agents adaptively assess their relative observation quality over time and fuse the innovations accordingly. Under rather weak assumptions on the statistical model and the inter-agent communication, it is shown that, by properly tuning the consensus potential with respect to the innovation potential, the asymptotic information rate loss incurred in the learning process may be made negligible. As such, it is shown that the agent estimates are asymptotically efficient, in that their asymptotic covariance coincides with that of a centralized estimator (the inverse of the centralized Fisher information rate for Gaussian systems) with perfect global model information and having access to all observations at all times. The proof techniques are mainly based on convergence arguments for non-Markovian mixed time scale stochastic approximation procedures. Several approximation results developed in the process are of independent interest.Comment: Submitted to SIAM Journal on Control and Optimization journal. Initial Submission: Sept. 2011. Revised: Aug. 201

Distributed Constrained Recursive Nonlinear Least-Squares Estimation: Algorithms and Asymptotics

This paper focuses on the problem of recursive nonlinear least squares parameter estimation in multi-agent networks, in which the individual agents observe sequentially over time an independent and identically distributed (i.i.d.) time-series consisting of a nonlinear function of the true but unknown parameter corrupted by noise. A distributed recursive estimator of the \emph{consensus} + \emph{innovations} type, namely $\mathcal{CIWNLS}$, is proposed, in which the agents update their parameter estimates at each observation sampling epoch in a collaborative way by simultaneously processing the latest locally sensed information~(\emph{innovations}) and the parameter estimates from other agents~(\emph{consensus}) in the local neighborhood conforming to a pre-specified inter-agent communication topology. Under rather weak conditions on the connectivity of the inter-agent communication and a \emph{global observability} criterion, it is shown that at every network agent, the proposed algorithm leads to consistent parameter estimates. Furthermore, under standard smoothness assumptions on the local observation functions, the distributed estimator is shown to yield order-optimal convergence rates, i.e., as far as the order of pathwise convergence is concerned, the local parameter estimates at each agent are as good as the optimal centralized nonlinear least squares estimator which would require access to all the observations across all the agents at all times. In order to benchmark the performance of the proposed distributed $\mathcal{CIWNLS}$ estimator with that of the centralized nonlinear least squares estimator, the asymptotic normality of the estimate sequence is established and the asymptotic covariance of the distributed estimator is evaluated. Finally, simulation results are presented which illustrate and verify the analytical findings.Comment: 28 pages. Initial Submission: Feb. 2016, Revised: July 2016, Accepted: September 2016, To appear in IEEE Transactions on Signal and Information Processing over Networks: Special Issue on Inference and Learning over Network

Gauging kinematical and internal symmetry groups for extended systems: the Galilean one-time and two-times harmonic oscillators

The possible external couplings of an extended non-relativistic classical system are characterized by gauging its maximal dynamical symmetry group at the center-of-mass. The Galilean one-time and two-times harmonic oscillators are exploited as models. The following remarkable results are then obtained: 1) a peculiar form of interaction of the system as a whole with the external gauge fields; 2) a modification of the dynamical part of the symmetry transformations, which is needed to take into account the alteration of the dynamics itself, induced by the {\it gauge} fields. In particular, the Yang-Mills fields associated to the internal rotations have the effect of modifying the time derivative of the internal variables in a scheme of minimal coupling (introduction of an internal covariant derivative); 3) given their dynamical effect, the Yang-Mills fields associated to the internal rotations apparently define a sort of Galilean spin connection, while the Yang-Mills fields associated to the quadrupole momentum and to the internal energy have the effect of introducing a sort of dynamically induced internal metric in the relative space.Comment: 32 pages, LaTex using the IOP preprint macro package (ioplppt.sty available at: http://www.iop.org/). The file is available at: http://www.fis.unipr.it/papers/1995.html The file is a uuencoded tar gzip file with the IOP preprint style include

A magnetized torus for modeling Sgr A* millimeter images and spectra

Context. The supermassive black hole, Sagittarius (Sgr) A*, in the centre of our Galaxy has the largest angular size in the sky among all astrophysical black holes. Its shadow, assuming no rotation, spans ~ 50 microarcsec. Resolving such dimensions has long been out of reach for astronomical instruments until a new generation of interferometers being operational during this decade. Of particular interest is the Event Horizon Telescope (EHT) with resolution ~ 20 microarcsec in the millimeter-wavelength range 0.87 mm - 1.3 mm. Aims. We investigate the ability of the fully general relativistic Komissarov (2006) analytical magnetized torus model to account for observable constraints at Sgr A* in the centimeter and millimeter domains. The impact of the magnetic field geometry on the observables is also studied. Methods. We calculate ray-traced centimeter- and millimeter-wavelength synchrotron spectra and images of a magnetized accretion torus surrounding the central black hole in Sgr A*. We assume stationarity, axial symmetry, constant specific angular momentum and polytropic equation of state. A hybrid population of thermal and non-thermal electrons is considered. Results. We show that the torus model is capable of reproducing spectral constraints in the millimeter domain, and in particular in the observable domain of the EHT. However, the torus model is not yet able to fit the centimeter spectrum. 1.3 mm images at high inclinations are in agreement with observable constraints. Conclusions. The ability of the torus model to account for observations of Sgr A* in the millimeter domain is interesting in the perspective of the future EHT. Such an analytical model allows very fast computations. It will thus be a suitable test bed for investigating large domains of physical parameters, as well as non-black-hole compact object candidates and alternative theories of gravity.Comment: Major changes wrt the June 2014 version. Accepted by A&

Leptons from Dark Matter Annihilation in Milky Way Subhalos

Numerical simulations of dark matter collapse and structure formation show that in addition to a large halo surrounding the baryonic component of our galaxy, there also exists a significant number of subhalos that extend hundreds of kiloparsecs beyond the edge of the observable Milky Way. We find that for dark matter (DM) annihilation models, galactic subhalos can significantly modify the spectrum of electrons and positrons as measured at our galactic position. Using data from the recent Via Lactea II simulation we include the subhalo contribution of electrons and positrons as boundary source terms for simulations of high energy cosmic ray propagation with a modified version of the publicly available GALPROP code. Focusing on the DM DM -> 4e annihilation channel, we show that including subhalos leads to a better fit to both the Fermi and PAMELA data. The best fit gives a dark matter particle mass of 1.2 TeV, for boost factors of 90 in the main halo and 1950-3800 in the subhalos (depending on assumptions about the background), in contrast to the 0.85 TeV mass that gives the best fit in the main halo-only scenario. These fits suggest that at least a third of the observed electron cosmic rays from DM annihilation could come from subhalos, opening up the possibility of a relaxation of recent stringent constraints from inverse Compton gamma rays originating from the high-energy leptons.Comment: 8 pages, 13 figures; added referenc

Lattice model study of the thermodynamic interplay of polymer crystallization and liquid-liquid demixing

We report Monte Carlo simulations of a lattice-polymer model that can account for both polymer crystallization and liquid-liquid demixing in solutions of semiflexible homopolymers. In our model, neighboring polymer segments can have isotropic interactions that affect demixing, and anisotropic interactions that are responsible for freezing. However, our simulations show that the isotropic interactions also have a noticeable effect on the freezing curve, as do the anisotropic interactions on demixing. As the relative strength of the isotropic interactions is reduced, the liquid-liquid demixing transition disappears below the freezing curve. A simple, extended Flory-Huggins theory accounts quite well for the phase behavior observed in the simulations.Comment: Revtex, 7 pages, the content accepted by J. Chem. Phy

Methyl 2-(4-ferrocenylbenzamido)thiophene-3-carboxylate and ethyl 2-(4-ferrocenylbenzamido)-1,3-thiazole-4-acetate, a unique ferrocen

The conformations and hydrogen bonding in the thiophene and thiazole title compounds, [Fe(C₅H₅)(C₂₀H₁₄NO₃S)], (I), and [Fe(C₅H₅)(C₁₉H₁₇N₂O₃S)], (II), are discussed. The sequence (C₅H₄)-(C₆H₄)-(CONH)-(C₄H₂S)-(CO₂Me) of rings and moieties in (I) is close to being planar; all consecutive interplanar angles are less than 10°. An intramolecular N-H...O=Cester hydrogen bond [graph set S(6), N...O = 2.768 (2) Å and N-H...O = 134 (2)°] effects the molecular planarity, and aggregation occurs via hydrogen-bonded chains formed from intermolecular Car-H...O=Cester/amide interactions along [010], with C...O distances ranging from 3.401 (3) to 3.577 (2) Å. The thiazole system in (II) crystallizes with two molecules in the asymmetric unit; these differ in the conformation along their long molecular axes; for example, the interplanar angle between the phenylene (C₆H₄) and thiazole (C₃NS) rings is 8.1 (2)° in one molecule and 27.66 (14)° in the other. Intermolecular N-H...O=Cester hydrogen bonds [N...O = 2.972 (4) and 2.971 (3) Å], each augmented by a Cphenylene-H...O=Cester interaction [3.184 (5) and 3.395 (4) Å], form motifs with graph set R¹₂(7) and generate chains along [100]. The amide C=O groups do not participate in hydrogen bonding. Compound (II) is the first reported ferrocenyl-containing thiazole structure