TaxAI: A Dynamic Economic Simulator and Benchmark for Multi-Agent Reinforcement Learning

Taxation and government spending are crucial tools for governments to promote economic growth and maintain social equity. However, the difficulty in accurately predicting the dynamic strategies of diverse self-interested households presents a challenge for governments to implement effective tax policies. Given its proficiency in modeling other agents in partially observable environments and adaptively learning to find optimal policies, Multi-Agent Reinforcement Learning (MARL) is highly suitable for solving dynamic games between the government and numerous households. Although MARL shows more potential than traditional methods such as the genetic algorithm and dynamic programming, there is a lack of large-scale multi-agent reinforcement learning economic simulators. Therefore, we propose a MARL environment, named \textbf{TaxAI}, for dynamic games involving $N$ households, government, firms, and financial intermediaries based on the Bewley-Aiyagari economic model. Our study benchmarks 2 traditional economic methods with 7 MARL methods on TaxAI, demonstrating the effectiveness and superiority of MARL algorithms. Moreover, TaxAI's scalability in simulating dynamic interactions between the government and 10,000 households, coupled with real-data calibration, grants it a substantial improvement in scale and reality over existing simulators. Therefore, TaxAI is the most realistic economic simulator, which aims to generate feasible recommendations for governments and individuals.Comment: 26 pages, 8 figures, 12 table

Self-Organized Coverage and Capacity Optimization for Cellular Mobile Networks

Die zur Erfüllung der zu erwartenden Steigerungen übertragener Datenmengen notwendige größere Heterogenität und steigende Anzahl von Zellen werden in der Zukunft zu einer deutlich höheren Komplexität bei Planung und Optimierung von Funknetzen führen. Zusätzlich erfordern räumliche und zeitliche Änderungen der Lastverteilung eine dynamische Anpassung von Funkabdeckung und -kapazität (Coverage-Capacity-Optimization, CCO). Aktuelle Planungs- und Optimierungsverfahren sind hochgradig von menschlichem Einfluss abhängig, was sie zeitaufwändig und teuer macht. Aus diesen Grnden treffen Ansätze zur besseren Automatisierung des Netzwerkmanagements sowohl in der Industrie, als auch der Forschung auf groes Interesse.Selbstorganisationstechniken (SO) haben das Potential, viele der aktuell durch Menschen gesteuerten Abläufe zu automatisieren. Ihnen wird daher eine zentrale Rolle bei der Realisierung eines einfachen und effizienten Netzwerkmanagements zugeschrieben. Die vorliegende Arbeit befasst sich mit selbstorganisierter Optimierung von Abdeckung und Übertragungskapazität in Funkzellennetzwerken. Der Parameter der Wahl hierfür ist die Antennenneigung. Die zahlreichen vorhandenen Ansätze hierfür befassen sich mit dem Einsatz heuristischer Algorithmen in der Netzwerkplanung. Im Gegensatz dazu betrachtet diese Arbeit den verteilten Einsatz entsprechender Optimierungsverfahren in den betreffenden Netzwerkknoten. Durch diesen Ansatz können zentrale Fehlerquellen (Single Point of Failure) und Skalierbarkeitsprobleme in den kommenden heterogenen Netzwerken mit hoher Knotendichte vermieden werden.Diese Arbeit stellt einen "Fuzzy Q-Learning (FQL)"-basierten Ansatz vor, ein einfaches Maschinenlernverfahren mit einer effektiven Abstraktion kontinuierlicher Eingabeparameter. Das CCO-Problem wird als Multi-Agenten-Lernproblem modelliert, in dem jede Zelle versucht, ihre optimale Handlungsstrategie (d.h. die optimale Anpassung der Antennenneigung) zu lernen. Die entstehende Dynamik der Interaktion mehrerer Agenten macht die Fragestellung interessant. Die Arbeit betrachtet verschiedene Aspekte des Problems, wie beispielsweise den Unterschied zwischen egoistischen und kooperativen Lernverfahren, verteiltem und zentralisiertem Lernen, sowie die Auswirkungen einer gleichzeitigen Modifikation der Antennenneigung auf verschiedenen Knoten und deren Effekt auf die Lerneffizienz.Die Leistungsfähigkeit der betrachteten Verfahren wird mittels eine LTE-Systemsimulators evaluiert. Dabei werden sowohl gleichmäßig verteilte Zellen, als auch Zellen ungleicher Größe betrachtet. Die entwickelten Ansätze werden mit bekannten Lösungen aus der Literatur verglichen. Die Ergebnisse zeigen, dass die vorgeschlagenen Lösungen effektiv auf Änderungen im Netzwerk und der Umgebung reagieren können. Zellen stellen sich selbsttätig schnell auf Ausfälle und Inbetriebnahmen benachbarter Systeme ein und passen ihre Antennenneigung geeignet an um die Gesamtleistung des Netzes zu verbessern. Die vorgestellten Lernverfahren erreichen eine bis zu 30 Prozent verbesserte Leistung als bereits bekannte Ansätze. Die Verbesserungen steigen mit der Netzwerkgröße.The challenging task of cellular network planning and optimization will become more and more complex because of the expected heterogeneity and enormous number of cells required to meet the traffic demands of coming years. Moreover, the spatio-temporal variations in the traffic patterns of cellular networks require their coverage and capacity to be adapted dynamically. The current network planning and optimization procedures are highly manual, which makes them very time consuming and resource inefficient. For these reasons, there is a strong interest in industry and academics alike to enhance the degree of automation in network management. Especially, the idea of Self-Organization (SO) is seen as the key to simplified and efficient cellular network management by automating most of the current manual procedures. In this thesis, we study the self-organized coverage and capacity optimization of cellular mobile networks using antenna tilt adaptations. Although, this problem is widely studied in literature but most of the present work focuses on heuristic algorithms for network planning tool automation. In our study we want to minimize this reliance on these centralized tools and empower the network elements for their own optimization. This way we can avoid the single point of failure and scalability issues in the emerging heterogeneous and densely deployed networks.In this thesis, we focus on Fuzzy Q-Learning (FQL), a machine learning technique that provides a simple learning mechanism and an effective abstraction level for continuous domain variables. We model the coverage-capacity optimization as a multi-agent learning problem where each cell is trying to learn its optimal action policy i.e. the antenna tilt adjustments. The network dynamics and the behavior of multiple learning agents makes it a highly interesting problem. We look into different aspects of this problem like the effect of selfish learning vs. cooperative learning, distributed vs. centralized learning as well as the effect of simultaneous parallel antenna tilt adaptations by multiple agents and its effect on the learning efficiency.We evaluate the performance of the proposed learning schemes using a system level LTE simulator. We test our schemes in regular hexagonal cell deployment as well as in irregular cell deployment. We also compare our results to a relevant learning scheme from literature. The results show that the proposed learning schemes can effectively respond to the network and environmental dynamics in an autonomous way. The cells can quickly respond to the cell outages and deployments and can re-adjust their antenna tilts to improve the overall network performance. Additionally the proposed learning schemes can achieve up to 30 percent better performance than the available scheme from literature and these gains increases with the increasing network size

Biomechanical Locomotion Heterogeneity in Synthetic Crowds

Synthetic crowd simulation combines rule sets at different conceptual layers to represent the dynamic nature of crowds while adhering to basic principles of human steering, such as collision avoidance and goal completion. In this dissertation, I explore synthetic crowd simulation at the steering layer using a critical approach to define the central theme of the work, the impact of model representation and agent diversity in crowds. At the steering layer, simulated agents make regular decisions, or actions, related to steering which are often responsible for the emergent behaviours found in the macro-scale crowd. Because of this bottom-up impact of a steering model's defining rule-set, I postulate that biomechanics and diverse biomechanics may alter the outcomes of dynamic synthetic-crowds-based outcomes. This would mean that an assumption of normativity and/or homogeneity among simulated agents and their mobility would provide an inaccurate representation of a scenario. If these results are then used to make real world decisions, say via policy or design, then those populations not represented in the simulated scenario may experience a lack of representation in the actualization of those decisions. A focused literature review shows that applications of both biomechanics and diverse locomotion representation at this layer of modelling are very narrow and often not present. I respond to the narrowness of this representation by addressing both biomechanics and heterogeneity separately. To address the question of performance and importance of locomotion biomechanics in crowd simulation, I use a large scale comparative approach. The industry standard synthetic crowd models are tested under a battery of benchmarks derived from prior work in comparative analysis of synthetic crowds as well as new scenarios derived from built environments. To address the question of the importance of heterogeneity in locomotion biomechanics, I define tiers of impact in the multi-agent crowds model at the steering layer--from the action space, to the agent space, to the crowds space. To this end, additional models and layers are developed to address the modelling and application of heterogeneous locomotion biomechanics in synthetic crowds. The results of both studies form a research arc which shows that the biomechanics in steering models provides important fidelity in several applications and that heterogeneity in the model of locomotion biomechanics directly impacts both qualitative and quantitative synthetic crowds outcomes. As well, systems, approaches, and pitfalls regarding the analysis of steering model and human mobility diversity are described

How to Identify Investor's types in real financial markets by means of agent based simulation

The paper proposes a computational adaptation of the principles underlying principal component analysis with agent based simulation in order to produce a novel modeling methodology for financial time series and financial markets. Goal of the proposed methodology is to find a reduced set of investor s models (agents) which is able to approximate or explain a target financial time series. As computational testbed for the study, we choose the learning system L FABS which combines simulated annealing with agent based simulation for approximating financial time series. We will also comment on how L FABS s architecture could exploit parallel computation to scale when dealing with massive agent simulations. Two experimental case studies showing the efficacy of the proposed methodology are reported.Comment: 18 pages, in pres

Adaptive-Aggressive Traders Don't Dominate

For more than a decade Vytelingum's Adaptive-Aggressive (AA) algorithm has been recognized as the best-performing automated auction-market trading-agent strategy currently known in the AI/Agents literature; in this paper, we demonstrate that it is in fact routinely outperformed by another algorithm when exhaustively tested across a sufficiently wide range of market scenarios. The novel step taken here is to use large-scale compute facilities to brute-force exhaustively evaluate AA in a variety of market environments based on those used for testing it in the original publications. Our results show that even in these simple environments AA is consistently out-performed by IBM's GDX algorithm, first published in 2002. We summarize here results from more than one million market simulation experiments, orders of magnitude more testing than was reported in the original publications that first introduced AA. A 2019 ICAART paper by Cliff claimed that AA's failings were revealed by testing it in more realistic experiments, with conditions closer to those found in real financial markets, but here we demonstrate that even in the simple experiment conditions that were used in the original AA papers, exhaustive testing shows AA to be outperformed by GDX. We close this paper with a discussion of the methodological implications of our work: any results from previous papers where any one trading algorithm is claimed to be superior to others on the basis of only a few thousand trials are probably best treated with some suspicion now. The rise of cloud computing means that the compute-power necessary to subject trading algorithms to millions of trials over a wide range of conditions is readily available at reasonable cost: we should make use of this; exhaustive testing such as is shown here should be the norm in future evaluations and comparisons of new trading algorithms.Comment: To be published as a chapter in "Agents and Artificial Intelligence" edited by Jaap van den Herik, Ana Paula Rocha, and Luc Steels; forthcoming 2019/2020. 24 Pages, 1 Figure, 7 Table