An Intelligent Hybrid Protocol for Effective Load Balancing and Energy Efficient Routing for MANETs

MANET (Mobile ad hoc network) is an autonomous decentralised network. And it is a collection of wireless mobile nodes that dynamically form a temporary network without the reliance of any infrastructure or central administration. Routing is a challenging task in manet. When the size and complexity increases the important challenge in manet is to avoid congestion with effective load balancing and improve energy, QoS parameters inside the network. In this work we propose a new hybrid protocol by combining ACO and Predator prey (LV) model which known as ACRRCC (Ant colony based rate regulating congestion control) method, which works efficiently in two phases. The efficient and optimal routing strategy is done by phase I using ant colony optimization. In phase II the congestion is majorly controlled by employing a mathematical model named predator-prey model which regulates the rate of the traffic flow in the network path. Performance of our proposed hybrid model ACRRCC yields good results under simulation study when compared with simple ACO

Smart Grid communications in high traffic environments

The establishment of a previously non-existent data class known as the Smart Grid will pose many difficulties on current and future communication infrastructure. It is imperative that the Smart Grid (SG), as the reactionary and monitory arm of the Power Grid (PG), be able to communicate effectively between grid controllers and individual User Equipment (UE). By doing so, the successful implementation of SG applications can occur, including support for higher capacities of Renewable Energy Resources. As the SG matures, the number of UEs required is expected to rise increasing the traffic in an already burdened communications network. This thesis aims to optimally allocate radio resources such that the SG Quality of Service (QoS) requirements are satisfied with minimal effect on pre-existing traffic. To address this resource allocation problem, a Lotka-Volterra (LV) based resource allocation and scheduler was developed due to its ability to easily adapt to the dynamics of a telecommunications environment. Unlike previous resource allocation algorithms, the LV scheme allocated resources to each class as a function of its growth rate. By doing so, the QoS requirements of the SG were satisfied, with minimal effect on pre-existing traffic. Class queue latencies were reduced by intelligent scheduling of periodic traffic and forward allocation of resources. This thesis concludes that the SG will have a large effect on the telecommunications environment if not successfully controlled and monitored. This effect can be minimized by utilizing the proposed LV based resource allocation and scheduler system. Furthermore, it was shown that the allocation of periodic SG radio channels was optimized by continual updates of the LV model. This ensured the QoS requirements of the SG are achieved and provided enhanced performance. Successful integration of SG UEs in a wireless network can pave the way for increased capacity of Renewable and Intermittent Energy Resources operating on the PG

Systems and algorithms for wireless sensor networks based on animal and natural behavior

In last decade, there have been many research works about wireless sensor networks (WSNs) focused on improving the network performance as well as increasing the energy efficiency and communications effectiveness. Many of these new mechanisms have been implemented using the behaviors of certain animals, such as ants, bees, or schools of fish.These systems are called bioinspired systems and are used to improve aspects such as handling large-scale networks, provide dynamic nature, and avoid resource constraints, heterogeneity, unattended operation, or robustness, amongmanyothers.Therefore, thispaper aims to studybioinspired mechanisms in the field ofWSN, providing the concepts of these behavior patterns in which these new approaches are based. The paper will explain existing bioinspired systems in WSNs and analyze their impact on WSNs and their evolution. In addition, we will conduct a comprehensive review of recently proposed bioinspired systems, protocols, and mechanisms. Finally, this paper will try to analyze the applications of each bioinspired mechanism as a function of the imitated animal and the deployed application. Although this research area is considered an area with highly theoretical content, we intend to show the great impact that it is generating from the practical perspective.Sendra, S.; Parra Boronat, L.; Lloret, J.; Khan, S. (2015). Systems and algorithms for wireless sensor networks based on animal and natural behavior. International Journal of Distributed Sensor Networks. 2015:1-19. doi:10.1155/2015/625972S1192015Iram, R., Sheikh, M. I., Jabbar, S., & Minhas, A. A. (2011). Computational intelligence based optimization in wireless sensor network. 2011 International Conference on Information and Communication Technologies. doi:10.1109/icict.2011.5983561Lloret, J., Bosch, I., Sendra, S., & Serrano, A. (2011). A Wireless Sensor Network for Vineyard Monitoring That Uses Image Processing. Sensors, 11(6), 6165-6196. doi:10.3390/s110606165Lloret, J., Garcia, M., Bri, D., & Sendra, S. (2009). A Wireless Sensor Network Deployment for Rural and Forest Fire Detection and Verification. Sensors, 9(11), 8722-8747. doi:10.3390/s91108722Dasgupta, P. (2008). A Multiagent Swarming System for Distributed Automatic Target Recognition Using Unmanned Aerial Vehicles. IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans, 38(3), 549-563. doi:10.1109/tsmca.2008.918619Quwaider, M., & Biswas, S. (2012). Delay Tolerant Routing Protocol Modeling for Low Power Wearable Wireless Sensor Networks. Network Protocols and Algorithms, 4(3). doi:10.5296/npa.v4i3.2054Sendra, S., Lloret, J., Garcia, M., & Toledo, J. F. (2011). Power Saving and Energy Optimization Techniques for Wireless Sensor Neworks (Invited Paper). Journal of Communications, 6(6). doi:10.4304/jcm.6.6.439-459Liu, M., & Song, C. (2012). Ant-Based Transmission Range Assignment Scheme for Energy Hole Problem in Wireless Sensor Networks. International Journal of Distributed Sensor Networks, 8(12), 290717. doi:10.1155/2012/290717Riva, G., & Finochietto, J. M. (2012). Pheromone-based In-Network Processing for Wireless Sensor Network Monitoring Systems. Network Protocols and Algorithms, 4(4). doi:10.5296/npa.v4i4.2206Garcia, M., Sendra, S., Lloret, J., & Canovas, A. (2011). Saving energy and improving communications using cooperative group-based Wireless Sensor Networks. Telecommunication Systems, 52(4), 2489-2502. doi:10.1007/s11235-011-9568-3Kim, J.-Y., Sharma, T., Kumar, B., Tomar, G. S., Berry, K., & Lee, W.-H. (2014). Intercluster Ant Colony Optimization Algorithm for Wireless Sensor Network in Dense Environment. International Journal of Distributed Sensor Networks, 10(4), 457402. doi:10.1155/2014/457402Dressler, F., & Akan, O. B. (2010). A survey on bio-inspired networking. Computer Networks, 54(6), 881-900. doi:10.1016/j.comnet.2009.10.024Atakan, B., & Akan, O. B. (2006). Immune System Based Distributed Node and Rate Selection in Wireless Sensor Networks. 2006 1st Bio-Inspired Models of Network, Information and Computing Systems. doi:10.1109/bimnics.2006.361806Di Pietro, R., & Verde, N. V. (2011). Introducing epidemic models for data survivability in Unattended Wireless Sensor Networks. 2011 IEEE International Symposium on a World of Wireless, Mobile and Multimedia Networks. doi:10.1109/wowmom.2011.5986165Marwaha, S., Indulska, J., & Portmann, M. (2009). Biologically Inspired Ant-Based Routing in Mobile Ad hoc Networks (MANET): A Survey. 2009 Symposia and Workshops on Ubiquitous, Autonomic and Trusted Computing. doi:10.1109/uic-atc.2009.95Jha, V., Khetarpal, K., & Sharma, M. (2011). A survey of nature inspired routing algorithms for MANETs. 2011 3rd International Conference on Electronics Computer Technology. doi:10.1109/icectech.2011.5942042Fernandez-Marquez, J. L., Di Marzo Serugendo, G., Montagna, S., Viroli, M., & Arcos, J. L. (2012). Description and composition of bio-inspired design patterns: a complete overview. Natural Computing, 12(1), 43-67. doi:10.1007/s11047-012-9324-yCamilo, T., Carreto, C., Silva, J. S., & Boavida, F. (2006). An Energy-Efficient Ant-Based Routing Algorithm for Wireless Sensor Networks. Lecture Notes in Computer Science, 49-59. doi:10.1007/11839088_5Selvakennedy, S., Sinnappan, S., & Shang, Y. (2006). T-ANT: A Nature-Inspired Data Gathering Protocol for Wireless Sensor Networks. Journal of Communications, 1(2). doi:10.4304/jcm.1.2.22-29Almshreqi, A. M. S., Ali, B. M., Rasid, M. F. A., Ismail, A., & Varahram, P. (2012). An improved routing mechanism using bio-inspired for energy balancing in wireless sensor networks. The International Conference on Information Network 2012. doi:10.1109/icoin.2012.6164367Chen, G., Guo, T.-D., Yang, W.-G., & Zhao, T. (2006). An improved ant-based routing protocol in Wireless Sensor Networks. 2006 International Conference on Collaborative Computing: Networking, Applications and Worksharing. doi:10.1109/colcom.2006.361893Okdem, S., & Karaboga, D. (2006). Routing in Wireless Sensor Networks Using Ant Colony Optimization. First NASA/ESA Conference on Adaptive Hardware and Systems (AHS’06). doi:10.1109/ahs.2006.63Salehpour, A.-A., Mirmobin, B., Afzali-Kusha, A., & Mohammadi, S. (2008). An energy efficient routing protocol for cluster-based wireless sensor networks using ant colony optimization. 2008 International Conference on Innovations in Information Technology. doi:10.1109/innovations.2008.4781748Wen, Y., Chen, Y., & Pan, M. (2008). Adaptive ant-based routing in wireless sensor networks using Energy*Delay metrics. Journal of Zhejiang University-SCIENCE A, 9(4), 531-538. doi:10.1631/jzus.a071382Liao, W.-H., Kao, Y., & Wu, R.-T. (2011). Ant colony optimization based sensor deployment protocol for wireless sensor networks. Expert Systems with Applications, 38(6), 6599-6605. doi:10.1016/j.eswa.2010.11.079Pavai, K., Sivagami, A., & Sridharan, D. (2009). Study of Routing Protocols in Wireless Sensor Networks. 2009 International Conference on Advances in Computing, Control, and Telecommunication Technologies. doi:10.1109/act.2009.133Juan, L., Chen, S., & Chao, Z. (2007). Ant System Based Anycast Routing in Wireless Sensor Networks. 2007 International Conference on Wireless Communications, Networking and Mobile Computing. doi:10.1109/wicom.2007.603Wang, C., & Lin, Q. (2008). Swarm intelligence optimization based routing algorithm for Wireless Sensor Networks. 2008 International Conference on Neural Networks and Signal Processing. doi:10.1109/icnnsp.2008.4590326Jiang, H., Wang, M., Liu, M., & Yan, J. (2012). A quantum-inspired ant-based routing algorithm for WSNs. Proceedings of the 2012 IEEE 16th International Conference on Computer Supported Cooperative Work in Design (CSCWD). doi:10.1109/cscwd.2012.6221881Okazaki, A. M., & Frohlich, A. A. (2011). Ant-based Dynamic Hop Optimization Protocol: A routing algorithm for Mobile Wireless Sensor Networks. 2011 IEEE GLOBECOM Workshops (GC Wkshps). doi:10.1109/glocomw.2011.6162356Hui, X., Zhigang, Z., & Xueguang, Z. (2009). A Novel Routing Protocol in Wireless Sensor Networks Based on Ant Colony Optimization. 2009 International Conference on Environmental Science and Information Application Technology. doi:10.1109/esiat.2009.460AbdelSalam, H. S., & Olariu, S. (2012). BEES: BioinspirEd backbonE Selection in Wireless Sensor Networks. IEEE Transactions on Parallel and Distributed Systems, 23(1), 44-51. doi:10.1109/tpds.2011.100Da Silva Rego, A., Celestino, J., dos Santos, A., Cerqueira, E. C., Patel, A., & Taghavi, M. (2012). BEE-C: A bio-inspired energy efficient cluster-based algorithm for data continuous dissemination in Wireless Sensor Networks. 2012 18th IEEE International Conference on Networks (ICON). doi:10.1109/icon.2012.6506592Neshat, M., Sepidnam, G., Sargolzaei, M., & Toosi, A. N. (2012). Artificial fish swarm algorithm: a survey of the state-of-the-art, hybridization, combinatorial and indicative applications. Artificial Intelligence Review, 42(4), 965-997. doi:10.1007/s10462-012-9342-2Antoniou, P., Pitsillides, A., Blackwell, T., & Engelbrecht, A. (2009). Employing the flocking behavior of birds for controlling congestion in autonomous decentralized networks. 2009 IEEE Congress on Evolutionary Computation. doi:10.1109/cec.2009.4983153Ruihua, Z., Zhiping, J., Xin, L., & Dongxue, H. (2011). Double cluster-heads clustering algorithm for wireless sensor networks using PSO. 2011 6th IEEE Conference on Industrial Electronics and Applications. doi:10.1109/iciea.2011.5975688Kulkarni, R. V., Venayagamoorthy, G. K., & Cheng, M. X. (2009). Bio-inspired node localization in wireless sensor networks. 2009 IEEE International Conference on Systems, Man and Cybernetics. doi:10.1109/icsmc.2009.5346107Kulkarni, R. V., & Venayagamoorthy, G. K. (2010). Bio-inspired Algorithms for Autonomous Deployment and Localization of Sensor Nodes. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 40(6), 663-675. doi:10.1109/tsmcc.2010.2049649Xin Song, Cuirong Wang, Wang, J., & Bin Zhang. (2010). A hierarchical routing protocol based on AFSO algorithm for WSN. 2010 International Conference On Computer Design and Applications. doi:10.1109/iccda.2010.5541265Gao, X. Z., Wu, Y., Zenger, K., & Huang, X. (2010). A Knowledge-Based Artificial Fish-Swarm Algorithm. 2010 13th IEEE International Conference on Computational Science and Engineering. doi:10.1109/cse.2010.49Wang, L., & Ma, L. (2011). A hybrid artificial fish swarm algorithm for Bin-packing problem. Proceedings of 2011 International Conference on Electronic & Mechanical Engineering and Information Technology. doi:10.1109/emeit.2011.6022829Yiyue, W., Hongmei, L., & Hengyang, H. (2012). Wireless Sensor Network Deployment Using an Optimized Artificial Fish Swarm Algorithm. 2012 International Conference on Computer Science and Electronics Engineering. doi:10.1109/iccsee.2012.453Yang, X.-S. (2010). A New Metaheuristic Bat-Inspired Algorithm. Studies in Computational Intelligence, 65-74. doi:10.1007/978-3-642-12538-6_6Goyal, S., & Patterh, M. S. (2013). Performance of BAT Algorithm on Localization of Wireless Sensor Network. INTERNATIONAL JOURNAL OF COMPUTERS & TECHNOLOGY, 6(3), 351-358. doi:10.24297/ijct.v6i3.4481Krishnanand, K. N., & Ghose, D. (2006). Glowworm swarm based optimization algorithm for multimodal functions with collective robotics applications. Multiagent and Grid Systems, 2(3), 209-222. doi:10.3233/mgs-2006-2301Apostolopoulos, T., & Vlachos, A. (2011). Application of the Firefly Algorithm for Solving the Economic Emissions Load Dispatch Problem. International Journal of Combinatorics, 2011, 1-23. doi:10.1155/2011/523806Liao, W.-H., Kao, Y., & Li, Y.-S. (2011). A sensor deployment approach using glowworm swarm optimization algorithm in wireless sensor networks. Expert Systems with Applications, 38(10), 12180-12188. doi:10.1016/j.eswa.2011.03.053Sun, Y., Jiang, Q., & Zhang, K. (2012). A clustering scheme for Reachback Firefly Synchronicity in wireless sensor networks. 2012 3rd IEEE International Conference on Network Infrastructure and Digital Content. doi:10.1109/icnidc.2012.6418705Zungeru, A. M., Ang, L.-M., & Seng, K. P. (2012). Termite-Hill. International Journal of Swarm Intelligence Research, 3(4), 1-22. doi:10.4018/jsir.2012100101KumarE, S., S. M., K., & Kumar B. P., V. (2014). Clustering Protocol for Wireless Sensor Networks based on Rhesus Macaque (Macaca mulatta) Animal's Social Behavior. International Journal of Computer Applications, 87(8), 20-27. doi:10.5120/15229-3754Breza, M., & McCann, J. A. (2008). Lessons in Implementing Bio-inspired Algorithms on Wireless Sensor Networks. 2008 NASA/ESA Conference on Adaptive Hardware and Systems. doi:10.1109/ahs.2008.72Aziz, N. A. B. A., Mohemmed, A. W., & Sagar, B. S. D. (2007). Particle Swarm Optimization and Voronoi diagram for Wireless Sensor Networks coverage optimization. 2007 International Conference on Intelligent and Advanced Systems. doi:10.1109/icias.2007.4658528Falcon, R., Li, X., Nayak, A., & Stojmenovic, I. (2012). A harmony-seeking firefly swarm to the periodic replacement of damaged sensors by a team of mobile robots. 2012 IEEE International Conference on Communications (ICC). doi:10.1109/icc.2012.6363859Antoniou, P., & Pitsillides, A. (2010). A bio-inspired approach for streaming applications in wireless sensor networks based on the Lotka–Volterra competition model. Computer Communications, 33(17), 2039-2047. doi:10.1016/j.comcom.2010.07.020Benahmed, K., Merabti, M., & Haffaf, H. (2012). Inspired Social Spider Behavior for Secure Wireless Sensor Networks. International Journal of Mobile Computing and Multimedia Communications, 4(4), 1-10. doi:10.4018/jmcmc.2012100101Alrajeh, N. A., & Lloret, J. (2013). Intrusion Detection Systems Based on Artificial Intelligence Techniques in Wireless Sensor Networks. International Journal of Distributed Sensor Networks, 9(10), 351047. doi:10.1155/2013/351047Hussain, S., Matin, A. W., & Islam, O. (2007). Genetic Algorithm for Hierarchical Wireless Sensor Networks. Journal of Networks, 2(5). doi:10.4304/jnw.2.5.87-97Hussain, S., Matin, A. W., & Islam, O. (2007). Genetic Algorithm for Energy Efficient Clusters in Wireless Sensor Networks. Fourth International Conference on Information Technology (ITNG’07). doi:10.1109/itng.2007.97Ferentinos, K. P., & Tsiligiridis, T. A. (2007). Adaptive design optimization of wireless sensor networks using genetic algorithms. Computer Networks, 51(4), 1031-1051. doi:10.1016/j.comnet.2006.06.013Jia, J., Chen, J., Chang, G., & Tan, Z. (2009). Energy efficient coverage control in wireless sensor networks based on multi-objective genetic algorithm. Computers & Mathematics with Applications, 57(11-12), 1756-1766. doi:10.1016/j.camwa.2008.10.036Nan, G.-F., Li, M.-Q., & Li, J. (2007). Estimation of Node Localization with a Real-Coded Genetic Algorithm in WSNs. 2007 International Conference on Machine Learning and Cybernetics. doi:10.1109/icmlc.2007.4370265Saleem, K., Fisal, N., Abdullah, M. S., Zulkarmwan, A. B., Hafizah, S., & Kamilah, S. (2009). Proposed Nature Inspired Self-Organized Secure Autonomous Mechanism for WSNs. 2009 First Asian Conference on Intelligent Information and Database Systems. doi:10.1109/aciids.2009.75Jabbari, A., & Lang, W. (2010). Advanced Bio-inspired Plausibility Checking in a Wireless Sensor Network Using Neuro-immune Systems: Autonomous Fault Diagnosis in an Intelligent Transportation System. 2010 Fourth International Conference on Sensor Technologies and Applications. doi:10.1109/sensorcomm.2010.24Ponnusamy, V., & Abdullah, A. (2010). Biologically Inspired (Botany) Mobile Agent Based Self-Healing Wireless Sensor Network. 2010 Sixth International Conference on Intelligent Environments. doi:10.1109/ie.2010.46Li, J., Cui, Z., & Shi, Z. (2012). An Improved Artificial Plant Optimization Algorithm for Coverage Problem in WSN. Sensor Letters, 10(8), 1874-1878. doi:10.1166/sl.2012.2627Sendra, S., Llario, F., Parra, L., & Lloret, J. (2014). Smart Wireless Sensor Network to Detect and Protect Sheep and Goats to Wolf Attacks. Recent Advances in Communications and Networking Technology, 2(2), 91-101. doi:10.2174/22117407112016660012Sendra, S., Granell, E., Lloret, J., & Rodrigues, J. J. P. C. (2013). Smart Collaborative Mobile System for Taking Care of Disabled and Elderly People. Mobile Networks and Applications, 19(3), 287-302. doi:10.1007/s11036-013-0445-zGarcia, M., Sendra, S., Lloret, G., & Lloret, J. (2011). Monitoring and control sensor system for fish feeding in marine fish farms. IET Communications, 5(12), 1682-1690. doi:10.1049/iet-com.2010.0654Sendra, S., Lloret, J., Rodrigues, J. J. P. C., & Aguiar, J. M. (2013). Underwater Wireless Communications in Freshwater at 2.4 GHz. IEEE Communications Letters, 17(9), 1794-1797. doi:10.1109/lcomm.2013.072313.131214Lloret, J., Sendra, S., Ardid, M., & Rodrigues, J. J. P. C. (2012). Underwater Wireless Sensor Communications in the 2.4 GHz ISM Frequency Band. Sensors, 12(4), 4237-4264. doi:10.3390/s12040423

Socially-aware congestion control in ad-hoc networks: Current status and the way forward

Ad-hoc social networks (ASNETs) represent a special type of traditional ad-hoc network in whicha user’s social properties (such as the social connections and communications metadata as wellas application data) are leveraged for offering enhanced services in a distributed infrastructurelessenvironments. However, the wireless medium, due to limited bandwidth, can easily suffer from theproblem of congestion when social metadata and application data are exchanged among nodes—a problem that is compounded by the fact that some nodes may act selfishly and not share itsresources. While a number of congestion control schemes have been proposed for the traditional ad-hoc networks, there has been limited focus on incorporating social awareness into congestion controlschemes. We revisit the existing traditional ad-hoc congestion control and data distribution protocolsand motivate the need for embedding social awareness into these protocols to improve performance.We report that although some work is available in opportunistic network that uses socially-awaretechniques to control the congestion issue, this area is largely unexplored and warrants more researchattention. In this regards, we highlight the current research progress and identify multiple futuredirections of research

Enabling Cyber Physical Systems with Wireless Sensor Networking Technologies, Multiagent System Paradigm, and Natural Ecosystems

Wireless sensor networks (WSNs) are key components in the emergent cyber physical systems (CPSs). They may include hundreds of spatially distributed sensors which interact to solve complex tasks going beyond their individual capabilities. Due to the limited capabilities of sensors, sensor actions cannot meet CPS requirements while controlling and coordinating the operations of physical and engineered systems. To overcome these constraints, we explore the ecosystem metaphor for WSNs with the aim of taking advantage of the efficient adaptation behavior and communication mechanisms of living organisms. By mapping these organisms onto sensors and ecosystems onto WSNs, we highlight shortcomings that prevent WSNs from delivering the capabilities of ecosystems at several levels, including structure, topology, goals, communications, and functions. We then propose an agent-based architecture that migrates complex processing tasks outside the physical sensor network while incorporating missing characteristics of autonomy, intelligence, and context awareness to the WSN. Unlike existing works, we use software agents to map WSNs to natural ecosystems and enhance WSN capabilities to take advantage of bioinspired algorithms. We extend our architecture and propose a new intelligent CPS framework where several control levels are embedded in the physical system, thereby allowing agents to support WSNs technologies in enabling CPSs

Dynamics of Macrosystems; Proceedings of a Workshop, September 3-7, 1984

There is an increasing awareness of the important and persuasive role that instability and random, chaotic motion play in the dynamics of macrosystems. Further research in the field should aim at providing useful tools, and therefore the motivation should come from important questions arising in specific macrosystems. Such systems include biochemical networks, genetic mechanisms, biological communities, neutral networks, cognitive processes and economic structures. This list may seem heterogeneous, but there are similarities between evolution in the different fields. It is not surprising that mathematical methods devised in one field can also be used to describe the dynamics of another. IIASA is attempting to make progress in this direction. With this aim in view this workshop was held at Laxenburg over the period 3-7 September 1984. These Proceedings cover a broad canvas, ranging from specific biological and economic problems to general aspects of dynamical systems and evolutionary theory

Many-agent Reinforcement Learning

Multi-agent reinforcement learning (RL) solves the problem of how each agent should behave optimally in a stochastic environment in which multiple agents are learning simultaneously. It is an interdisciplinary domain with a long history that lies in the joint area of psychology, control theory, game theory, reinforcement learning, and deep learning. Following the remarkable success of the AlphaGO series in single-agent RL, 2019 was a booming year that witnessed significant advances in multi-agent RL techniques; impressive breakthroughs have been made on developing AIs that outperform humans on many challenging tasks, especially multi-player video games. Nonetheless, one of the key challenges of multi-agent RL techniques is the scalability; it is still non-trivial to design efficient learning algorithms that can solve tasks including far more than two agents ($N \gg 2$), which I name by \emph{many-agent reinforcement learning} (MARL\footnote{I use the world of ``MARL" to denote multi-agent reinforcement learning with a particular focus on the cases of many agents; otherwise, it is denoted as ``Multi-Agent RL" by default.}) problems. In this thesis, I contribute to tackling MARL problems from four aspects. Firstly, I offer a self-contained overview of multi-agent RL techniques from a game-theoretical perspective. This overview fills the research gap that most of the existing work either fails to cover the recent advances since 2010 or does not pay adequate attention to game theory, which I believe is the cornerstone to solving many-agent learning problems. Secondly, I develop a tractable policy evaluation algorithm -- $\alpha^\alpha$-Rank -- in many-agent systems. The critical advantage of $\alpha^\alpha$-Rank is that it can compute the solution concept of $\alpha$-Rank tractably in multi-player general-sum games with no need to store the entire pay-off matrix. This is in contrast to classic solution concepts such as Nash equilibrium which is known to be $PPAD$-hard in even two-player cases. $\alpha^\alpha$-Rank allows us, for the first time, to practically conduct large-scale multi-agent evaluations. Thirdly, I introduce a scalable policy learning algorithm -- mean-field MARL -- in many-agent systems. The mean-field MARL method takes advantage of the mean-field approximation from physics, and it is the first provably convergent algorithm that tries to break the curse of dimensionality for MARL tasks. With the proposed algorithm, I report the first result of solving the Ising model and multi-agent battle games through a MARL approach. Fourthly, I investigate the many-agent learning problem in open-ended meta-games (i.e., the game of a game in the policy space). Specifically, I focus on modelling the behavioural diversity in meta-games, and developing algorithms that guarantee to enlarge diversity during training. The proposed metric based on determinantal point processes serves as the first mathematically rigorous definition for diversity. Importantly, the diversity-aware learning algorithms beat the existing state-of-the-art game solvers in terms of exploitability by a large margin. On top of the algorithmic developments, I also contribute two real-world applications of MARL techniques. Specifically, I demonstrate the great potential of applying MARL to study the emergent population dynamics in nature, and model diverse and realistic interactions in autonomous driving. Both applications embody the prospect that MARL techniques could achieve huge impacts in the real physical world, outside of purely video games

COMPLEX ADAPTIVE SYSTEMS AND THE THRESHOLD EFFECT: TOWARDS A GENERAL TOOL FOR STUDYING DYNAMIC PHENOMENA ACROSS DIVERSE DOMAINS

Most interesting phenomena in natural and social systems include transitions and oscillations among their various phases. A new phase begins when the system reaches a threshold that marks a qualitative change in system characteristics. These threshold effects are found all around us. In economics, this could be movement from a bull market to a bear market; in sociology, it could be the spread of political dissent, culminating in rebellion; in biology, the immune response to infection or disease as the body moves from sickness to health. Complex Adaptive Systems (CAS) has proven to be a powerful framework for exploring these and other related phenomena. Our hypothesis is that by modeling differing complex systems we can use the known causes and mechanisms in one domain to gain insight into the controlling properties of similar effects in another domain. To that end, we have created a general CAS model; one that is flexible enough so that it can be individually tailored and mapped to phenomena in various domains, yet retains sufficient commonality across applications to facilitate a deeper, cross-disciplinary understanding of these phenomena. In this work, we focus on the threshold effect. We show that the general model successfully replicates key features of a CAS. And we demonstrate its general applicability by adapting the model to three domains: cancer cells and the immune response; political dissent in a polity; and a marine ecosystem