Neuroevolution in Games: State of the Art and Open Challenges

This paper surveys research on applying neuroevolution (NE) to games. In neuroevolution, artificial neural networks are trained through evolutionary algorithms, taking inspiration from the way biological brains evolved. We analyse the application of NE in games along five different axes, which are the role NE is chosen to play in a game, the different types of neural networks used, the way these networks are evolved, how the fitness is determined and what type of input the network receives. The article also highlights important open research challenges in the field.Comment: - Added more references - Corrected typos - Added an overview table (Table 1

Preference Learning for Move Prediction and Evaluation Function Approximation in Othello

This paper investigates the use of preference learning as an approach to move prediction and evaluation function approximation, using the game of Othello as a test domain. Using the same sets of features, we compare our approach with least squares temporal difference learning, direct classification, and with the Bradley-Terry model, fitted using minorization-maximization (MM). The results show that the exact way in which preference learning is applied is critical to achieving high performance. Best results were obtained using a combination of board inversion and pair-wise preference learning. This combination significantly outperformed the others under test, both in terms of move prediction accuracy, and in the level of play achieved when using the learned evaluation function as a move selector during game play

Evolving Players for an Ancient Game: Hnefatafl

Hnefatafl is an ancient Norse game - an ancestor of chess. In this paper, we report on the development of computer players for this game. In the spirit of Blondie24, we evolve neural networks as board evaluation functions for different versions of the game. An unusual aspect of this game is that there is no general agreement on the rules: it is no longer much played, and game historians attempt to infer the rules from scraps of historical texts, with ambiguities often resolved on gut feeling as to what the rules must have been in order to achieve a balanced game. We offer the evolutionary method as a means by which to judge the merits of alternative rule set

Warm-Start AlphaZero Self-Play Search Enhancements

Recently, AlphaZero has achieved landmark results in deep reinforcement learning, by providing a single self-play architecture that learned three different games at super human level. AlphaZero is a large and complicated system with many parameters, and success requires much compute power and fine-tuning. Reproducing results in other games is a challenge, and many researchers are looking for ways to improve results while reducing computational demands. AlphaZero's design is purely based on self-play and makes no use of labeled expert data ordomain specific enhancements; it is designed to learn from scratch. We propose a novel approach to deal with this cold-start problem by employing simple search enhancements at the beginning phase of self-play training, namely Rollout, Rapid Action Value Estimate (RAVE) and dynamically weighted combinations of these with the neural network, and Rolling Horizon Evolutionary Algorithms (RHEA). Our experiments indicate that most of these enhancements improve the performance of their baseline player in three different (small) board games, with especially RAVE based variants playing strongly

Home Country Effects on Internationalization: Chinese Agrifood Investment in Advanced Economies

Home country effects on internationalization has been conventionally conceived as a contrast to the pull of host countries determinants. While scholarship acknowledges that home country support matters more to internationalizing emerging market multinational enterprises, the focus of extant literature has been underpinned by assumptions of stable macro-level and unidirectional institutional support for the internationalization of firms. This thesis contrasts with previous studies by repositioning the conversation to incorporate the temporal dimension, and investigate the multi-level relationships across institutions, industries and markets in the home country and the varied effects on internationalization. Chinese agrifood investment to advanced economies from 2008 to 2017 against the backdrop of rebalancing and consumption-led growth economy is the phenomenon and research context. The overarching research question is “How do home country effects shape the internationalization of Chinese firms?”. This is addressed in four contextual and case study chapters. Drawing on interdisciplinary literature and applying an abductive research process, I developed a dynamic home country relational model to study the internationalization process of Chinese firms that enriches existing process and institutional frameworks. There are four central findings presented in this thesis. First, home country support engenders different meanings constructed by heterogeneous dispensers and recipients who adopt discretionary selection in a competitive environment. Second, experienced agrifood firms have learned to deliberately avoid controversial farmland purchases and targeted downstream businesses in advanced economies to access resources and gain management skills. Third, wealthy non-agricultural Chinese groups lacking in specialized industry knowledge, face compounded challenges diversifying into agrifood sector and internationalizing simultaneously. Fourth, risk perception and risk mitigation have accentuated as internationalization of Chinese firms evolved, shifting from self-checking to tightening of regulatory controls and reinforced by businesses’ confirmation of support. This study has enhanced the understanding of evolving institutions, and the nuances and irregularity of internationalization processes through the explanation of complex interactions and responses from the perspective of home country actors

Spatial-temporal reasoning applications of computational intelligence in the game of Go and computer networks

Spatial-temporal reasoning is the ability to reason with spatial images or information about space over time. In this dissertation, computational intelligence techniques are applied to computer Go and computer network applications. Among four experiments, the first three are related to the game of Go, and the last one concerns the routing problem in computer networks. The first experiment represents the first training of a modified cellular simultaneous recurrent network (CSRN) trained with cellular particle swarm optimization (PSO). Another contribution is the development of a comprehensive theoretical study of a 2x2 Go research platform with a certified 5 dan Go expert. The proposed architecture successfully trains a 2x2 game tree. The contribution of the second experiment is the development of a computational intelligence algorithm calledcollective cooperative learning (CCL). CCL learns the group size of Go stones on a Go board with zero knowledge by communicating only with the immediate neighbors. An analysis determines the lower bound of a design parameter that guarantees a solution. The contribution of the third experiment is the proposal of a unified system architecture for a Go robot. A prototype Go robot is implemented for the first time in the literature. The last experiment tackles a disruption-tolerant routing problem for a network suffering from link disruption. This experiment represents the first time that the disruption-tolerant routing problem has been formulated with a Markov Decision Process. In addition, the packet delivery rate has been improved under a range of link disruption levels via a reinforcement learning approach --Abstract, page iv

The emergence of clusters in societal transition : a coevolutionary perspective on the TCM cluster at Tonghua/China

New industries are recognized as new impetus to national wealth. At the same time, they are increasingly becoming geographically concentrated in some well defined areas. But current studies on the emergence of industrial clusters tend to analyze favorable driving factors. This dissertation takes the example of a Chinese endogenous industrial cluster, the traditional Chinese medicine (TCM) cluster at Tonghua, a small peripheral city in Northeastern China, to contribute to the theoretical understanding of the emergence of industrial cluster as a co-evolutionary process of organizations, institutions and firms, or, to put it more broadly, as economic evolution embedded in complex socio-economic contexts. The recent advance in evolutionary and co-evolutionary economics which considers the economy and economic landscape as dynamic process instead of equilibrium can be regarded as a part of broader and more intellectual turn of quest for history in social sciences. Although the principle of "history matters" is widely acknowledged, it tends to be reduced to a quite simple concept of "path dependence". However, path dependence cannot offer space for new path creation, except from an external shock. Accordingly, the role of human conscious action or Schumpeterian innovation should be added to path analysis through the concept of path creation. Furthermore, and more importantly, history should be understood as context, and historical context can be explored through the understanding of multi-paths and interaction among them over time. So path inter-dependence (co-evolution between paths) would be useful to better understand the complexity of real history. Since the industrial cluster is composed of interconnected firms and is also subject to changes in institution and technology, I will focus on the multi-way causal relationship between firm, institution and technology. The theorizing is not entirely new, but most of the theoretical and empirical discussions are at the national or industrial level, not regional or local one. A competitive cluster can be regarded as a co-evolutionary hotspot in which multiple populations actively interact and are interconnected. Co-evolution itself is a dynamic and evolutionary process. So I will adopt a dynamic and evolutionary view to examine co-evolutionary degree or co-evolutionary effects in the Tonghua pharmaceutical cluster through time. After a brief introduction which deals with the national institutional changes that are highly associated with new venture creation, entrepreneurship, and innovation, with registrations on drug and healthcare system, and with changes in market demand of China’s pharmaceutical industry and geographical distribution, I will collect evidences from three aspects based upon field survey and second hand data, i.e., the history of the enterprises, the origin of entrepreneurship, and the knowledge of evolution, linking their respective generative relationships through the genealogical method. In this volume, the evolution of the Tonghua pharmaceutical firm organization, the formation of local entrepreneurship, historical accumulation of knowledge, and particular knowledge of transfer among generations of firms will be discussed, then I will probe into co-adaption and co-evolution between local formal and informal institutions and organizations in Tonghua’s TCM industry. In addition, I will try to understand the co-evolutionary process at different geographical levels (namely, national and local). In summary, my main findings include the following several points. Firstly, in the course of the emergence of Tonghua’s pharmaceutical industry, local social networks and the traditional alliance between enterprises and government have played important roles. Secondly, the most important factor that influences the evolution of endogenous industrial clusters such as the Tonghua pharmaceutical industry in transitional countries is not the change in technology, but the change in fundamental national institutions. Thirdly, the success of the Tonghua pharmaceutical industry can be ascribed to the creation of multiple paths largely based on initial conditions, which implies that economic policy should have historical consciousness, namely, new economic innovation should make full use of both historical legacies and existing assets. Finally, it is co-adaption and co-selection of firm organization, institution, and technology that have jointly made Tonghua’s pharmaceutical industry become highly competitive, which means that whether one region can grasp new opportunities partially depends on its capabilities to coordinate a varity of development agents.Neue Industrien werden im Allgemeinen als Impuls der Entwicklung zu nationalem Wohlstand verstanden. Zugleich sind sie überwiegend an einigen geographisch genau definierten Orten konzentriert. Aktuelle Studien zur Emergenz dieser Industrie-Cluster neigen dazu, entsprechende begünstigende Faktoren zu analysieren. Mit dem Beispiel eines endogenen Clusters in China, dem Cluster der Traditionellen Chinesischen Medizin (TCM) in Tonghua, will diese Dissertation zum theoretischen Verständnis der Emergenz von Industrie-Clustern unter der Perspektive eines ko-evolutorischen Prozesses von Form der Organisation, Institutionen und Unternehmen beitragen. Oder, um es etwas breiter auszudrücken, diese Emergenz als ökonomische Evolution zu verstehen, die in einen komplexen sozio-ökonomischen Kontext eingebettet ist. Obgleich der Vorstellung, Geschichte habe eine Bedeutung („history matters“), überwiegend in der Forschung zugestimmt wird, bleibt diese oft auf das Konzept der Pfadabhängigkeit beschränkt. Das aber eröffnet keinen Raum für die Betrachtung endogener Pfad-Bildung. Dem Konzept der Pfad-Bildung entsprechend sollte jedoch die Pfadanalyse ergänzt werden um bewusste Handlungen des Menschen oder auch um Innovationen im Schumpeterschen Sinn. Wichtiger ist außerdem, dass Geschichte als ein Kontext verstanden werden sollte, in dem mehrere Pfade ko-existieren und im Zeitverlauf auch interagieren. So wäre ein Konzept der Pfad-Interdependenz (oder der Ko-Evolution von Pfaden) nützlich zum besseren Verständnis der Komplexität „wirklicher“ Geschichte. Weil das Industriecluster sich aus untereinander verflochtenen Unternehmen zusammen setzt und zugleich Gegenstand von Änderungen in den Institutionen und der Technologie ist, konzentriert sich die Dissertation auf vielseitige kausale Beziehungen von Unternehmen, Institutionen und Technologie. Ein wettbewerbsfähiges Cluster kann aus geographischer Sicht als ein „hot spot“ der Ko-evolution betrachtet werden, in dem verschiedenartige Populationen aktiv untereinander agieren und daher miteinander verflochten sind. Ko-Evolution selbst ist dann ein dynamischer und evolutorischer Prozess. Die Arbeit wählt diese Perspektive, um das Maß und die Wirkungen der Ko-Evolution im Pharma-Cluster von Tonghua im Zeitverlauf zu analysieren. Die Dissertation fußt auf empirischen Erhebungen, ergänzt um eine Dokumenten-Analyse, zur Geschichte der Unternehmen, der Herkunft der Unternehmerschaft sowie der Evolution von Wissen. Sie diskutiert die Evolution in den Organisationsformen der Pharma-Unternehmen in Tonghua, die Bildung einer lokalen Unternehmerschaft, die historische Akkumulation von Wissen und den besonderen Wissenstransfer zwischen Generationen von Unternehmen. Schließlich untersucht sie die Ko-Adaption und Ko-Evolution von lokalen formalen und informellen Institutionen und Organisationen der TCM-Industrie in Tonghua. Die folgenden Punkte betreffen die wichtigsten Ergebnisse der Dissertation: Erstens haben sehr langfristige und dichte lokale soziale Netzwerke eine erhebliche Rolle im Lauf der Emergenz der Pharma-Industrie in Tonghua gespielt. Zweitens ist der wichtigste Faktor in der Pharma-Industrie nicht im technologischen Fortschritt durch Anstrengungen bei Forschung und Entwicklung (FuE) zu sehen, sondern im institutionellen Wandel sowohl auf nationaler als auch auf lokaler Ebene. Drittens kann der Erfolg der Pharma-Industrie in Tonghua der Bildung multipler Pfade zugeschrieben werden, die auf bestimmten Anfangsbedingungen gründen. Das bedeutet, dass die neue ökonomische Entwicklungspolitik sowohl das historische Erbe als auch bestehende Aktivposten in vollem Umfang nutzen sollte. Schließlich ist festzustellen, dass Ko-Adaption und Ko-Selektion der Unternehmens-Organisation, von Institutionen und Technologie zusammen die Pharma-Industrie von Tonghua in hohem Maße wettbewerbsfähig gemacht haben. Ob eine Region neue Gelegenheiten ergreifen kann, hängt folglich teilweise von ihrer Fähigkeit ab, eine Vielfalt von Entwicklungs-Agenten zu koordinieren