Glosarium Matematika

273 p.; 24 cm

Many-agent Reinforcement Learning

Multi-agent reinforcement learning (RL) solves the problem of how each agent should behave optimally in a stochastic environment in which multiple agents are learning simultaneously. It is an interdisciplinary domain with a long history that lies in the joint area of psychology, control theory, game theory, reinforcement learning, and deep learning. Following the remarkable success of the AlphaGO series in single-agent RL, 2019 was a booming year that witnessed significant advances in multi-agent RL techniques; impressive breakthroughs have been made on developing AIs that outperform humans on many challenging tasks, especially multi-player video games. Nonetheless, one of the key challenges of multi-agent RL techniques is the scalability; it is still non-trivial to design efficient learning algorithms that can solve tasks including far more than two agents ($N \gg 2$), which I name by \emph{many-agent reinforcement learning} (MARL\footnote{I use the world of ``MARL" to denote multi-agent reinforcement learning with a particular focus on the cases of many agents; otherwise, it is denoted as ``Multi-Agent RL" by default.}) problems. In this thesis, I contribute to tackling MARL problems from four aspects. Firstly, I offer a self-contained overview of multi-agent RL techniques from a game-theoretical perspective. This overview fills the research gap that most of the existing work either fails to cover the recent advances since 2010 or does not pay adequate attention to game theory, which I believe is the cornerstone to solving many-agent learning problems. Secondly, I develop a tractable policy evaluation algorithm -- $\alpha^\alpha$-Rank -- in many-agent systems. The critical advantage of $\alpha^\alpha$-Rank is that it can compute the solution concept of $\alpha$-Rank tractably in multi-player general-sum games with no need to store the entire pay-off matrix. This is in contrast to classic solution concepts such as Nash equilibrium which is known to be $PPAD$-hard in even two-player cases. $\alpha^\alpha$-Rank allows us, for the first time, to practically conduct large-scale multi-agent evaluations. Thirdly, I introduce a scalable policy learning algorithm -- mean-field MARL -- in many-agent systems. The mean-field MARL method takes advantage of the mean-field approximation from physics, and it is the first provably convergent algorithm that tries to break the curse of dimensionality for MARL tasks. With the proposed algorithm, I report the first result of solving the Ising model and multi-agent battle games through a MARL approach. Fourthly, I investigate the many-agent learning problem in open-ended meta-games (i.e., the game of a game in the policy space). Specifically, I focus on modelling the behavioural diversity in meta-games, and developing algorithms that guarantee to enlarge diversity during training. The proposed metric based on determinantal point processes serves as the first mathematically rigorous definition for diversity. Importantly, the diversity-aware learning algorithms beat the existing state-of-the-art game solvers in terms of exploitability by a large margin. On top of the algorithmic developments, I also contribute two real-world applications of MARL techniques. Specifically, I demonstrate the great potential of applying MARL to study the emergent population dynamics in nature, and model diverse and realistic interactions in autonomous driving. Both applications embody the prospect that MARL techniques could achieve huge impacts in the real physical world, outside of purely video games

Enhancing remanufacturing automation using deep learning approach

In recent years, remanufacturing has significant interest from researchers and practitioners to improve efficiency through maximum value recovery of products at end-of-life (EoL). It is a process of returning used products, known as EoL products, to as-new condition with matching or higher warranty than the new products. However, these remanufacturing processes are complex and time-consuming to implement manually, causing reduced productivity and posing dangers to personnel. These challenges require automating the various remanufacturing process stages to achieve higher throughput, reduced lead time, cost and environmental impact while maximising economic gains. Besides, as highlighted by various research groups, there is currently a shortage of adequate remanufacturing-specific technologies to achieve full automation. -- This research explores automating remanufacturing processes to improve competitiveness by analysing and developing deep learning-based models for automating different stages of the remanufacturing processes. Analysing deep learning algorithms represents a viable option to investigate and develop technologies with capabilities to overcome the outlined challenges. Deep learning involves using artificial neural networks to learn high-level abstractions in data. Deep learning (DL) models are inspired by human brains and have produced state-of-the-art results in pattern recognition, object detection and other applications. The research further investigates the empirical data of torque converter components recorded from a remanufacturing facility in Glasgow, UK, using the in-case and cross-case analysis to evaluate the remanufacturing inspection, sorting, and process control applications. -- Nevertheless, the developed algorithm helped capture, pre-process, train, deploy and evaluate the performance of the respective processes. The experimental evaluation of the in-case and cross-case analysis using model prediction accuracy, misclassification rate, and model loss highlights that the developed models achieved a high prediction accuracy of above 99.9% across the sorting, inspection and process control applications. Furthermore, a low model loss between 3x10-3 and 1.3x10-5 was obtained alongside a misclassification rate that lies between 0.01% to 0.08% across the three applications investigated, thereby highlighting the capability of the developed deep learning algorithms to perform the sorting, process control and inspection in remanufacturing. The results demonstrate the viability of adopting deep learning-based algorithms in automating remanufacturing processes, achieving safer and more efficient remanufacturing. -- Finally, this research is unique because it is the first to investigate using deep learning and qualitative torque-converter image data for modelling remanufacturing sorting, inspection and process control applications. It also delivers a custom computational model that has the potential to enhance remanufacturing automation when utilised. The findings and publications also benefit both academics and industrial practitioners. Furthermore, the model is easily adaptable to other remanufacturing applications with minor modifications to enhance process efficiency in today's workplaces.In recent years, remanufacturing has significant interest from researchers and practitioners to improve efficiency through maximum value recovery of products at end-of-life (EoL). It is a process of returning used products, known as EoL products, to as-new condition with matching or higher warranty than the new products. However, these remanufacturing processes are complex and time-consuming to implement manually, causing reduced productivity and posing dangers to personnel. These challenges require automating the various remanufacturing process stages to achieve higher throughput, reduced lead time, cost and environmental impact while maximising economic gains. Besides, as highlighted by various research groups, there is currently a shortage of adequate remanufacturing-specific technologies to achieve full automation. -- This research explores automating remanufacturing processes to improve competitiveness by analysing and developing deep learning-based models for automating different stages of the remanufacturing processes. Analysing deep learning algorithms represents a viable option to investigate and develop technologies with capabilities to overcome the outlined challenges. Deep learning involves using artificial neural networks to learn high-level abstractions in data. Deep learning (DL) models are inspired by human brains and have produced state-of-the-art results in pattern recognition, object detection and other applications. The research further investigates the empirical data of torque converter components recorded from a remanufacturing facility in Glasgow, UK, using the in-case and cross-case analysis to evaluate the remanufacturing inspection, sorting, and process control applications. -- Nevertheless, the developed algorithm helped capture, pre-process, train, deploy and evaluate the performance of the respective processes. The experimental evaluation of the in-case and cross-case analysis using model prediction accuracy, misclassification rate, and model loss highlights that the developed models achieved a high prediction accuracy of above 99.9% across the sorting, inspection and process control applications. Furthermore, a low model loss between 3x10-3 and 1.3x10-5 was obtained alongside a misclassification rate that lies between 0.01% to 0.08% across the three applications investigated, thereby highlighting the capability of the developed deep learning algorithms to perform the sorting, process control and inspection in remanufacturing. The results demonstrate the viability of adopting deep learning-based algorithms in automating remanufacturing processes, achieving safer and more efficient remanufacturing. -- Finally, this research is unique because it is the first to investigate using deep learning and qualitative torque-converter image data for modelling remanufacturing sorting, inspection and process control applications. It also delivers a custom computational model that has the potential to enhance remanufacturing automation when utilised. The findings and publications also benefit both academics and industrial practitioners. Furthermore, the model is easily adaptable to other remanufacturing applications with minor modifications to enhance process efficiency in today's workplaces

Multi-dimensional local binary pattern texture descriptors and their application for medical image analysis

Texture can be broadly stated as spatial variation of image intensities. Texture analysis and classification is a well researched area for its importance to many computer vision applications. Consequently, much research has focussed on deriving powerful and efficient texture descriptors. Local binary patterns (LBP) and its variants are simple yet powerful texture descriptors. LBP features describe the texture neighbourhood of a pixel using simple comparison operators, and are often calculated based on varying neighbourhood radii to provide multi-resolution texture descriptions. A comprehensive evaluation of different LBP variants on a common benchmark dataset is missing in the literature. This thesis presents the performance for different LBP variants on texture classification and retrieval tasks. The results show that multi-scale local binary pattern variance (LBPV) gives the best performance over eight benchmarked datasets. Furthermore, improvements to the Dominant LBP (D-LBP) by ranking dominant patterns over complete training set and Compound LBP (CM-LBP) by considering 16 bits binary codes are suggested which are shown to outperform their original counterparts. The main contribution of the thesis is the introduction of multi-dimensional LBP features, which preserve the relationships between different scales by building a multi-dimensional histogram. The results on benchmarked classification and retrieval datasets clearly show that the multi-dimensional LBP (MD-LBP) improves the results compared to conventional multi-scale LBP. The same principle is applied to LBPV (MD-LBPV), again leading to improved performance. The proposed variants result in relatively large feature lengths which is addressed using three different feature length reduction techniques. Principle component analysis (PCA) is shown to give the best performance when the feature length is reduced to match that of conventional multi-scale LBP. The proposed multi-dimensional LBP variants are applied for medical image analysis application. The first application is nailfold capillary (NC) image classification. Performance of MD-LBPV on NC images is highest, whereas for second application, HEp-2 cell classification, performance of MD-LBP is highest. It is observed that the proposed texture descriptors gives improved texture classification accuracy

Aspects of Signal Processing in Noisy Neurons

In jüngerer Zeit hat sich die Erkenntnis durchgesetzt, daß statistische Einflüsse, oft Rauschen genannt, die Verarbeitung von Signalen nicht notwendig behindern, sondern unterstützen können. Dieser Effekt ist als stochastische Resonanz bekannt geworden. Es liegt nahe, daß die Evolution Wege gefunden hat, diese Phänomen zur Optimierung der Informationsverarbeitung im Nervensystem auszunutzen. Diese Dissertation untersucht am Beispiel des pulserzeugenden Integratorneurons mit Leckstrom, ob die Kodierung periodischer Signale in Neuronen durch das ohnehin im Nervensystem vorhandene Rauschen verbessert wird. Die Untersuchung erfolgt mit den Methoden der Theorie der Punktprozesse. Die Verteilung der Intervalle zwischen zwei beliebigen aufeinanderfolgenden Pulsen, die das Neuron aussendet, wird aus einem Integralgleichungsansatz numerisch bestimmt und die zeitliche Ordnung der Pulsfolgen relativ zum periodischen Signal als Markoffkette beschrieben. Daneben werden einige Näherungsmodelle für die Pulsintervallverteilung, die weitergehende analytische Untersuchungen erlauben, vorgestellt und ihre Zuverlässigkeit geprüft. Als wesentliches Ergebnis wird gezeigt, daß im Modellneuron zwei Arten rauschinduzierter Resonanz auftreten: zum einen klassiche stochastische Resonanz, d.h. ein optimales Signal-Rausch-Verhältnis der evozierten Pulsfolge bei einer bestimmten Amplitude des Eingangsrauschens. Hinzu tritt eine Resonanz bezüglich der Frequenz des Eingangssignals oder Reizes. Reize eines bestimmten Frequenzbereichs werden in Pulsfolgen kodiert, die zeitlich deutlich strukturiert sind, währ! end Stimuli außerhalb des bevorzugten Frequenzbandes zeitlich homogenere Pulsfolgen auslösen. Für diese zweifache Resonanz wird der Begriff stochastische Doppelresonanz eingeführt. Der Effekt wird auf elementare Mechanismen zurückgeführt und seine Abhängigkeit von den Eigenschaften des Reizes umfassend untersucht. Dabei zeigt sich ,daß die Reizantwort des Neurons einfachen Skalengesetzen unterliegt. Insbesondere ist die optimale skalierte Rauschamplitude ein universeller Parameter des Modells, der vom Reiz unabhängig zu sein scheint. Die optimale Reizfrequenz hängt hingegen linear von der skalierten Reizamplitude ab, wobei die Proportionalitätskonstante vom Gleichstromanteil des Reizes bestimmt wird (Basisstrom). Während große Basisströme Frequenz und Amplitude nahezu entkoppeln, so daß Reize beliebiger Amplitude in zeitlich wohlstrukturierten Pulsfolgen kodiert werden, erlauben es kleine Basisströme, das optimale Frequenzband durch Veränderung der Reizamplitude zu wählen

Identifying the molecular components that matter: a statistical modelling approach to linking functional genomics data to cell physiology

Functional genomics technologies, in which thousands of mRNAs, proteins, or metabolites can be measured in single experiments, have contributed to reshape biological investigations. One of the most important issues in the analysis of the generated large datasets is the selection of relatively small sub-sets of variables that are predictive of the physiological state of a cell or tissue. In this thesis, a truly multivariate variable selection framework using diverse functional genomics data has been developed, characterized, and tested. This framework has also been used to prove that it is possible to predict the physiological state of the tumour from the molecular state of adjacent normal cells. This allows us to identify novel genes involved in cell to cell communication. Then, using a network inference technique networks representing cell-cell communication in prostate cancer have been inferred. The analysis of these networks has revealed interesting properties that suggests a crucial role of directional signals in controlling the interplay between normal and tumour cell to cell communication. Experimental verification performed in our laboratory has provided evidence that one of the identified genes could be a novel tumour suppressor gene. In conclusion, the findings and methods reported in this thesis have contributed to further understanding of cell to cell interaction and multivariate variable selection not only by applying and extending previous work, but also by proposing novel approaches that can be applied to any functional genomics data