Efficient machine learning: models and accelerations

One of the key enablers of the recent unprecedented success of machine learning is the adoption of very large models. Modern machine learning models typically consist of multiple cascaded layers such as deep neural networks, and at least millions to hundreds of millions of parameters (i.e., weights) for the entire model. The larger-scale model tend to enable the extraction of more complex high-level features, and therefore, lead to a significant improvement of the overall accuracy. On the other side, the layered deep structure and large model sizes also demand to increase computational capability and memory requirements. In order to achieve higher scalability, performance, and energy efficiency for deep learning systems, two orthogonal research and development trends have attracted enormous interests. The first trend is the acceleration while the second is the model compression. The underlying goal of these two trends is the high quality of the models to provides accurate predictions. In this thesis, we address these two problems and utilize different computing paradigms to solve real-life deep learning problems. To explore in these two domains, this thesis first presents the cogent confabulation network for sentence completion problem. We use Chinese language as a case study to describe our exploration of the cogent confabulation based text recognition models. The exploration and optimization of the cogent confabulation based models have been conducted through various comparisons. The optimized network offered a better accuracy performance for the sentence completion. To accelerate the sentence completion problem in a multi-processing system, we propose a parallel framework for the confabulation recall algorithm. The parallel implementation reduce runtime, improve the recall accuracy by breaking the fixed evaluation order and introducing more generalization, and maintain a balanced progress in status update among all neurons. A lexicon scheduling algorithm is presented to further improve the model performance. As deep neural networks have been proven effective to solve many real-life applications, and they are deployed on low-power devices, we then investigated the acceleration for the neural network inference using a hardware-friendly computing paradigm, stochastic computing. It is an approximate computing paradigm which requires small hardware footprint and achieves high energy efficiency. Applying this stochastic computing to deep convolutional neural networks, we design the functional hardware blocks and optimize them jointly to minimize the accuracy loss due to the approximation. The synthesis results show that the proposed design achieves the remarkable low hardware cost and power/energy consumption. Modern neural networks usually imply a huge amount of parameters which cannot be fit into embedded devices. Compression of the deep learning models together with acceleration attracts our attention. We introduce the structured matrices based neural network to address this problem. Circulant matrix is one of the structured matrices, where a matrix can be represented using a single vector, so that the matrix is compressed. We further investigate a more flexible structure based on circulant matrix, called block-circulant matrix. It partitions a matrix into several smaller blocks and makes each submatrix is circulant. The compression ratio is controllable. With the help of Fourier Transform based equivalent computation, the inference of the deep neural network can be accelerated energy efficiently on the FPGAs. We also offer the optimization for the training algorithm for block circulant matrices based neural networks to obtain a high accuracy after compression

Cognition-based approaches for high-precision text mining

This research improves the precision of information extraction from free-form text via the use of cognitive-based approaches to natural language processing (NLP). Cognitive-based approaches are an important, and relatively new, area of research in NLP and search, as well as linguistics. Cognitive approaches enable significant improvements in both the breadth and depth of knowledge extracted from text. This research has made contributions in the areas of a cognitive approach to automated concept recognition in. Cognitive approaches to search, also called concept-based search, have been shown to improve search precision. Given the tremendous amount of electronic text generated in our digital and connected world, cognitive approaches enable substantial opportunities in knowledge discovery. The generation and storage of electronic text is ubiquitous, hence opportunities for improved knowledge discovery span virtually all knowledge domains. While cognition-based search offers superior approaches, challenges exist due to the need to mimic, even in the most rudimentary way, the extraordinary powers of human cognition. This research addresses these challenges in the key area of a cognition-based approach to automated concept recognition. In addition it resulted in a semantic processing system framework for use in applications in any knowledge domain. Confabulation theory was applied to the problem of automated concept recognition. This is a relatively new theory of cognition using a non-Bayesian measure, called cogency, for predicting the results of human cognition. An innovative distance measure derived from cogent confabulation and called inverse cogency, to rank order candidate concepts during the recognition process. When used with a multilayer perceptron, it improved the precision of concept recognition by 5% over published benchmarks. Additional precision improvements are anticipated. These research steps build a foundation for cognition-based, high-precision text mining. Long-term it is anticipated that this foundation enables a cognitive-based approach to automated ontology learning. Such automated ontology learning will mimic human language cognition, and will, in turn, enable the practical use of cognitive-based approaches in virtually any knowledge domain --Abstract, page iii

The contribution of executive dysfunction to memory impairment and confabulation in schizophrenia

Study 1. Using a cognitive-process approach, 25 schizophrenic patients were matched with 25 healthy volunteers and compared on tests of memory and executive function. The schizophrenia group was found to have a significant impairment in immediate memory with relatively spared long-delay and recognition memory. Memory deficits were irrespective of the encoding strategies used and were unrelated to chronicity. In addition, the schizophrenic patients performed worse than controls on tests of executive function which was supported by some significant correlations between aspects of memory and executive function. The pattern of performance resembled that found in patients with subcortical or frontal lesions. Study 2. To examine further executive aspects of memory, an attempt to demonstrate confabulation in schizophrenia was made. Twelve schizophrenic patients were matched with 12 volunteers, 8 of whom were normal healthy subjects, with the remained being depressed patients. The subjects were asked to recall a set of experimental narratives, with confabulation being defined as the recall of ideas not present in the narrative. Subjects were also examined on a number of neuropsychological tests and the patients were assessed on the Krawiecka scale. Variable amounts of confabulation were observed in all the schizophrenic patients while only one control subject confabulated. The form of confabulation differed from those observed in other patients in that the original ideas were spontaneously rearranged to produce new ones. Confabulation was found to be related to difficulties in suppressing inappropriate responses and formal thought disorder. Study 3. Three schizophrenic patients previously identified as confabulators, were intensively studied to establish the mechanisms of narrative confabulation in schizophrenia. Patients were administered experimental tasks as well as standard neuropsychological tests of memory and executive function. Assessment of current symptoms was made using the SANS and SAPS scales. The severity of cognitive impairment was found to reflect the severity of confabulation, but memory impairment was neither nor sufficient to account for confabulation. Within the spectrum of executive deficits, impairments in response suppression and response monitoring, but not planning or generation were consistently associated with confabulation. The findings from the experimental tasks suggest that faults occur at both input and output. At the input stage, narrative material is encoded in a disorganised manner while at the output stage, this disorganisation is compounded by faulty editing processes. Study 4. Four schizophrenic patients who were known confabulators with narrative material, were subjected to an experimental autobiographical questionnaire designed to establish whether schizophrenic patients confabulate in response to questions calling on the recollection of personal facts and events. In addition, a number of neuropsychological tests were administered and current symptoms was assessed with the SANS and SAPS scales. All patients were observed to confabulate to varying degrees, particularly in response to questions relating to personal episodes rather than facts. For two patients, personal delusional systems were found to play a role in confabulation by providing a framework on which to base certain confabulatory recollections. Memory impairment was not found to be a necessary component to autobiographical confabulation but deficits in response suppression and response monitoring were observed to be related to the verification process performed during this task. Study 5. In an attempt to establish which anatomical regions may be at fault in schizophrenia when patients are engaged in response suppression tasks, six normal subjects were studied using positron emission tomography (PET) to identify anatomical regions involved when performing the Hayling Test. Subjects were also required to perform a control condition in which they had to read out the last word of given sentences. Compared to the control task, response initiation was associated with left sided activation of the frontal operculum, inferior frontal gyrus, middle temporal gyrus and right anterior cingulate gyrus, whereas response suppression was associated with left frontal operculum, inferior frontal gyrus and right anterior cingulate gyrus activation only. The difference between the two parts of the Hayling Test was in the increased activation of the left middle temporal gyrus and the left inferior frontal region (Brodmann's area 44/6) during response initiation

Neuromorphic Learning Systems for Supervised and Unsupervised Applications

The advancements in high performance computing (HPC) have enabled the large-scale implementation of neuromorphic learning models and pushed the research on computational intelligence into a new era. Those bio-inspired models are constructed on top of unified building blocks, i.e. neurons, and have revealed potentials for learning of complex information. Two major challenges remain in neuromorphic computing. Firstly, sophisticated structuring methods are needed to determine the connectivity of the neurons in order to model various problems accurately. Secondly, the models need to adapt to non-traditional architectures for improved computation speed and energy efficiency. In this thesis, we address these two problems and apply our techniques to different cognitive applications. This thesis first presents the self-structured confabulation network for anomaly detection. Among the machine learning applications, unsupervised detection of the anomalous streams is especially challenging because it requires both detection accuracy and real-time performance. Designing a computing framework that harnesses the growing computing power of the multicore systems while maintaining high sensitivity and specificity to the anomalies is an urgent research need. We present AnRAD (Anomaly Recognition And Detection), a bio-inspired detection framework that performs probabilistic inferences. We leverage the mutual information between the features and develop a self-structuring procedure that learns a succinct confabulation network from the unlabeled data. This network is capable of fast incremental learning, which continuously refines the knowledge base from the data streams. Compared to several existing anomaly detection methods, the proposed approach provides competitive detection accuracy as well as the insight to reason the decision making. Furthermore, we exploit the massive parallel structure of the AnRAD framework. Our implementation of the recall algorithms on the graphic processing unit (GPU) and the Xeon Phi co-processor both obtain substantial speedups over the sequential implementation on general-purpose microprocessor (GPP). The implementation enables real-time service to concurrent data streams with diversified contexts, and can be applied to large problems with multiple local patterns. Experimental results demonstrate high computing performance and memory efficiency. For vehicle abnormal behavior detection, the framework is able to monitor up to 16000 vehicles and their interactions in real-time with a single commodity co-processor, and uses less than 0.2ms for each testing subject. While adapting our streaming anomaly detection model to mobile devices or unmanned systems, the key challenge is to deliver required performance under the stringent power constraint. To address the paradox between performance and power consumption, brain-inspired hardware, such as the IBM Neurosynaptic System, has been developed to enable low power implementation of neural models. As a follow-up to the AnRAD framework, we proposed to port the detection network to the TrueNorth architecture. Implementing inference based anomaly detection on a neurosynaptic processor is not straightforward due to hardware limitations. A design flow and the supporting component library are developed to flexibly map the learned detection networks to the neurosynaptic cores. Instead of the popular rate code, burst code is adopted in the design, which represents numerical value using the phase of a burst of spike trains. This does not only reduce the hardware complexity, but also increases the result\u27s accuracy. A Corelet library, NeoInfer-TN, is implemented for basic operations in burst code and two-phase pipelines are constructed based on the library components. The design can be configured for different tradeoffs between detection accuracy, hardware resource consumptions, throughput and energy. We evaluate the system using network intrusion detection data streams. The results show higher detection rate than some conventional approaches and real-time performance, with only 50mW power consumption. Overall, it achieves 10^8 operations per Joule. In addition to the modeling and implementation of unsupervised anomaly detection, we also investigate a supervised learning model based on neural networks and deep fragment embedding and apply it to text-image retrieval. The study aims at bridging the gap between image and natural language. It continues to improve the bidirectional retrieval performance across the modalities. Unlike existing works that target at single sentence densely describing the image objects, we elevate the topic to associating deep image representations with noisy texts that are only loosely correlated. Based on text-image fragment embedding, our model employs a sequential configuration, connects two embedding stages together. The first stage learns the relevancy of the text fragments, and the second stage uses the filtered output from the first one to improve the matching results. The model also integrates multiple convolutional neural networks (CNN) to construct the image fragments, in which rich context information such as human faces can be extracted to increase the alignment accuracy. The proposed method is evaluated with both synthetic dataset and real-world dataset collected from picture news website. The results show up to 50% ranking performance improvement over the comparison models

Hybrid discourse modeling and summarization for a speech-to-speech translation system

The thesis discusses two parts of the speech-to-speech translation system VerbMobil: the dialogue model and one of its applications, multilingual summary generation. In connection with the dialogue model, two topics are of special interest: (a) the use of a default unification operation called overlay as the fundamental operation for dialogue management; and (b) an intentional model that is able to describe intentions in dialogue on five levels in a language-independent way. Besides the actual generation algorithm developed, we present a comprehensive evaluation of the summarization functionality. In addition to precision and recall, a new characterization - confabulation - is defined that provides a more precise understanding of the performance of complex natural language processing systems.Die vorliegende Arbeit behandelt hauptsächlich zwei Themen, die für das VerbMobil-System, ein Übersetzungssystem gesprochener Spontansprache, entwickelt wurden: das Dialogmodell und als Applikation die multilinguale Generierung von Ergebnissprotokollen. Für die Dialogmodellierung sind zwei Themen von besonderem Interesse. Das erste behandelt eine in der vorliegenden Arbeit formalisierte Default-Unifikations-Operation namens Overlay, die als fundamentale Operation für Diskursverarbeitung dient. Das zweite besteht aus einem intentionalen Modell, das Intentionen eines Dialogs auf fünf Ebenen in einer sprachunabhängigen Repräsentation darstellt. Neben dem für die Protokollgenerierung entwickelten Generierungsalgorithmus wird eine umfassende Evaluation zur Protokollgenerierungsfunktionalität vorgestellt. Zusätzlich zu "precision" und "recall" wird ein neues Maß - Konfabulation (Engl.: "confabulation") - vorgestellt, das eine präzisere Charakterisierung der Qualität eines komplexen Sprachverarbeitungssystems ermöglicht

How the Mind Refigures Memory: The Role of Social Construction and Fallibility in the Fictions of Faulkner, Woolf, and Nabokov

This thesis argues that some literary works of William Faulkner, Virginia Woolf, and Vladimir Nabokov both engage and represent fictional memory and support certain claims made by memorial studies that explain memories as coming into existence through a dynamic process, being transformed from their original state to incorporate knowledge learned at a time later than that of the memory’s formation. The thesis examines how it is that the mind is socially conditioned into a predetermined notion of reality, maintained by collective memory. This conditioning takes place at the onset of memory formation and results in limiting the mind to a finite number of memories. Rather than continuously creating new memories, the mind compiles very few memories that conform to social reality. This aggregate effect creates the allusion that new memories are created throughout life; whereas, the idea of a new memory is actually synonymous with a product of the imagination, a product that is limited in most after a certain point in development. Faulkner’s As I Lay Dying and The Sound and the Fury exhibit the mnemonic processes of association. This thesis shows that memorial association, while helping to strengthen long-term memories, directly causes confabulation; however, what these texts, along with Faulkner’s Light in August and Nabokov’s Speak, Memory, and Virginia Woolf’s To the Lighthouse also demonstrate is a questioning of a learned notion of reality. I argue that this reality is an entirely subjective construct and one that prevents certain experiences from becoming memories

Hybrid discourse modeling and summarization for a speech-to-speech translation system

The thesis discusses two parts of the speech-to-speech translation system VerbMobil: the dialogue model and one of its applications, multilingual summary generation. In connection with the dialogue model, two topics are of special interest: (a) the use of a default unification operation called overlay as the fundamental operation for dialogue management; and (b) an intentional model that is able to describe intentions in dialogue on five levels in a language-independent way. Besides the actual generation algorithm developed, we present a comprehensive evaluation of the summarization functionality. In addition to precision and recall, a new characterization - confabulation - is defined that provides a more precise understanding of the performance of complex natural language processing systems.Die vorliegende Arbeit behandelt hauptsächlich zwei Themen, die für das VerbMobil-System, ein Übersetzungssystem gesprochener Spontansprache, entwickelt wurden: das Dialogmodell und als Applikation die multilinguale Generierung von Ergebnissprotokollen. Für die Dialogmodellierung sind zwei Themen von besonderem Interesse. Das erste behandelt eine in der vorliegenden Arbeit formalisierte Default-Unifikations-Operation namens Overlay, die als fundamentale Operation für Diskursverarbeitung dient. Das zweite besteht aus einem intentionalen Modell, das Intentionen eines Dialogs auf fünf Ebenen in einer sprachunabhängigen Repräsentation darstellt. Neben dem für die Protokollgenerierung entwickelten Generierungsalgorithmus wird eine umfassende Evaluation zur Protokollgenerierungsfunktionalität vorgestellt. Zusätzlich zu "precision" und "recall" wird ein neues Maß - Konfabulation (Engl.: "confabulation") - vorgestellt, das eine präzisere Charakterisierung der Qualität eines komplexen Sprachverarbeitungssystems ermöglicht

Neuromorphic Systems for Pattern Recognition and Uav Trajectory Planning

Detection and control are two essential components in an intelligent system. This thesis investigates novel techniques in both areas with a focus on the applications of handwritten text recognition and UAV flight control. Recognizing handwritten texts is a challenging task due to many different writing styles and lack of clear boundary between adjacent characters. The difficulty is greatly increased if the detection algorithms is solely based on pattern matching without information of dynamics of handwriting trajectories. Motivated by the aforementioned challenges, this thesis first investigates the pattern recognition problem. We use offline handwritten texts recognition as a case study to explore the performance of a recurrent belief propagation model. We first develop a probabilistic inference network to post process the recognition results of deep Convolutional Neural Network (CNN) (e.g. LeNet) and collect individual characters to form words. The output of the inference network is a set of words and their probability. A series of post processing and improvement techniques are then introduced to further increase the recognition accuracy. We study the performance of proposed model through various comparisons. The results show that it significantly improves the accuracy by correcting deletion, insertion and replacement errors, which are the main sources of invalid candidate words. Deep Reinforcement Learning (DRL) has widely been applied to control the autonomous systems because it provides solutions for various complex decision-making tasks that previously could not be solved solely with deep learning. To enable autonomous Unmanned Aerial Vehicles (UAV), this thesis presents a two-level trajectory planning framework for UAVs in an indoor environment. A sequence of waypoints is selected at the higher-level, which leads the UAV from its current position to the destination. At the lower-level, an optimal trajectory is generated analytically between each pair of adjacent waypoints. The goal of trajectory generation is to maintain the stability of the UAV, and the goal of the waypoints planning is to select waypoints with the lowest control thrust throughout the entire trip while avoiding collisions with obstacles. The entire framework is implemented using DRL, which learns the highly complicated and nonlinear interaction between those two levels, and the impact from the environment. Given the pre-planned trajectory, this thesis further presents an actor-critic reinforcement learning framework that realizes continuous trajectory control of the UAV through a set of desired waypoints. We construct a deep neural network and develop reinforcement learning for better trajectory tracking. In addition, Field Programmable Gate Arrays (FPGA) based hardware acceleration is designed for energy efficient real-time control. If we are to integrate the trajectory planning model onto a UAV system for real-time on-board planning, a key challenge is how to deliver required performance under strict memory and computational constraints. Techniques that compress Deep Neural Network (DNN) models attract our attention because they allow optimized neural network models to be efficiently deployed on platforms with limited energy and storage capacity. However, conventional model compression techniques prune the DNN after it is fully trained, which is very time-consuming especially when the model is trained using DRL. To overcome the limitation, we present an early phase integrated neural network weight compression system for DRL based waypoints planning. By applying pruning at an early phase, the compression of the DRL model can be realized without significant overhead in training. By tightly integrating pruning and retraining at the early phase, we achieve a higher model compression rate, reduce more memory and computing complexity, and improve the success rate compared to the original work

Stability and plasticity : constructing cognitive agents

Cataloged from PDF version of article.The AI field is currently dominated by domain-specific approaches to intelligence and cognition instead of being driven by the aim of modeling general human intelligence and cognition. This is despite the fact that the work widely regarded as marking the birth of AI was the project of creating a general cognitive architecture by Newell and Simon 1959. This thesis aims to examine recently designed models and their various cognitive features and limitations in preparation for building our own comprehensive model that would aim to address their limitations and give a better account for human cognition. The models differ in the kind of cognitive capabilities they view as the most important. They also differ in whether their foundation is built on symbolic or sub-symbolic atomic structures. Furthermore, we will look at studies in the philosophy and cognitive psychology domain in order to better understand the requirements that need to be met in order for a system to emulate general human cognition.Bozyiğit, ÖgeM.S

The Recognition of Words in Pure Alexia and Hemianopic Alexia: a Neuropsychological Study of 6 Patients

During my PhD I investigated how shape and motion information are processed by the rat visual system, so as to establish how advanced is the representation of higher-order visual information in this species and, ultimately, to understand to what extent rats can present a valuable alternative to monkeys, as experimental models, in vision studies. Specifically, in my thesis work, I have investigated: 1) The possible visual strategies underlying shape recognition. 2) The ability of rat visual cortical areas to represent motion and shape information. My work contemplated two different, but complementary experimental approaches: psychophysical measurements of the rat\u2019s recognition ability and strategy, and in vivo extracellular recordings in anaesthetized animals passively exposed to various (static and moving) visual stimulation. The first approach implied training the rats to an invariant object recognition task, i.e. to tolerate different ranges of transformations in the object\u2019s appearance, and the application of an mage classification technique known as The Bubbles to reveal the visual strategy the animals were able, under different conditions of stimulus discriminability, to adopt in order to perform the task. The second approach involved electrophysiological exploration of different visual areas in the rat\u2019s cortex, in order to investigate putative functional hierarchies (or streams of processing) in the computation of motion and shape information. Results show, on one hand, that rats are able, under conditions of highly stimulus discriminability, to adopt a shape-based, view-invariant, multi-featural recognition strategy; on the other hand, the functional properties of neurons recorded from different visual areas suggest the presence of a putative shape-based, ventral-like stream of processing in the rat\u2019s visual cortex. The general purpose of my work is and has been the unveiling the neural mechanisms that make object recognition happen, with the goal of eventually 1) be able to relate my findings on rats to those on more visually-advanced species, such as human and non-human primates; and 2) collect enough biological data to support the artificial simulation of visual recognition processes, which still presents an important scientific challeng