26 research outputs found

    Efficient Algorithms for Artificial Neural Networks and Explainable AI

    Get PDF
    Artificial neural networks have allowed some remarkable progress in fields such as pattern recognition and computer vision. However, the increasing complexity of artificial neural networks presents a challenge for efficient computation. In this thesis, we first introduce a novel matrix multiplication method to reduce the complexity of artificial neural networks, where we demonstrate its suitability to compress fully connected layers of artificial neural networks. Our method outperforms other state-of-the-art methods when tested on standard publicly available datasets. This thesis then focuses on Explainable AI, which can be critical in fields like finance and medicine, as it can provide explanations for some decisions taken by sub-symbolic AI models behaving like a black box such as Artificial neural networks and transformation based learning approaches. We have also developed a new framework that facilitates the use of Explainable AI with tabular datasets. Our new framework Exmed, enables nonexpert users to prepare data, train models, and apply Explainable AI techniques effectively.Additionally, we propose a new algorithm that identifies the overall influence of input features and minimises the perturbations that alter the decision taken by a given model. Overall, this thesis introduces innovative and comprehensive techniques to enhance the efficiency of fully connected layers in artificial neural networks and provide a new approach to explain their decisions. These methods have significant practical applications in various fields, including portable medical devices

    Rule-based system architecting of Earth observation satellite systems

    Get PDF
    Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Aeronautics and Astronautics, 2012.Cataloged from PDF version of thesis.Includes bibliographical references (p. 399-412).System architecting is concerned with exploring the tradespace of early, high-level, system design decisions with a holistic, value-centric view. In the last few years, several tools and methods have been developed to support the system architecting process, focusing on the representation of an architecture as a set of interrelated decisions. These tools are best suited for applications that focus on breadth - i.e., enumerating a large and representative part of the architectural tradespace -as opposed to depth - modeling fidelity. However, some problems in system architecting require good modeling depth in order to provide useful results. In some cases, a very large body of expert knowledge is required. Current tools are not designed to handle such large bodies of knowledge because they lack scalability and traceability. As the size of the knowledge base increases, it becomes harder: a) to modify existing knowledge or add new knowledge; b) to trace the results of the tool to the model assumptions or knowledge base. This thesis proposes a holistic framework for architecture tradespace exploration of large complex systems that require a large body of expert knowledge. It physically separates the different bodies of knowledge required to solve a system architecting problem (i.e., knowledge about the domain, knowledge about the class of optimization or search problem, knowledge about the particular instance of problem) by using a rule-based expert system. It provides a generic population-based heuristic algorithm for search, which can be augmented with rules that encode knowledge about the domain, or about the optimization problem or class of problems. It identifies five major classes of system architecting problems from the perspective of optimization and search, and provides rules to enumerate architectures and search through the architectural tradespace of each class. A methodology is also defined to assess the value of an architecture using a rule-based approach. This methodology is based on a decomposition of stakeholder needs into requirements and a systematic comparison between system requirements and system capabilities using the rules engine. The framework is applied to the domain of Earth observing satellite systems (EOSS). Three EOSS are studied in depth: the NASA Earth Observing System, the NRC Earth Science Decadal Survey, and the Iridium GEOscan program. The ability of the framework to produce useful results is shown, and specific insights and recommendations are drawn.by Daniel Selva Valero.Ph.D

    Tiny Machine Learning Environment: Enabling Intelligence on Constrained Devices

    Get PDF
    Running machine learning algorithms (ML) on constrained devices at the extreme edge of the network is problematic due to the computational overhead of ML algorithms, available resources on the embedded platform, and application budget (i.e., real-time requirements, power constraints, etc.). This required the development of specific solutions and development tools for what is now referred to as TinyML. In this dissertation, we focus on improving the deployment and performance of TinyML applications, taking into consideration the aforementioned challenges, especially memory requirements. This dissertation contributed to the construction of the Edge Learning Machine environment (ELM), a platform-independent open-source framework that provides three main TinyML services, namely shallow ML, self-supervised ML, and binary deep learning on constrained devices. In this context, this work includes the following steps, which are reflected in the thesis structure. First, we present the performance analysis of state-of-the-art shallow ML algorithms including dense neural networks, implemented on mainstream microcontrollers. The comprehensive analysis in terms of algorithms, hardware platforms, datasets, preprocessing techniques, and configurations shows similar performance results compared to a desktop machine and highlights the impact of these factors on overall performance. Second, despite the assumption that TinyML only permits models inference provided by the scarcity of resources, we have gone a step further and enabled self-supervised on-device training on microcontrollers and tiny IoT devices by developing the Autonomous Edge Pipeline (AEP) system. AEP achieves comparable accuracy compared to the typical TinyML paradigm, i.e., models trained on resource-abundant devices and then deployed on microcontrollers. Next, we present the development of a memory allocation strategy for convolutional neural networks (CNNs) layers, that optimizes memory requirements. This approach reduces the memory footprint without affecting accuracy nor latency. Moreover, e-skin systems share the main requirements of the TinyML fields: enabling intelligence with low memory, low power consumption, and low latency. Therefore, we designed an efficient Tiny CNN architecture for e-skin applications. The architecture leverages the memory allocation strategy presented earlier and provides better performance than existing solutions. A major contribution of the thesis is given by CBin-NN, a library of functions for implementing extremely efficient binary neural networks on constrained devices. The library outperforms state of the art NN deployment solutions by drastically reducing memory footprint and inference latency. All the solutions proposed in this thesis have been implemented on representative devices and tested in relevant applications, of which results are reported and discussed. The ELM framework is open source, and this work is clearly becoming a useful, versatile toolkit for the IoT and TinyML research and development community

    Symmetry in Chaotic Systems and Circuits

    Get PDF
    Symmetry can play an important role in the field of nonlinear systems and especially in the design of nonlinear circuits that produce chaos. Therefore, this Special Issue, titled “Symmetry in Chaotic Systems and Circuits”, presents the latest scientific advances in nonlinear chaotic systems and circuits that introduce various kinds of symmetries. Applications of chaotic systems and circuits with symmetries, or with a deliberate lack of symmetry, are also presented in this Special Issue. The volume contains 14 published papers from authors around the world. This reflects the high impact of this Special Issue

    Intrinsic Hardware Evolution on the Transistor Level

    Get PDF
    This thesis presents a novel approach to the automated synthesis of analog circuits. Evolutionary algorithms are used in conjunction with a fitness evaluation on a dedicated ASIC that serves as the analog substrate for the newly bred candidate solutions. The advantage of evaluating the candidate circuits directly in hardware is twofold. First, it may speed up the evolutionary algorithms, because hardware tests can usually be performed faster than simulations. Second, the evolved circuits are guaranteed to work on a real piece of silicon. The proposed approach is realized as a hardware evolution system consisting of an IBM compatible general purpose computer that hosts the evolutionary algorithm, an FPGA-based mixed signal test board, and the analog substrate. The latter one is designed as a Field Programmable Transistor Array (FPTA) whose programmable transistor cells can be almost freely connected. The transistor cells can be configured to adopt one out of 75 different channel geometries. The chip was produced in a 0.6µm CMOS process and provides ample means for the input and output of analog signals. The configuration is stored in SRAM cells embedded in the programmable transistor cells. The hardware evolution system is used for numerous evolution experiments targeted at a wide variety of different circuit functionalities. These comprise logic gates, Gaussian function circuits, D/A converters, low- and highpass filters, tone discriminators, and comparators. The experimental results are thoroughly analyzed and discussed with respect to related work

    Approximate Computing Survey, Part I: Terminology and Software & Hardware Approximation Techniques

    Full text link
    The rapid growth of demanding applications in domains applying multimedia processing and machine learning has marked a new era for edge and cloud computing. These applications involve massive data and compute-intensive tasks, and thus, typical computing paradigms in embedded systems and data centers are stressed to meet the worldwide demand for high performance. Concurrently, the landscape of the semiconductor field in the last 15 years has constituted power as a first-class design concern. As a result, the community of computing systems is forced to find alternative design approaches to facilitate high-performance and/or power-efficient computing. Among the examined solutions, Approximate Computing has attracted an ever-increasing interest, with research works applying approximations across the entire traditional computing stack, i.e., at software, hardware, and architectural levels. Over the last decade, there is a plethora of approximation techniques in software (programs, frameworks, compilers, runtimes, languages), hardware (circuits, accelerators), and architectures (processors, memories). The current article is Part I of our comprehensive survey on Approximate Computing, and it reviews its motivation, terminology and principles, as well it classifies and presents the technical details of the state-of-the-art software and hardware approximation techniques.Comment: Under Review at ACM Computing Survey

    Unconventional programming: non-programmable systems

    Get PDF
    Die Forschung aus dem Bereich der unkonventionellen und natürlichen Informationsverarbeitungssysteme verspricht kontrollierbare Rechenprozesse in ungewöhnlichen Medien zu realisieren, zum Beispiel auf der molekularen Ebene oder in Bakterienkolonien. Vielversprechende Eigenschaften dieser Systeme sind das nichtlineare Verhalten und der hohe Verknüpfungsgrad der beteiligten Komponenten in Analogie zu Neuronen im Gehirn. Da aber Programmierung meist auf Prinzipien wie Modularisierung, Kapselung und Vorhersagbarkeit beruht sind diese Systeme oft schwer- bzw. unprogrammierbar. Im Gegensatz zu vielen Arbeiten über unkonventionelle Rechensysteme soll in dieser Arbeit aber nicht hauptsächlich nach neuen rechnenden Systemen und Anwendungen dieser gesucht werden. Stattdessen konzentriert sich diese Dissertation auf unkonventionelle Programmieransätze, die sowohl für unkonventionelle Computer als auch für herkommliche digitale Rechner neue Perspektiven eröffnen sollen. Hauptsächlich in Bezug auf ein Modell künstlicher chemischer Neuronen werden Ansätze für unkonventionelle Programmierverfahren, basierend auf Evolutionären Algorithmen, Informationstheorie und Selbstorganisation bis hin zur Selbstassemblierung untersucht. Ein spezielles Augenmerk liegt dabei auf dem Problem der Symbolkodierung: Oft gibt es mehrere oder sogar unendlich viele Möglichkeiten, Informationen in den Zuständen eines komplexen dynamischen Systems zu kodieren. In Neuronalen Netzen gibt es unter anderem die Spikefrequenz aber auch Populationskodes. In Abhängigkeit von den weiteren Eigenschaften des Systems, beispielsweise von der Informationsverarbeitungsaufgabe und dem gewünschten Eingabe-Ausgabeverhalten dürften sich verschiedene Kodierungen als unterschiedlich nützlich erweisen. Daher werden hier Methoden betrachtet um die verschiedene Symbolkodierungmethoden zu evaluieren, zu analysieren und um nach neuen, geeigneten Kodierungen zu suchen.Unconventional and natural computing research offers controlled information modification processes in uncommon media, for example on the molecular scale or in bacteria colonies. Promising aspects of such systems are often the non-linear behavior and the high connectivity of the involved information processing components in analogy to neurons in the nervous system. Unfortunately, such properties make the system behavior hard to understand, hard to predict and thus also hard to program with common engineering principles like modularization and composition, leading to the term of non-programmable systems. In contrast to many unconventional computing works that are often focused on finding novel computing substrates and potential applications, unconventional programming approaches for such systems are the theme of this thesis: How can new programming concepts open up new perspectives for unconventional but hopefully also for traditional, digital computing systems? Mostly based on a model of artificial wet chemical neurons, different unconventional programming approaches from evolutionary algorithms, information theory, self-organization and self-assembly are explored. A particular emphasis is given on the problem of symbol encodings: Often there are multiple or even an unlimited number of possibilities to encode information in the phase space of dynamical systems, e.g. spike frequencies or population coding in neural networks. But different encodings will probably be differently useful, dependent on the system properties, the information transformation task and the desired connectivity to other systems. Hence methods are investigated that can evaluate, analyse as well as identify suitable symbol encoding schemes

    Tiny Machine Learning Environment: Enabling Intelligence on Constrained Devices

    Get PDF
    Running machine learning algorithms (ML) on constrained devices at the extreme edge of the network is problematic due to the computational overhead of ML algorithms, available resources on the embedded platform, and application budget (i.e., real-time requirements, power constraints, etc.). This required the development of specific solutions and development tools for what is now referred to as TinyML. In this dissertation, we focus on improving the deployment and performance of TinyML applications, taking into consideration the aforementioned challenges, especially memory requirements. This dissertation contributed to the construction of the Edge Learning Machine environment (ELM), a platform-independent open source framework that provides three main TinyML services, namely shallow ML, self-supervised ML, and binary deep learning on constrained devices. In this context, this work includes the following steps, which are reflected in the thesis structure. First, we present the performance analysis of state of the art shallow ML algorithms including dense neural networks, implemented on mainstream microcontrollers. The comprehensive analysis in terms of algorithms, hardware platforms, datasets, pre-processing techniques, and configurations shows similar performance results compared to a desktop machine and highlights the impact of these factors on overall performance. Second, despite the assumption that TinyML only permits models inference provided by the scarcity of resources, we have gone a step further and enabled self-supervised on-device training on microcontrollers and tiny IoT devices by developing the Autonomous Edge Pipeline (AEP) system. AEP achieves comparable accuracy compared to the typical TinyML paradigm, i.e., models trained on resource-abundant devices and then deployed on microcontrollers. Next, we present the development of a memory allocation strategy for convolutional neural networks (CNNs) layers, that optimizes memory requirements. This approach reduces the memory footprint without affecting accuracy nor latency. Moreover, e-skin systems share the main requirements of the TinyML fields: enabling intelligence with low memory, low power consumption, and low latency. Therefore, we designed an efficient Tiny CNN architecture for e-skin applications. The architecture leverages the memory allocation strategy presented earlier and provides better performance than existing solutions. A major contribution of the thesis is given by CBin-NN, a library of functions for implementing extremely efficient binary neural networks on constrained devices. The library outperforms state of the art NN deployment solutions by drastically reducing memory footprint and inference latency. All the solutions proposed in this thesis have been implemented on representative devices and tested in relevant applications, of which results are reported and discussed. The ELM framework is open source, and this work is clearly becoming a useful, versatile toolkit for the IoT and TinyML research and development community

    Secure Outsourced Computation on Encrypted Data

    Get PDF
    Homomorphic encryption (HE) is a promising cryptographic technique that supports computations on encrypted data without requiring decryption first. This ability allows sensitive data, such as genomic, financial, or location data, to be outsourced for evaluation in a resourceful third-party such as the cloud without compromising data privacy. Basic homomorphic primitives support addition and multiplication on ciphertexts. These primitives can be utilized to represent essential computations, such as logic gates, which subsequently can support more complex functions. We propose the construction of efficient cryptographic protocols as building blocks (e.g., equality, comparison, and counting) that are commonly used in data analytics and machine learning. We explore the use of these building blocks in two privacy-preserving applications. One application leverages our secure prefix matching algorithm, which builds on top of the equality operation, to process geospatial queries on encrypted locations. The other applies our secure comparison protocol to perform conditional branching in private evaluation of decision trees. There are many outsourced computations that require joint evaluation on private data owned by multiple parties. For example, Genome-Wide Association Study (GWAS) is becoming feasible because of the recent advances of genome sequencing technology. Due to the sensitivity of genomic data, this data is encrypted using different keys possessed by different data owners. Computing on ciphertexts encrypted with multiple keys is a non-trivial task. Current solutions often require a joint key setup before any computation such as in threshold HE or incur large ciphertext size (at best, grows linearly in the number of involved keys) such as in multi-key HE. We propose a hybrid approach that combines the advantages of threshold and multi-key HE to support computations on ciphertexts encrypted with different keys while vastly reducing ciphertext size. Moreover, we propose the SparkFHE framework to support large-scale secure data analytics in the Cloud. SparkFHE integrates Apache Spark with Fully HE to support secure distributed data analytics and machine learning and make two novel contributions: (1) enabling Spark to perform efficient computation on large datasets while preserving user privacy, and (2) accelerating intensive homomorphic computation through parallelization of tasks across clusters of computing nodes. To our best knowledge, SparkFHE is the first addressing these two needs simultaneously
    corecore