Scalable and Efficient Methods for Uncertainty Estimation and Reduction in Deep Learning

Neural networks (NNs) can achieved high performance in various fields such as computer vision, and natural language processing. However, deploying NNs in resource-constrained safety-critical systems has challenges due to uncertainty in the prediction caused by out-of-distribution data, and hardware non-idealities. To address the challenges of deploying NNs in resource-constrained safety-critical systems, this paper summarizes the (4th year) PhD thesis work that explores scalable and efficient methods for uncertainty estimation and reduction in deep learning, with a focus on Computation-in-Memory (CIM) using emerging resistive non-volatile memories. We tackle the inherent uncertainties arising from out-of-distribution inputs and hardware non-idealities, crucial in maintaining functional safety in automated decision-making systems. Our approach encompasses problem-aware training algorithms, novel NN topologies, and hardware co-design solutions, including dropout-based \emph{binary} Bayesian Neural Networks leveraging spintronic devices and variational inference techniques. These innovations significantly enhance OOD data detection, inference accuracy, and energy efficiency, thereby contributing to the reliability and robustness of NN implementations

2022 roadmap on neuromorphic computing and engineering

Modern computation based on von Neumann architecture is now a mature cutting-edge science. In the von Neumann architecture, processing and memory units are implemented as separate blocks interchanging data intensively and continuously. This data transfer is responsible for a large part of the power consumption. The next generation computer technology is expected to solve problems at the exascale with 10$^{18}$ calculations each second. Even though these future computers will be incredibly powerful, if they are based on von Neumann type architectures, they will consume between 20 and 30 megawatts of power and will not have intrinsic physically built-in capabilities to learn or deal with complex data as our brain does. These needs can be addressed by neuromorphic computing systems which are inspired by the biological concepts of the human brain. This new generation of computers has the potential to be used for the storage and processing of large amounts of digital information with much lower power consumption than conventional processors. Among their potential future applications, an important niche is moving the control from data centers to edge devices. The aim of this roadmap is to present a snapshot of the present state of neuromorphic technology and provide an opinion on the challenges and opportunities that the future holds in the major areas of neuromorphic technology, namely materials, devices, neuromorphic circuits, neuromorphic algorithms, applications, and ethics. The roadmap is a collection of perspectives where leading researchers in the neuromorphic community provide their own view about the current state and the future challenges for each research area. We hope that this roadmap will be a useful resource by providing a concise yet comprehensive introduction to readers outside this field, for those who are just entering the field, as well as providing future perspectives for those who are well established in the neuromorphic computing community

Memristor devices based on low-bandwidth manganites

This dissertation investigates the phenomenon of resistive switching (RS) in lowbandwidth mixed-valence perovskite manganite oxides. In particular, the compounds Pr0.6Ca0.4MnO3 and Gd1−xCaxMnO3 with x between 0 and 1 are studied. The steps of sample fabrication, crystalline properties and measurements to verify the quality of the devices are also reported. The thin film memristor devices were fabricated from target pellets using pulsed laser deposition on single crystal SrTiO3 substrates. The crystallinity was verified using X-ray diffraction and the elemental composition by energy dispersive X-ray spectroscopy. The fabricated thin films were used to create memristor devices by depositing patterned metal electrodes on them by either DC magnetron sputtering or e-beam physical vapor deposition. When the studied materials were combined with a reactive electrode material, the formed interface exhibited the phenomenon of resistive switching, where the resistance of the device can be modified non-volatilely by application of electric field to the terminals of the device. The noble metals Au and Ag were found to be optimal for the passive interfaces, and Al as the active interface. The RS properties of the devices made with the optimal electrode configuration were studied in detail. The devices were found to have asymmetric bipolar RS with promising characteristics. The studies encompassed varying the calcium doping of the samples, studying the endurance and timing characteristics of the RS phenomenon as well as measuring the device characteristics as a function of temperature. The RS properties were found to vary significantly over the calcium doping range. When the measurement results were used in a conduction model analysis, the switching properties were found to be correlated with the trap-energy level of the Al/GCMOinterface region. Lastly, the GCMO memristor devices were modeled successfully using a compact model compatible with circuit simulators and the biologicallyinspired spike-timing-dependent plasticity learning rule was demonstrated. In conclusion, GCMO is a promising new material for RS-based neuromorphic applications due to its stable switching properties. The unexpected differences between GCMO and PCMO show that there are still many unexplored RS properties and behaviors within the manganite family that can be explored in future research

Novel electrical and chemical findings on SIOx-based ReRAM devices

Existing non-volatile flash memory technologies are characterised by slow access time, high power consumption and a quickly approaching scaling limit. Filamentary resistive RAM (ReRAM) is an emerging type of storage device that relies on the electrically driven change in resistance of a thin film sandwiched between two electrodes. The active region is often a binary oxide that develops a restorable conductive filament thanks to the electrically driven movement of oxygen. This technology offers potential sub-10 nm scalability, nanosecond programming with direct overwriting (unlike FLASH) and an appealing sub pJ/bit power consumption (compared to nJ/bit of FLASH). In this thesis, metal-insulator-metal ReRAM devices with a TiN/SiOx/TiN structure are used. While other binary oxides have been used in the literature, SiOx must be used in its amorphous form allowing for easier fabrication, and is an extremely well-studied material as its CMOS compatibility dates back 40 years. Using the above devices, it was possible to observe data storage performance comparable to the one of other types of ReRAM. More interestingly, it was observed that the resistance states of this family of devices may be programmed using nanosecond pulses of identical magnitude, possibly leading to simple programming circuits. Consequently, it is shown that this programming method may also be used to gradually increase or decrease the device resistance state as well as have devices enter states that relax over time. These types of behaviour mean that SiOx devices may be used in neuromorphic networks that require components whose behaviour resembles the one of the neuronal synapsis or the mammalian brain’s forgetting process. The literature reports on endurance-hindering electrode deformation phenomena during the operation of oxide-based ReRAM devices. A residual gas analyser (RGA) was used to detect that oxygen species are emitted during operation and therefore confirmed that such phenomena are caused by oxygen emission. Using SIMS (secondary ion mass spectroscopy) analysis on devices switched in atmospheres containing isotopically labelled oxygen, it was observed that, under deformed regions, it is possible to find incorporated atmospheric oxygen. Additionally, reducing atmospheric pressure had negative impact on device reliability. SiOx-based filamentary ReRAM is a strong candidate in the search for alternatives to flash memory. Moreover, these devices display behaviour that may be useful in applications trying to emulate the mammalian brain. Having observed device dependence on its atmosphere, endurance issues may now be addressed using electrodes capable of either adsorbing oxygen without bubbling or letting it go through without cracking

Memristor-based design solutions for mitigating parametric variations in IoT applications

PhD ThesisRapid advancement of the internet of things (IoT) is predicated by two important factors of the electronic technology, namely device size and energy-efficiency. With smaller size comes the problem of process, voltage and temperature (PVT) variations of delays which are the key operational parameters of devices. Parametric variability is also an obstacle on the way to allowing devices to work in systems with unpredictable power sources, such as those powered by energy-harvesters. Designers tackle these problems holistically by developing new techniques such as asynchronous logic, where mechanisms such as matching delays are widely used to adapt to delay variations. To mitigate energy efficiency and power interruption issues the matching delays need to be ideally retained in a non-volatile storage. Meanwhile, a resistive memory called memristor becomes a promising component for power-restricted applications owing to its inherent non-volatility. While providing non-volatility, the use of memristor in delay matching incurs some power overheads. This creates the first challenge on the way of introducing memristors into IoT devices for the delay matching. Another important factor affecting the use of memristors in IoT devices is the dependence of the memristor value on temperature. For example, a memristance decoder used in the memristor-based components must be able to correct the read data without incurring significant overheads on the overall system. This creates the second challenge for overcoming the temperature effect in memristance decoding process. In this research, we propose methods for improving PVT tolerance and energy characteristics of IoT devices from the perspective of above two main challenges: (i) utilising memristor to enhance the energy efficiency of the delay element (DE), and (ii) improving the temperature awareness and energy robustness of the memristance decoder. For memristor-based delay element (MemDE), we applied a memristor between two inverters to vary the path resistance, which determines the RC delay. This allows power saving due to the low number of switching components and the absence of external delay storage. We also investigate a solution for avoiding the unintended tuning (UT) and a timing model to estimate the proper pulse width for memristance tuning. The simulation results based on UMC 180nm technology and VTEAM model show the MemDE can provide the delay between 0.55ns and 1.44ns which is compatible to the 4-bit multiplexerbased delay element (MuxDE) in the same technology while consuming thirteen times less power. The key contribution within (i) is the development of low-power MemDE to mitigate the timing mismatch caused by PVT variations. To estimate the temperature effect on memristance, we develop an empirical temperature model which fits both titanium dioxide and silver chalcogenide memristors. The temperature experiments are conducted using the latter device, and the results confirm the validity of the proposed model with the accuracy R-squared >88%. The memristance decoder is designed to deliver two key advantages. Firstly, the temperature model is integrated into the VTEAM model to enable the temperature compensation. Secondly, it supports resolution scalability to match the energy budget. The simulation results of the 2-bit decoder based on UMC 65nm technology show the energy can be varied between 49fJ and 98fJ. This is the second major contribution to address the challenge (ii). This thesis gives future research directions into an in-depth study of the memristive electronics as a variation-robust energy-efficient design paradigm and its impact on developing future IoT applications.sponsored by the Royal Thai Governmen

Towards Data Reliable, Low-Power, and Repairable Resistive Random Access Memories

A series of breakthroughs in memristive devices have demonstrated the potential of memristor arrays to serve as next generation resistive random access memories (ReRAM), which are fast, low-power, ultra-dense, and non-volatile. However, memristors' unique device characteristics also make them prone to several sources of error. Owing to the stochastic filamentary nature of memristive devices, various recoverable errors can affect the data reliability of a ReRAM. Permanent device failures further limit the lifetime of a ReRAM. This dissertation developed low-power solutions for more reliable and longer-enduring ReRAM systems. In this thesis, we first look into a data reliability issue known as write disturbance. Writing into a memristor in a crossbar could disturb the stored values in other memristors that are on the same memory line as the target cell. Such disturbance is accumulative over time which may lead to complete data corruption. To address this problem, we propose the use of two regular memristors on each word to keep track of the disturbance accumulation and trigger a refresh to restore the weakened data, once it becomes necessary. We also investigate the considerable variation in the write-time characteristics of individual memristors. With such variation, conventional fixed-pulse write schemes not only waste significant energy, but also cannot guarantee reliable completion of the write operations. We address such variation by proposing an adaptive write scheme that adjusts the width of the write pulses for each memristor. Our scheme embeds an online monitor to detect the completion of a write operation and takes into account the parasitic effect of line-shared devices in access-transistor-free memristive arrays. We further investigate the use of this method to shorten the test time of memory march algorithms by eliminating the need of a verifying read right after a write, which is commonly employed in the test sequences of march algorithms.Finally, we propose a novel mechanism to extend the lifetime of a ReRAM by protecting it against hard errors through the exploitation of a unique feature of bipolar memristive devices. Our solution proposes an unorthodox use of complementary resistive switches (a particular implementation of memristive devices) to provide an ``in-place spare'' for each memory cell at negligible extra cost. The in-place spares are then utilized by a repair scheme to repair memristive devices that have failed at a stuck-at-ON state at a page-level granularity. Furthermore, we explore the use of in-place spares in lieu of other memory reliability and yield enhancement solutions, such as error correction codes (ECC) and spare rows. We demonstrate that with the in-place spares, we can yield the same lifetime as a baseline ReRAM with either significantly fewer spare rows or a lighter-weight ECC, both of which can save on energy consumption and area

Electronic systems for the restoration of the sense of touch in upper limb prosthetics

In the last few years, research on active prosthetics for upper limbs focused on improving the human functionalities and the control. New methods have been proposed for measuring the user muscle activity and translating it into the prosthesis control commands. Developing the feed-forward interface so that the prosthesis better follows the intention of the user is an important step towards improving the quality of life of people with limb amputation. However, prosthesis users can neither feel if something or someone is touching them over the prosthesis and nor perceive the temperature or roughness of objects. Prosthesis users are helped by looking at an object, but they cannot detect anything otherwise. Their sight gives them most information. Therefore, to foster the prosthesis embodiment and utility, it is necessary to have a prosthetic system that not only responds to the control signals provided by the user, but also transmits back to the user the information about the current state of the prosthesis. This thesis presents an electronic skin system to close the loop in prostheses towards the restoration of the sense of touch in prosthesis users. The proposed electronic skin system inlcudes an advanced distributed sensing (electronic skin), a system for (i) signal conditioning, (ii) data acquisition, and (iii) data processing, and a stimulation system. The idea is to integrate all these components into a myoelectric prosthesis. Embedding the electronic system and the sensing materials is a critical issue on the way of development of new prostheses. In particular, processing the data, originated from the electronic skin, into low- or high-level information is the key issue to be addressed by the embedded electronic system. Recently, it has been proved that the Machine Learning is a promising approach in processing tactile sensors information. Many studies have been shown the Machine Learning eectiveness in the classication of input touch modalities.More specically, this thesis is focused on the stimulation system, allowing the communication of a mechanical interaction from the electronic skin to prosthesis users, and the dedicated implementation of algorithms for processing tactile data originating from the electronic skin. On system level, the thesis provides design of the experimental setup, experimental protocol, and of algorithms to process tactile data. On architectural level, the thesis proposes a design ow for the implementation of digital circuits for both FPGA and integrated circuits, and techniques for the power management of embedded systems for Machine Learning algorithms

Structural, Thermodynamic, and Electronic Properties of Mixed Ionic/Electronic Conductor Materials

Due to the mainstream CMOS technology facing a rapid approach to the fundamental downscaling limit, beyond CMOS technologies are under active investigation and development with the intention of revolutionizing and sustaining a wide range of applications including sensors, cryptography, neuromorphic and quantum computing, memory, and logic, among others. Resistive switching electronics, for example, are devices that can change their electrical resistance with an applied external field. Despite their simple metal-insulator-metal structure, resistive switching devices exhibit an intricate set of IV characteristics based on the chemical composition of the solid electrolyte that ranges from non-volatile bipolar and non-polar switching to volatile threshold switching (abrupt but reversible change in resistance). This rich variety of electrical responses offer new alternatives to traditional CMOS applications in the areas of RF-signal switching, relaxation oscillators, over-voltage protection, and notably, memory cells and two-terminal non-linear selector devices. With the aim of unraveling the physics behind two of such conduction mechanisms, filamentary and threshold, in electrochemical cells consisting solid mixed ionic-electronic conductor electrolytes, this work focused on using first-principles calculations to characterize the structural, thermodynamic, and electronic properties of copper-doped amorphous silicon dioxide and copper-doped germanium-based glassy chalcogenides. The Cu/a-SiO2 system is a promising candidate for resistive switching memory applications. The conduction mechanism in the low-resistance state is known to be filamentary, that is, a physical metallic filament bridges between the metallic electrodes through the amorphous silica. However, many fundamental materials processes that would aid the design and optimization of this devices, such the shape and size of stable metallic filaments, remain unknown. In the first part of this work, the morphology and diffusion of small copper clusters embedded in amorphous silicon dioxide were characterized by density functional theory calculations. The average formation energy of a single copper ion in the amorphous matrix is found to be 2.4 eV, about 50% lower than in the case of silicon dioxide in its cristobalite or quartz phases. The theoretical predictions show that copper clusters with an equiaxed morphology are always energetically favorable relative to the dissolved copper ions in a-SiO2; hence, stable clusters do not exhibit a critical size. The stochasticity in the atomistic structure of the amorphous silicon dioxide leads to a broad distribution activation energies for diffusion of copper in the matrix, ranging from 0.4 to 1.1 eV. Since ab initio molecular dynamics are prohibitively expensive to simulate the switching process in Cu/a-SiO2 electrochemical metallization cells, a multi-scale simulation approach based on electrochemical dynamics with implicit degrees of freedom and density functional theory was developed to study the electronic evolution during metallic filament formation cells. These simulations suggest that the electronic transport in the low-resistance configuration is attributed to copper derived states belonging to the filament bridging between electrodes. Interestingly, the participation of states derived from intrinsic defects in the amorphous SiO2 around the Fermi energy are negligible and do not contribute to conduction. Unlike the Cu/a-SiO2 system which only exhibits filamentary switching, copper-doped germanium-based glassy chalcogenides show filamentary or threshold type of conduction depending on the chemical composition of the glass and copper concentration. Ab initio molecular dynamics based on density functional theory is used to understand the atomistic origin of the electronic transport in these materials. The theoretical predictions show that glasses containing tellurium tend to favor the formation of copper clusters; hence, these materials exhibit filamentary conduction. Threshold conduction is predicted to be the transport mechanism for glassy sulfides and selenides due to the ability of copper to remain dissolved in the amorphous matrix even at high metal concentration. Furthermore, the charge carrier transport in sulfur and selenium glasses was found to be assisted by defective states derived from chalcogen atoms whose bonds exhibit a polar character. Finally, taking advantage of the van der Waals gap as intercalation sites and crystal order in molybdenum disulfide, a novel mixed ionic-electronic conductor material based on copper and silver intercalation of MoS2 is proposed. The theoretical predictions show that on average, the intercalation energy of copper into MoS2 is 1 eV, while intercalation of silver shows a strong dependence on concentration ranging from 2.2 to 0.75 eV for low and high concentrations, respectively. The activation energy for diffusion of copper and silver intercalated within the van der Waals gap of MoS2 is predicted to be 0.32 and 0.38 eV, respectively, comparable to other superionic conductors. Upon Cu and Ag intercalation, MoS2 undergoes a semiconductor-to-metal transition, where the in-plane and out-of-plane conductances are comparable and exhibit a linear dependence with metal concentration