Towards Accurate and High-Speed Spiking Neuromorphic Systems with Data Quantization-Aware Deep Networks

Deep Neural Networks (DNNs) have gained immense success in cognitive applications and greatly pushed today's artificial intelligence forward. The biggest challenge in executing DNNs is their extremely data-extensive computations. The computing efficiency in speed and energy is constrained when traditional computing platforms are employed in such computational hungry executions. Spiking neuromorphic computing (SNC) has been widely investigated in deep networks implementation own to their high efficiency in computation and communication. However, weights and signals of DNNs are required to be quantized when deploying the DNNs on the SNC, which results in unacceptable accuracy loss. %However, the system accuracy is limited by quantizing data directly in deep networks deployment. Previous works mainly focus on weights discretize while inter-layer signals are mainly neglected. In this work, we propose to represent DNNs with fixed integer inter-layer signals and fixed-point weights while holding good accuracy. We implement the proposed DNNs on the memristor-based SNC system as a deployment example. With 4-bit data representation, our results show that the accuracy loss can be controlled within 0.02% (2.3%) on MNIST (CIFAR-10). Compared with the 8-bit dynamic fixed-point DNNs, our system can achieve more than 9.8x speedup, 89.1% energy saving, and 30% area saving.Comment: 6 pages, 4 figure

Brain-Inspired Computing: Neuromorphic System Designs and Applications

In nowadays big data environment, the conventional computing platform based on von Neumann architecture encounters the bottleneck of the increasing requirement of computation capability and efficiency. The “brain-inspired computing” Neuromorphic Computing has demonstrated great potential to revolutionize the technology world. It is considered as one of the most promising solutions by achieving tremendous computing and power efficiency on a single chip. The neuromorphic computing systems represent great promise for many scientific and intelligent applications. Many designs have been proposed and realized with traditional CMOS technology, however, the progress is slow. Recently, the rebirth of neuromorphic computing is inspired by the development of novel nanotechnology. In this thesis, I propose neuromorphic computing systems with the ReRAM (Memristor) crossbar array. It includes the work in three major parts: 1) Memristor devices modeling and related circuits design in resistive memory (ReRAM) technology by investigating their physical mechanism, statistical analysis, and intrinsic challenges. A weighted sensing scheme which assigns different weights to the cells on different bit lines was proposed. The area/power overhead of peripheral circuitry was effectively reduced while minimizing the amplitude of sneak paths. 2) Neuromorphic computing system designs by leveraging memristor devices and algorithm scaling in neural network and machine learning algorithms based on the similarity between memristive effect and biological synaptic behavior. First, a spiking neural network (SNN) with a rate coding model was developed in algorithm level and then mapped to hardware design for supervised learning. In addition, to further speed and accuracy improvement, another neuromorphic system adopting analog input signals with different voltage amplitude and a current sensing scheme was built. Moreover, the use of a single memristor crossbar for each neural net- work layer was explored. 3) The application-specific optimization for further reliability improvement of the developed neuromorphic systems. In this thesis, the impact of device failure on the memristor-based neuromorphic computing systems for cognitive applications was evaluated. Then, a retraining and a remapping design in algorithm level and hardware level were developed to rescue the large accuracy loss

SPICE Simulation of RRAM-Based Cross-Point Arrays Using the Dynamic Memdiode Model

We thoroughly investigate the performance of the Dynamic Memdiode Model (DMM) when used for simulating the synaptic weights in large RRAM-based cross-point arrays (CPA) intended for neuromorphic computing. The DMM is in line with Prof. Chua’s memristive devices theory, in which the hysteresis phenomenon in electroformed metal-insulator-metal structures is represented by means of two coupled equations: one equation for the current-voltage characteristic of the device based on an extension of the quantum point-contact (QPC) model for dielectric breakdown and a second equation for the memory state, responsible for keeping track of the previous history of the device. By considering ex-situ training of the CPA aimed at classifying the handwritten characters of the MNIST database, we evaluate the performance of a Write-Verify iterative scheme for setting the crosspoint conductances to their target values. The total programming time, the programming error, and the inference accuracy obtained with such writing scheme are investigated in depth. The role played by parasitic components such as the line resistance as well as some CPA’s particular features like the dynamical range of the memdiodes are discussed. The interrelationship between the frequency and amplitude values of the write pulses is explored in detail. In addition, the effect of the resistance shift for the case of a CPA programmed with no errors is studied for a variety of input signals, providing a design guideline for selecting the appropriate pulse’s amplitude and frequency.Fil: Aguirre, Fernando Leonel. Universidad Tecnológica Nacional. Facultad Regional Buenos Aires. Unidad de Investigación y Desarrollo de las Ingenierías; Argentina. Consejo Nacional de Investigaciones Científicas y Técnicas; ArgentinaFil: Pazos, Sebastián Matías. Universidad Tecnológica Nacional. Facultad Regional Buenos Aires. Unidad de Investigación y Desarrollo de las Ingenierías; Argentina. Consejo Nacional de Investigaciones Científicas y Técnicas; ArgentinaFil: Palumbo, Félix Roberto Mario. Consejo Nacional de Investigaciones Científicas y Técnicas; Argentina. Universidad Tecnológica Nacional. Facultad Regional Buenos Aires. Unidad de Investigación y Desarrollo de las Ingenierías; ArgentinaFil: Suñé, Jordi. Universitat Autònoma de Barcelona; EspañaFil: Miranda, Enrique. Universitat Autònoma de Barcelona; Españ

Reliability-aware circuit design to mitigate impact of device defects and variability in emerging memristor-based applications

In the last decades, semiconductor industry has fostered a fast downscale in technology, propelling the large scale integration of CMOS-based systems. The benefits in miniaturization are numerous, highlighting faster switching frequency, lower voltage supply and higher device density. However, this aggressive scaling trend it has not been without challenges, such as leakage currents, yield reduction or the increase in the overall system power dissipation. New materials, changes in the device structures and new architectures are key to keep the miniaturization trend. It is foreseen that 2D integration will eventually come to an insurmountable physical and economic limit, in which new strategic directions are required, such as the development of new device structures, 3D architectures or heterogeneous systems that takes advantage of the best of different technologies, both the ones already consolidated as well as emergent ones that provide performance and efficiency improvements in applications. In this context, memristor arises as one of several candidates in the race to find suitable emergent devices. Memristor, a blend of the words memory and resistor, is a passive device postulated by Leon Chua in 1971. In contrast with the other fundamental passive elements, memristors have the distinctive feature of modifying their resistance according to the charge that passes through these devices, and remaining unaltered when charge no longer flows. Although when it appeared no physical device implementation was acknowledged, HP Labs claimed in 2008 the manufacture of the first real memristor. This milestone triggered an unexpectedly high research activity about memristors, both in searching new materials and structures as well as in potential applications. Nowadays, memristors are not only appreciated in memory systems by their nonvolatile storage properties, but in many other fields, such as digital computing, signal processing circuits, or non-conventional applications like neuromorphic computing or chaotic circuits. In spite of their promising features, memristors show a primarily downside: they show significant device variation and limited lifetime due degradation compared with other alternatives. This Thesis explores the challenges that memristor variation and malfunction imposes in potential applications. The main goal is to propose circuits and strategies that either avoid reliability problems or take advantage of them. Throughout a collection of scenarios in which reliability issues are present, their impact is studied by means of simulations. This thesis is contextualized and their objectives are exposed in Chapter 1. In Chapter 2 the memristor is introduced, at both conceptual and experimental levels, and different compact levels are presented to be later used in simulations. Chapter 3 deepens in the phenomena that causes the lack of reliability in memristors, and models that include these defects in simulations are provided. The rest of the Thesis covers different applications. Therefore, Chapter 4 exhibits nonvolatile memory systems, and specifically an online test method for faulty cells. Digital computing is presented in Chapter 5, where a solution for the yield reduction in logic operations due to memristors variability is proposed. Lastly, Chapter 6 reviews applications in the analog domain, and it focuses in the exploitation of results observed in faulty memristor-based interconnect mediums for chaotic systems synchronization purposes. Finally, the Thesis concludes in Chapter 7 along with perspectives about future work.Este trabajo desarrolla un novedoso dispositivo condensador basado en el uso de la nanotecnología. El dispositivo parte del concepto existente de metal-aislador-metal (MIM), pero en lugar de una capa aislante continua, se utilizan nanopartículas dieléctricas. Las nanopartículas son principalmente de óxido de silicio (sílice) y poliestireno (PS) y los valores de diámetro son 255nm y 295nm respectivamente. Las nanopartículas contribuyen a una alta relación superficie/volumen y están fácilmente disponibles a bajo costo. La tecnología de depósito desarrollada en este trabajo se basa en la técnica de electrospray, que es una tecnología de fabricación ascendente (bottom-up) que permite el procesamiento por lotes y logra un buen compromiso entre una gran superficie y un bajo tiempo de depósito. Con el objetivo de aumentar la superficie de depósito, la configuración de electrospray ha sido ajustada para permitir áreas de depósito de 1cm2 a 25cm2. El dispositivo fabricado, los llamados condensadores de metal aislante de nanopartículas (NP-MIM) ofrecen valores de capacidad más altos que un condensador convencional similar con una capa aislante continua. En el caso de los NP-MIM de sílice, se alcanza un factor de hasta 1000 de mejora de la capacidad, mientras que los NP-MIM de poliestireno exhibe una ganancia de capacidad en el rango de 11. Además, los NP-MIM de sílice muestran comportamientos capacitivos en específicos rangos de frecuencias que depende de la humedad y el grosor de la capa de nanopartículas, mientras que los NP-MIM de poliestireno siempre mantienen su comportamiento capacitivo. Los dispositivos fabricados se han caracterizado mediante medidas de microscopía electrónica de barrido (SEM) complementadas con perforaciones de haz de iones focalizados (FIB) para caracterizar la topografía de los NP-MIMs. Los dispositivos también se han caracterizado por medidas de espectroscopia de impedancia, a diferentes temperaturas y humedades. El origen de la capacitancia aumentada está asociado en parte a la humedad en las interfaces de las nanopartículas. Se ha desarrollado un modelo de un circuito basado en elementos distribuidos para ajustar y predecir el comportamiento eléctrico de los NP-MIMs. En resumen, esta tesis muestra el diseño, fabricación, caracterización y modelización de un nuevo y prometedor condensador nanopartículas metal-aislante-metal que puede abrir el camino al desarrollo de una nueva tecnología de supercondensadores MIM

Reliability-aware circuit design to mitigate impact of device defects and variability in emerging memristor-based applications

In the last decades, semiconductor industry has fostered a fast downscale in technology, propelling the large scale integration of CMOS-based systems. The benefits in miniaturization are numerous, highlighting faster switching frequency, lower voltage supply and higher device density. However, this aggressive scaling trend it has not been without challenges, such as leakage currents, yield reduction or the increase in the overall system power dissipation. New materials, changes in the device structures and new architectures are key to keep the miniaturization trend. It is foreseen that 2D integration will eventually come to an insurmountable physical and economic limit, in which new strategic directions are required, such as the development of new device structures, 3D architectures or heterogeneous systems that takes advantage of the best of different technologies, both the ones already consolidated as well as emergent ones that provide performance and efficiency improvements in applications. In this context, memristor arises as one of several candidates in the race to find suitable emergent devices. Memristor, a blend of the words memory and resistor, is a passive device postulated by Leon Chua in 1971. In contrast with the other fundamental passive elements, memristors have the distinctive feature of modifying their resistance according to the charge that passes through these devices, and remaining unaltered when charge no longer flows. Although when it appeared no physical device implementation was acknowledged, HP Labs claimed in 2008 the manufacture of the first real memristor. This milestone triggered an unexpectedly high research activity about memristors, both in searching new materials and structures as well as in potential applications. Nowadays, memristors are not only appreciated in memory systems by their nonvolatile storage properties, but in many other fields, such as digital computing, signal processing circuits, or non-conventional applications like neuromorphic computing or chaotic circuits. In spite of their promising features, memristors show a primarily downside: they show significant device variation and limited lifetime due degradation compared with other alternatives. This Thesis explores the challenges that memristor variation and malfunction imposes in potential applications. The main goal is to propose circuits and strategies that either avoid reliability problems or take advantage of them. Throughout a collection of scenarios in which reliability issues are present, their impact is studied by means of simulations. This thesis is contextualized and their objectives are exposed in Chapter 1. In Chapter 2 the memristor is introduced, at both conceptual and experimental levels, and different compact levels are presented to be later used in simulations. Chapter 3 deepens in the phenomena that causes the lack of reliability in memristors, and models that include these defects in simulations are provided. The rest of the Thesis covers different applications. Therefore, Chapter 4 exhibits nonvolatile memory systems, and specifically an online test method for faulty cells. Digital computing is presented in Chapter 5, where a solution for the yield reduction in logic operations due to memristors variability is proposed. Lastly, Chapter 6 reviews applications in the analog domain, and it focuses in the exploitation of results observed in faulty memristor-based interconnect mediums for chaotic systems synchronization purposes. Finally, the Thesis concludes in Chapter 7 along with perspectives about future work.Este trabajo desarrolla un novedoso dispositivo condensador basado en el uso de la nanotecnología. El dispositivo parte del concepto existente de metal-aislador-metal (MIM), pero en lugar de una capa aislante continua, se utilizan nanopartículas dieléctricas. Las nanopartículas son principalmente de óxido de silicio (sílice) y poliestireno (PS) y los valores de diámetro son 255nm y 295nm respectivamente. Las nanopartículas contribuyen a una alta relación superficie/volumen y están fácilmente disponibles a bajo costo. La tecnología de depósito desarrollada en este trabajo se basa en la técnica de electrospray, que es una tecnología de fabricación ascendente (bottom-up) que permite el procesamiento por lotes y logra un buen compromiso entre una gran superficie y un bajo tiempo de depósito. Con el objetivo de aumentar la superficie de depósito, la configuración de electrospray ha sido ajustada para permitir áreas de depósito de 1cm2 a 25cm2. El dispositivo fabricado, los llamados condensadores de metal aislante de nanopartículas (NP-MIM) ofrecen valores de capacidad más altos que un condensador convencional similar con una capa aislante continua. En el caso de los NP-MIM de sílice, se alcanza un factor de hasta 1000 de mejora de la capacidad, mientras que los NP-MIM de poliestireno exhibe una ganancia de capacidad en el rango de 11. Además, los NP-MIM de sílice muestran comportamientos capacitivos en específicos rangos de frecuencias que depende de la humedad y el grosor de la capa de nanopartículas, mientras que los NP-MIM de poliestireno siempre mantienen su comportamiento capacitivo. Los dispositivos fabricados se han caracterizado mediante medidas de microscopía electrónica de barrido (SEM) complementadas con perforaciones de haz de iones focalizados (FIB) para caracterizar la topografía de los NP-MIMs. Los dispositivos también se han caracterizado por medidas de espectroscopia de impedancia, a diferentes temperaturas y humedades. El origen de la capacitancia aumentada está asociado en parte a la humedad en las interfaces de las nanopartículas. Se ha desarrollado un modelo de un circuito basado en elementos distribuidos para ajustar y predecir el comportamiento eléctrico de los NP-MIMs. En resumen, esta tesis muestra el diseño, fabricación, caracterización y modelización de un nuevo y prometedor condensador nanopartículas metal-aislante-metal que puede abrir el camino al desarrollo de una nueva tecnología de supercondensadores MIM.Postprint (published version

Reliability-aware circuit design to mitigate impact of device defects and variability in emerging memristor-based applications

In the last decades, semiconductor industry has fostered a fast downscale in technology, propelling the large scale integration of CMOS-based systems. The benefits in miniaturization are numerous, highlighting faster switching frequency, lower voltage supply and higher device density. However, this aggressive scaling trend it has not been without challenges, such as leakage currents, yield reduction or the increase in the overall system power dissipation. New materials, changes in the device structures and new architectures are key to keep the miniaturization trend. It is foreseen that 2D integration will eventually come to an insurmountable physical and economic limit, in which new strategic directions are required, such as the development of new device structures, 3D architectures or heterogeneous systems that takes advantage of the best of different technologies, both the ones already consolidated as well as emergent ones that provide performance and efficiency improvements in applications. In this context, memristor arises as one of several candidates in the race to find suitable emergent devices. Memristor, a blend of the words memory and resistor, is a passive device postulated by Leon Chua in 1971. In contrast with the other fundamental passive elements, memristors have the distinctive feature of modifying their resistance according to the charge that passes through these devices, and remaining unaltered when charge no longer flows. Although when it appeared no physical device implementation was acknowledged, HP Labs claimed in 2008 the manufacture of the first real memristor. This milestone triggered an unexpectedly high research activity about memristors, both in searching new materials and structures as well as in potential applications. Nowadays, memristors are not only appreciated in memory systems by their nonvolatile storage properties, but in many other fields, such as digital computing, signal processing circuits, or non-conventional applications like neuromorphic computing or chaotic circuits. In spite of their promising features, memristors show a primarily downside: they show significant device variation and limited lifetime due degradation compared with other alternatives. This Thesis explores the challenges that memristor variation and malfunction imposes in potential applications. The main goal is to propose circuits and strategies that either avoid reliability problems or take advantage of them. Throughout a collection of scenarios in which reliability issues are present, their impact is studied by means of simulations. This thesis is contextualized and their objectives are exposed in Chapter 1. In Chapter 2 the memristor is introduced, at both conceptual and experimental levels, and different compact levels are presented to be later used in simulations. Chapter 3 deepens in the phenomena that causes the lack of reliability in memristors, and models that include these defects in simulations are provided. The rest of the Thesis covers different applications. Therefore, Chapter 4 exhibits nonvolatile memory systems, and specifically an online test method for faulty cells. Digital computing is presented in Chapter 5, where a solution for the yield reduction in logic operations due to memristors variability is proposed. Lastly, Chapter 6 reviews applications in the analog domain, and it focuses in the exploitation of results observed in faulty memristor-based interconnect mediums for chaotic systems synchronization purposes. Finally, the Thesis concludes in Chapter 7 along with perspectives about future work.Este trabajo desarrolla un novedoso dispositivo condensador basado en el uso de la nanotecnología. El dispositivo parte del concepto existente de metal-aislador-metal (MIM), pero en lugar de una capa aislante continua, se utilizan nanopartículas dieléctricas. Las nanopartículas son principalmente de óxido de silicio (sílice) y poliestireno (PS) y los valores de diámetro son 255nm y 295nm respectivamente. Las nanopartículas contribuyen a una alta relación superficie/volumen y están fácilmente disponibles a bajo costo. La tecnología de depósito desarrollada en este trabajo se basa en la técnica de electrospray, que es una tecnología de fabricación ascendente (bottom-up) que permite el procesamiento por lotes y logra un buen compromiso entre una gran superficie y un bajo tiempo de depósito. Con el objetivo de aumentar la superficie de depósito, la configuración de electrospray ha sido ajustada para permitir áreas de depósito de 1cm2 a 25cm2. El dispositivo fabricado, los llamados condensadores de metal aislante de nanopartículas (NP-MIM) ofrecen valores de capacidad más altos que un condensador convencional similar con una capa aislante continua. En el caso de los NP-MIM de sílice, se alcanza un factor de hasta 1000 de mejora de la capacidad, mientras que los NP-MIM de poliestireno exhibe una ganancia de capacidad en el rango de 11. Además, los NP-MIM de sílice muestran comportamientos capacitivos en específicos rangos de frecuencias que depende de la humedad y el grosor de la capa de nanopartículas, mientras que los NP-MIM de poliestireno siempre mantienen su comportamiento capacitivo. Los dispositivos fabricados se han caracterizado mediante medidas de microscopía electrónica de barrido (SEM) complementadas con perforaciones de haz de iones focalizados (FIB) para caracterizar la topografía de los NP-MIMs. Los dispositivos también se han caracterizado por medidas de espectroscopia de impedancia, a diferentes temperaturas y humedades. El origen de la capacitancia aumentada está asociado en parte a la humedad en las interfaces de las nanopartículas. Se ha desarrollado un modelo de un circuito basado en elementos distribuidos para ajustar y predecir el comportamiento eléctrico de los NP-MIMs. En resumen, esta tesis muestra el diseño, fabricación, caracterización y modelización de un nuevo y prometedor condensador nanopartículas metal-aislante-metal que puede abrir el camino al desarrollo de una nueva tecnología de supercondensadores MIM

Memristive crossbars as hardware accelerators: modelling, design and new uses

Digital electronics has given rise to reliable, affordable, and scalable computing devices. However, new computing paradigms present challenges. For example, machine learning requires repeatedly processing large amounts of data; this creates a bottleneck in conventional computers, where computing and memory are separated. To add to that, Moore’s “law” is plateauing and is thus unlikely to address the increasing demand for computational power. In-memory computing, and specifically hardware accelerators for linear algebra, may address both of these issues. Memristive crossbar arrays are a promising candidate for such hardware accelerators. Memristive devices are fast, energy-efficient, and—when arranged in a crossbar structure—can compute vector-matrix products. Unfortunately, they come with their own set of limitations. The analogue nature of these devices makes them stochastic and thus less reliable compared to digital devices. It does not, however, necessarily make them unsuitable for computing. Nevertheless, successful deployment of analogue hardware accelerators requires a proper understanding of their drawbacks, ways of mitigating the effects of undesired physical behaviour, and applications where some degree of stochasticity is tolerable. In this thesis, I investigate the effects of nonidealities in memristive crossbar arrays, introduce techniques of minimising those negative effects, and present novel crossbar circuit designs for new applications. I mostly focus on physical implementations of neural networks and investigate the influence of device nonidealities on classification accuracy. To make memristive neural networks more reliable, I explore committee machines, rearrangement of crossbar lines, nonideality-aware training, and other techniques. I find that they all may contribute to the higher accuracy of physically implemented neural networks, often comparable to the accuracy of their digital counterparts. Finally, I introduce circuits that extend dot product computations to higher-rank arrays, different linear algebra operations, and quaternion vectors and matrices. These present opportunities for using crossbar arrays in new ways, including the processing of coloured images