Technology-Circuit-Algorithm Tri-Design for Processing-in-Pixel-in-Memory (P2M)

The massive amounts of data generated by camera sensors motivate data processing inside pixel arrays, i.e., at the extreme-edge. Several critical developments have fueled recent interest in the processing-in-pixel-in-memory paradigm for a wide range of visual machine intelligence tasks, including (1) advances in 3D integration technology to enable complex processing inside each pixel in a 3D integrated manner while maintaining pixel density, (2) analog processing circuit techniques for massively parallel low-energy in-pixel computations, and (3) algorithmic techniques to mitigate non-idealities associated with analog processing through hardware-aware training schemes. This article presents a comprehensive technology-circuit-algorithm landscape that connects technology capabilities, circuit design strategies, and algorithmic optimizations to power, performance, area, bandwidth reduction, and application-level accuracy metrics. We present our results using a comprehensive co-design framework incorporating hardware and algorithmic optimizations for various complex real-life visual intelligence tasks mapped onto our P2M paradigm

Enhancing Reliability of Neural Networks at the Edge: Inverted Normalization with Stochastic Affine Transformations

Bayesian Neural Networks (BayNNs) naturally provide uncertainty in their predictions, making them a suitable choice in safety-critical applications. Additionally, their realization using memristor-based in-memory computing (IMC) architectures enables them for resource-constrained edge applications. In addition to predictive uncertainty, however, the ability to be inherently robust to noise in computation is also essential to ensure functional safety. In particular, memristor-based IMCs are susceptible to various sources of non-idealities such as manufacturing and runtime variations, drift, and failure, which can significantly reduce inference accuracy. In this paper, we propose a method to inherently enhance the robustness and inference accuracy of BayNNs deployed in IMC architectures. To achieve this, we introduce a novel normalization layer combined with stochastic affine transformations. Empirical results in various benchmark datasets show a graceful degradation in inference accuracy, with an improvement of up to $58.11\%$

Digital CMOS ISFET architectures and algorithmic methods for point-of-care diagnostics

Over the past decade, the surge of infectious diseases outbreaks across the globe is redefining how healthcare is provided and delivered to patients, with a clear trend towards distributed diagnosis at the Point-of-Care (PoC). In this context, Ion-Sensitive Field Effect Transistors (ISFETs) fabricated on standard CMOS technology have emerged as a promising solution to achieve a precise, deliverable and inexpensive platform that could be deployed worldwide to provide a rapid diagnosis of infectious diseases. This thesis presents advancements for the future of ISFET-based PoC diagnostic platforms, proposing and implementing a set of hardware and software methodologies to overcome its main challenges and enhance its sensing capabilities. The first part of this thesis focuses on novel hardware architectures that enable direct integration with computational capabilities while providing pixel programmability and adaptability required to overcome pressing challenges on ISFET-based PoC platforms. This section explores oscillator-based ISFET architectures, a set of sensing front-ends that encodes the chemical information on the duty cycle of a PWM signal. Two initial architectures are proposed and fabricated in AMS 0.35um, confirming multiple degrees of programmability and potential for multi-sensing. One of these architectures is optimised to create a dual-sensing pixel capable of sensing both temperature and chemical information on the same spatial point while modulating this information simultaneously on a single waveform. This dual-sensing capability, verified in silico using TSMC 0.18um process, is vital for DNA-based diagnosis where protocols such as LAMP or PCR require precise thermal control. The COVID-19 pandemic highlighted the need for a deliverable diagnosis that perform nucleic acid amplification tests at the PoC, requiring minimal footprint by integrating sensing and computational capabilities. In response to this challenge, a paradigm shift is proposed, advocating for integrating all elements of the portable diagnostic platform under a single piece of silicon, realising a ``Diagnosis-on-a-Chip". This approach is enabled by a novel Digital ISFET Pixel that integrates both ADC and memory with sensing elements on each pixel, enhancing its parallelism. Furthermore, this architecture removes the need for external instrumentation or memories and facilitates its integration with computational capabilities on-chip, such as the proposed ARM Cortex M3 system. These computational capabilities need to be complemented with software methods that enable sensing enhancement and new applications using ISFET arrays. The second part of this thesis is devoted to these methods. Leveraging the programmability capabilities available on oscillator-based architectures, various digital signal processing algorithms are implemented to overcome the most urgent ISFET non-idealities, such as trapped charge, drift and chemical noise. These methods enable fast trapped charge cancellation and enhanced dynamic range through real-time drift compensation, achieving over 36 hours of continuous monitoring without pixel saturation. Furthermore, the recent development of data-driven models and software methods open a wide range of opportunities for ISFET sensing and beyond. In the last section of this thesis, two examples of these opportunities are explored: the optimisation of image compression algorithms on chemical images generated by an ultra-high frame-rate ISFET array; and a proposed paradigm shift on surface Electromyography (sEMG) signals, moving from data-harvesting to information-focused sensing. These examples represent an initial step forward on a journey towards a new generation of miniaturised, precise and efficient sensors for PoC diagnostics.Open Acces

Simulation and implementation of novel deep learning hardware architectures for resource constrained devices

Corey Lammie designed mixed signal memristive-complementary metal–oxide–semiconductor (CMOS) and field programmable gate arrays (FPGA) hardware architectures, which were used to reduce the power and resource requirements of Deep Learning (DL) systems; both during inference and training. Disruptive design methodologies, such as those explored in this thesis, can be used to facilitate the design of next-generation DL systems

On Spike-Timing-Dependent-Plasticity, Memristive Devices, and Building a Self-Learning Visual Cortex

In this paper we present a very exciting overlap between emergent nanotechnology and neuroscience, which has been discovered by neuromorphic engineers. Specifically, we are linking one type of memristor nanotechnology devices to the biological synaptic update rule known as spike-time-dependent-plasticity (STDP) found in real biological synapses. Understanding this link allows neuromorphic engineers to develop circuit architectures that use this type of memristors to artificially emulate parts of the visual cortex. We focus on the type of memristors referred to as voltage or flux driven memristors and focus our discussions on a behavioral macro-model for such devices. The implementations result in fully asynchronous architectures with neurons sending their action potentials not only forward but also backward. One critical aspect is to use neurons that generate spikes of specific shapes. We will see how by changing the shapes of the neuron action potential spikes we can tune and manipulate the STDP learning rules for both excitatory and inhibitory synapses. We will see how neurons and memristors can be interconnected to achieve large scale spiking learning systems, that follow a type of multiplicative STDP learning rule. We will briefly extend the architectures to use three-terminal transistors with similar memristive behavior. We will illustrate how a V1 visual cortex layer can assembled and how it is capable of learning to extract orientations from visual data coming from a real artificial CMOS spiking retina observing real life scenes. Finally, we will discuss limitations of currently available memristors. The results presented are based on behavioral simulations and do not take into account non-idealities of devices and interconnects. The aim of this paper is to present, in a tutorial manner, an initial framework for the possible development of fully asynchronous STDP learning neuromorphic architectures exploiting two or three-terminal memristive type devices. All files used for the simulations are made available through the journal web site1

2022 roadmap on neuromorphic computing and engineering

Modern computation based on von Neumann architecture is now a mature cutting-edge science. In the von Neumann architecture, processing and memory units are implemented as separate blocks interchanging data intensively and continuously. This data transfer is responsible for a large part of the power consumption. The next generation computer technology is expected to solve problems at the exascale with 10$^{18}$ calculations each second. Even though these future computers will be incredibly powerful, if they are based on von Neumann type architectures, they will consume between 20 and 30 megawatts of power and will not have intrinsic physically built-in capabilities to learn or deal with complex data as our brain does. These needs can be addressed by neuromorphic computing systems which are inspired by the biological concepts of the human brain. This new generation of computers has the potential to be used for the storage and processing of large amounts of digital information with much lower power consumption than conventional processors. Among their potential future applications, an important niche is moving the control from data centers to edge devices. The aim of this roadmap is to present a snapshot of the present state of neuromorphic technology and provide an opinion on the challenges and opportunities that the future holds in the major areas of neuromorphic technology, namely materials, devices, neuromorphic circuits, neuromorphic algorithms, applications, and ethics. The roadmap is a collection of perspectives where leading researchers in the neuromorphic community provide their own view about the current state and the future challenges for each research area. We hope that this roadmap will be a useful resource by providing a concise yet comprehensive introduction to readers outside this field, for those who are just entering the field, as well as providing future perspectives for those who are well established in the neuromorphic computing community