Pruning random resistive memory for optimizing analogue AI

The rapid advancement of artificial intelligence (AI) has been marked by the large language models exhibiting human-like intelligence. However, these models also present unprecedented challenges to energy consumption and environmental sustainability. One promising solution is to revisit analogue computing, a technique that predates digital computing and exploits emerging analogue electronic devices, such as resistive memory, which features in-memory computing, high scalability, and nonvolatility. However, analogue computing still faces the same challenges as before: programming nonidealities and expensive programming due to the underlying devices physics. Here, we report a universal solution, software-hardware co-design using structural plasticity-inspired edge pruning to optimize the topology of a randomly weighted analogue resistive memory neural network. Software-wise, the topology of a randomly weighted neural network is optimized by pruning connections rather than precisely tuning resistive memory weights. Hardware-wise, we reveal the physical origin of the programming stochasticity using transmission electron microscopy, which is leveraged for large-scale and low-cost implementation of an overparameterized random neural network containing high-performance sub-networks. We implemented the co-design on a 40nm 256K resistive memory macro, observing 17.3% and 19.9% accuracy improvements in image and audio classification on FashionMNIST and Spoken digits datasets, as well as 9.8% (2%) improvement in PR (ROC) in image segmentation on DRIVE datasets, respectively. This is accompanied by 82.1%, 51.2%, and 99.8% improvement in energy efficiency thanks to analogue in-memory computing. By embracing the intrinsic stochasticity and in-memory computing, this work may solve the biggest obstacle of analogue computing systems and thus unleash their immense potential for next-generation AI hardware

Real-time Image Processing for Microscopy-based Label-free Imaging Flow Cytometry in a Microfluidic Chip

Imaging flow cytometry (IFC) is an emerging technology that acquires single-cell images at high-throughput for analysis of a cell population. Rich information that comes from high sensitivity and spatial resolution of a single-cell microscopic image is beneficial for single-cell analysis in various biological applications. In this paper, we present a fast image-processing pipeline (R-MOD: Real-time Moving Object Detector) based on deep learning for high-throughput microscopy-based label-free IFC in a microfluidic chip. The R-MOD pipeline acquires all single-cell images of cells in flow, and identifies the acquired images as a real-time process with minimum hardware that consists of a microscope and a high-speed camera. Experiments show that R-MOD has the fast and reliable accuracy (500 fps and 93.3% mAP), and is expected to be used as a powerful tool for biomedical and clinical applications.113Ysciescopu

Computer Simulation of a Nitric Oxide-Releasing Catheter with a Novel Stable Convection-Diffusion Equation Solver and Automatic Quantification of Lung Ultrasound Comets by Machine Learning

Biological transport processes often involve a boundary acting as separation of flow, most commonly in transport involving blood-contacting medical devices. The separation of flow creates two different scenarios of mass transport across the interface. No flow exists within the medical device and diffusion governs mass transport; both convection and diffusion exist when flow is present. The added convection creates a large concentration gradient around the interface. Computer simulation of such cases prove to be difficult and require proper shock capturing methods for the solutions to be stable, which is typically lacking in commercial solvers. In this thesis, we propose a second-order accurate numerical method for solving the convection-diffusion equation by using a gradient-limited Godunov-type convective flux and the multi-point flux approximation (MPFA) L-Method for the diffusion flux. We applied our solver towards simulation of a nitric oxide-releasing intravascular catheter. Intravascular catheters are essential for long-term vascular access in both diagnosis and treatment. Use of catheters are associated with risks for infection and thrombosis. Because infection and thrombosis lead to impaired flow and potentiality life threatening systemic infections, this leads to increased morbidity and mortality, requiring catheters to be replaced among other treatments for these complications. Nitric oxide (NO) is a potent antimicrobial and antithrombotic agent produced by vascular endothelial cells. The production level in vivo is so low that the physiological effects can only be seen around the endothelial cells. The catheter can incorporate a NO source in two major ways: by impregnating the catheter with NO-releasing compounds such as S-nitroso-N-acetyl penicillamine (SNAP) or using electrochemical reactions to generate NO from nitrites. We applied our solver to both situations to guide the design of the catheter. Simulations revealed that dissolved NO inside the catheter is depleted after 12 minutes without resupplying, and electrochemical release of NO requires 10.5 minutes to reach steady state. Lung edema is often present in patients with end-stage renal disease due to reduced filtration functions of the kidney. These patients require regular dialysis sessions to manage their fluid status. The clinical gold standard to quantify lung edema is to use CT, which exposes patients to high amounts of radiation and is not cost efficient. Fluid management in such patients becomes very challenging without a clear guideline of fluid to be removed during dialysis sessions. Hypotension during dialysis can limit fluid removal, even in the setting of ongoing fluid overload or congestive heart failure. Accurate assessment of the pulmonary fluid status is needed, so that fluid overload and congestive heart failure can be detected, especially in the setting of hypotension, allowing dialysis to be altered to improve fluid removal. Recently, reverberations in ultrasound signals, referred to as ``lung comets'' have emerged as a potential quantitative way to measure lung edema. Increased presence of lung comets is associated with higher amounts of pulmonary edema, higher mortality, and more adverse cardiac events. However, the lung comets are often counted by hand by physicians with single frames in lung ultrasound and high subjectivity has been found to exist among the counting by physicians. We applied image processing and neural network techniques as an attempt to provide an objective and accurate measurement of the amount of lung comets present. Our quantitative results are significantly correlated with diastolic blood pressure and ejection fraction.PHDBiomedical EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/163182/1/micw_1.pd

On the use of deep learning for phase recovery

Phase recovery (PR) refers to calculating the phase of the light field from its intensity measurements. As exemplified from quantitative phase imaging and coherent diffraction imaging to adaptive optics, PR is essential for reconstructing the refractive index distribution or topography of an object and correcting the aberration of an imaging system. In recent years, deep learning (DL), often implemented through deep neural networks, has provided unprecedented support for computational imaging, leading to more efficient solutions for various PR problems. In this review, we first briefly introduce conventional methods for PR. Then, we review how DL provides support for PR from the following three stages, namely, pre-processing, in-processing, and post-processing. We also review how DL is used in phase image processing. Finally, we summarize the work in DL for PR and outlook on how to better use DL to improve the reliability and efficiency in PR. Furthermore, we present a live-updating resource (https://github.com/kqwang/phase-recovery) for readers to learn more about PR.Comment: 82 pages, 32 figure

Automatic recognition of different types of acute leukaemia using peripheral blood cell images

[eng] Clinical pathologists have learned to identify morphological qualitative features to characterise the different normal cells, as well as the abnormal cell types whose presence in peripheral blood is the evidence of serious haematological diseases. A drawback of visual morphological analysis is that is time consuming, requires well-trained personnel and is prone to intra-observer variability, which is particularly true when dealing with blast cells. Indeed, subtle interclass morphological differences exist for leukaemia types, which turns into low specificity scores in the routine screening. They are well-known the difficulties that clinical pathologists have in the discrimination among different blasts and the subjectivity associated with their morphological recognition. The general objective of this thesis is the automatic recognition of different types of blast cells circulating in peripheral blood in acute leukaemia using digital image processing and machine learning techniques. In order to accomplish this objective, this thesis starts with a discrimination among normal mononuclear cells, reactive lymphocytes and three types of leukemic cells using traditional machine learning techniques and hand-crafted features obtained from cell segmentation. In the second part of the thesis, a new predictive system designed with two serially connected convolutional neural networks is developed for the diagnosis of acute leukaemia. This system was proved to distinguish neoplastic (leukaemia) and non-neoplastic (infections) diseases, as well as recognise the leukaemia lineage. Furthermore, it was evaluated for its integration in a real-clinical setting. This thesis also contributes in advancing the state of the art of the automatic recognition of acute leukaemia by providing a more realistic approach which reflects the real-life complexity of acute leukaemia diagnosis

Detection of acute promyelocytic leukemia in peripheral blood and bone marrow with annotation-free deep learning

While optical microscopy inspection of blood films and bone marrow aspirates by a hematologist is a crucial step in establishing diagnosis of acute leukemia, especially in low-resource settings where other diagnostic modalities are not available, the task remains time-consuming and prone to human inconsistencies. This has an impact especially in cases of Acute Promyelocytic Leukemia (APL) that require urgent treatment. Integration of automated computational hematopathology into clinical workflows can improve the throughput of these services and reduce cognitive human error. However, a major bottleneck in deploying such systems is a lack of sufficient cell morphological object-labels annotations to train deep learning models. We overcome this by leveraging patient diagnostic labels to train weakly-supervised models that detect different types of acute leukemia. We introduce a deep learning approach, Multiple Instance Learning for Leukocyte Identification (MILLIE), able to perform automated reliable analysis of blood films with minimal supervision. Without being trained to classify individual cells, MILLIE differentiates between acute lymphoblastic and myeloblastic leukemia in blood films. More importantly, MILLIE detects APL in blood films (AUC 0.94 ± 0.04) and in bone marrow aspirates (AUC 0.99 ± 0.01). MILLIE is a viable solution to augment the throughput of clinical pathways that require assessment of blood film microscopy

이중 합성곱 신경망을 이용한 백혈구 백분율 자동 분석 시스템에 관한 연구

학위논문 (박사)-- 서울대학교 대학원 공과대학 협동과정 바이오엔지니어링전공, 2017. 8. 김희찬.Leukocyte or white blood cell differential count is an essential examination modality of hematology laboratory in diagnosis of various blood disorders. However, it requires highly experienced hematologists for correct diagnosis from samples with inter- and intra-sample variations. Due to tedious, time and cost consuming procedure of manual differential count, there has been high demands for development of automated system. In order for it to be applicable in clinical hematology laboratories, an automated system will have to detect and classify leukocytes of different maturation stages, especially in bone marrow aspirate smears. This has been a challenging problem in computer vision, image processing, and machine learning, because of complex nature of bone marrow aspirate smear. The leukocyte has multiple maturation stages, and these maturation stages have small inter-class differences, so it is difficult to differentiate even with expert knowledge. Moreover, a problem of color, shape, and size variations among samples exists and a problem of touching cell due to high leukocyte density of bone marrow aspirate smear exists. In this dissertation, an automated leukocyte differential count system for bone marrow aspirate smear was developed to overcome problems of manual differential count and to fulfill clinically unmet needs. The system should perform the differential count with high accuracy and objectivity, and high throughput and efficiency. Moreover, it should overcome challenges of bone marrow aspirate smear. To this end, a large dataset of bone marrow smear was collected for development of a detection and a classification algorithms. Watershed transformation and saliency map were utilized for single-leukocyte detection, and the dual-stage convolutional neural network that learns global and local features of complex leukocyte maturation stages was proposed for classification. Lastly, a probability guidance algorithm was proposed for integration of detection and classification algorithms. The performance of proposed system was assessed with ten leukocyte maturation stages of myeloid and erythroid series in bone marrow aspirate smears. Total of 200 large (1388 × 1040) digital images of bone marrow aspirate smears and 2,323 small (96 × 96) single leukocyte digital images were collected. The proposed system showed a state-of-the-art performance. It achieved an average detection accuracy of 96.09% and an average classification accuracy of 97.06%, and it was able to differential count 100 leukocytes in 4 to 5 seconds. This proposes a new paradigm in diagnosis of blood disorder and showed a potential of deep learning, especially the convolutional neural network, in medical image processing. The proposed system is expected to increase the total number of analyzed leukocytes in a sample, which will provide more statistically reliable information of a patient for diagnosis.Chapter 1 Introduction 1 1.1. Introduction to Hematology 2 1.2. Introduction to Convolutional Neural Network 12 1.3. Thesis Objectives 16 Chapter 2 Leukocyte Data Collection 19 2.1. Sample Preparation and Acquisition 20 2.2. Dataset Collection and Preparation 23 Chapter 3 Leukocyte Classification 27 3.1. Introduction 28 3.2. Methods 36 3.2.1. Data Collection and Preparation 36 3.2.2. Data Oversampling and Augmentation 38 3.2.3. Convolutional Neural Network Architecture and Dual-stage Convolutional Neural Network 40 3.2.4. Convolutional Neural Network Training 43 3.2.5. Implementation 46 3.2.6. Evaluation Metrics 46 3.3. Results and Discussion 48 3.4. Conclusion 66 Chapter 4 Implementation of Automated Leukocyte Differential Count System 67 4.1. System Overview 68 4.2. Leukocyte Detection 70 4.2.1. Introduction 70 4.2.2. Detection Algorithm 75 4.2.3. Experimental Setup and Evaluation 81 4.2.4. Results and Discussion 82 4.3. Automated Leukocyte Differential Count System 92 4.3.1. Implementation of Detection and Classification Algorithms 92 4.3.2. Graphical User Interface Design 93 4.3.3. Probability Guidance Algorithm 95 4.3.4. Experimental Setup and Evaluation 97 4.3.5. Results and Discussion 98 4.4. Conclusion 102 Chapter 5 Thesis Summary and Future Work 104 5.1 Thesis Summary and Contributions 105 5.2 Future Work 109 Bibliography 115 Abstract in Korean 122Docto

Content aware multi-focus image fusion for high-magnification blood film microscopy

Automated digital high-magnification optical microscopy is key to accelerating biology research and improving pathology clinical pathways. High magnification objectives with large numerical apertures are usually preferred to resolve the fine structural details of biological samples, but they have a very limited depth-of-field. Depending on the thickness of the sample, analysis of specimens typically requires the acquisition of multiple images at different focal planes for each field-of-view, followed by the fusion of these planes into an extended depth-of-field image. This translates into low scanning speeds, increased storage space, and processing time not suitable for high-throughput clinical use. We introduce a novel content-aware multi-focus image fusion approach based on deep learning which extends the depth-of-field of high magnification objectives effectively. We demonstrate the method with three examples, showing that highly accurate, detailed, extended depth of field images can be obtained at a lower axial sampling rate, using 2-fold fewer focal planes than normally required

A PCNN Framework for Blood Cell Image Segmentation

This research presents novel methods for segmenting digital blood cell images under a Pulse Coupled Neural Network (PCNN) framework. A blood cell image contains different types of blood cells found in the peripheral blood stream such as red blood cells (RBCs), white blood cells (WBCs), and platelets. WBCs can be classified into five normal types – neutrophil, monocyte, lymphocyte, eosinophil, and basophil – as well as abnormal types such as lymphoblasts and others. The focus of this research is on identifying and counting RBCs, normal types of WBCs, and lymphoblasts. The total number of RBCs and WBCs, along with classification of WBCs, has important medical significance which includes providing a physician with valuable information for diagnosis of diseases such as leukemia. The approach comprises two phases – segmentation and cell separation – followed by classification of WBC types including detection of lymphoblasts. The first phase presents two methods based on PCNN and region growing to segment followed by a separate method that combines Circular Hough Transform (CHT) with a separation algorithm to find and separate each RBC and WBC object into separate images. The first method uses a standard PCNN to segment. The second method uses a region growing PCNN with a maximum region size to segment. The second phase presents a WBC classification method based on PCNN. It uses a PCNN to capture the texture features of an image as a sequence of entropy values known as a texture vector. First, the parameters of the texture vector PCNN are defined. This is then used to produce texture vectors for the training images. Each cell type is represented by several texture vectors across its instances. Then, given a test image to be classified, the texture vector PCNN is used to capture its texture vector, which is compared to the texture vectors for classification. This two-phase approach yields metrics based on the RBC and WBC counts, WBC classification, and identification of lymphoblasts. Both the standard and region growing PCNNs were successful in segmenting RBC and WBC objects, with better accuracy when using the standard PCNN. The separate method introduced with this research provided accurate WBC counts but less accurate RBC counts. The WBC subimages created with the separate method facilitated cell counting and WBC classification. Using a standard PCNN as a WBC classifier, introduced with this research, proved to be a successful classifier and lymphoblast detector. While RBC accuracy was low, WBC accuracy for total counts, WBC classification, and lymphoblast detection were overall above 96%