Acceleration of CNN Computation on a PIM-enabled GPU system

학위논문(박사) -- 서울대학교대학원 : 공과대학 전기·정보공학부, 2022. 8. 이혁재.최근, convolutional neural network (CNN)은 image processing 및 computer vision 등에서 널리 사용되고 있다. CNN은 연산 집약적인 convolutional layer와 메모리 집약적인 fully connected layer, batch normalization layer 및 activation layer 등 다양한 layer로 구성된다. 일반적으로 CNN을 처리하기 위해 GPU가 널리 사용되지만, CNN은 연산 집약적인 동시에 메모리 집약적이기에 성능이 제한된다. 또한, 고화질의 image 및 video application의 사용은 GPU와 메모리 간의 data 이동에 의한 부담을 증가시킨다. Processing-in-memory는 메모리에 연산기를 탑재하여 data 이동에 의한 부담을 줄일 수 있어, host GPU와 PIM을 함께 사용하는 system은 CNN을 처리하기에 적합하다. 먼저 convolutional layer의 연산량을 감소시키기 위해, 근사 연산을 수행할 수 있다. 그러나 기존의 근사 연산은 host GPU로 data를 load 한 후 data 간 유사도를 파악하기에, GPU와 DRAM 간의 data 이동량을 줄이지는 못한다. 이는 메모리 intensity를 증가시켜 메모리 bottleneck을 유발한다. 게다가, 근사 연산으로 인해 warp 간 load imbalance 또한 발생하게 되어 성능이 저하된다. 이를 해결하기 위해, 본 논문에서는 data 간 근사 비교를 PIM에서 수행하는 방법을 제안한다. 제안하는 방법은 PIM에서 data간 유사도를 파악한 후, 대표 data와 유사도 정보만을 GPU로 전송한다. GPU는 대표 data에 대해서만 연산을 수행하고, 유사도 정보에 기반하여 해당 결과를 재사용하여 최종 결과를 생성한다. 이때, 메모리에서의 data 비교로 인한 latency 증가를 방지하기 위해 DRAM의 bank 단과 TSV 단을 모두 활용하는 2-level PIM 구조를 제안한다. 또한, 대표 data를 적당한 address에 재배치한 후 GPU로 전송하여 GPU에서의 별도 작업 없이 load balancing을 가능하게 한다. 다음으로, batch normalization 등 non-convolutional layer의 높은 메모리 사용량으로 인한 메모리 bottleneck 문제를 해결하기 위해 PIM에서 non-convolutional layer를 수행할 수 있다. 기존 연구에서는 PIM으로 non-convolutional layer를 가속하였지만, 단순히 GPU와 PIM이 순차적으로 동작하는 상황을 가정하여 성능 향상에 한계가 있었다. 제안하는 방법은 non-convolutional layer가 ouptut feature map의 channel 단위로 수행된다는 점에 착안하여 host와 PIM을 pipeline적으로 수행함으로써 CNN 학습을 가속한다. PIM은 host에서 convolution 연산이 끝난 output feature map의 channel에 대해 non-convolution 연산을 수행한다. 역전파 과정에서 발생하는 weight update와 feature map gradient 계산에서의 convolution과 non-convolution 간 job 균형을 위해, 적절하게 non-convolution job을 분배하여 성능을 향상시킨다. 이에 더해, host와 PIM이 동시에 memory에 access하는 상황에서 전체 수행 시간을 최소화하기 위해 bank 소유권 기반의 host와 PIM 간 memory scheduling 알고리즘을 제안한다. 마지막으로, image processing application 처리를 위해 logic die에 탑재 가능한 PIM GPU 구조를 제안한다. GPU 기반의 PIM은 CUDA 기반의 application을 수행할 수 있어 딥러닝 및 image application의 처리에 적합하지만, GPU의 큰 용량의 on-chip SRAM은 logic die에 충분한 수의 computing unit의 탑재를 어렵게 한다. 본 논문에서는 PIM에 적합한 최적의 lightweight GPU 구조와 함께 이를 활용하기 위한 최적화 기법을 제안한다. Image processing application의 메모리 접근 패턴과 data locality가 보존되도록 각 computing unit에 data를 할당하고, 예측 가능한 data의 할당을 기반으로 prefetcher를 탑재하여 lightweight한 구조임에도 충분한 수의 computing unit을 탑재하여 높은 성능을 확보한다.Recently, convolutional neural networks (CNN) have been widely used in image processing and computer vision. CNNs are composed of various layers such as computation-intensive convolutional layer and memory-intensive fully connected layer, batch normalization layer, and activation layer. GPUs are often used to accelerate the CNN, but performance is limited by high computational costs and memory usage of the convolution. Also, increasing demand for high resolution image applications increases the burden of data movement between GPU and memory. By performing computations on the memory, processing-in-memory (PIM) is expected to mitigate the overhead caused by data transfer. Therefore, a system that uses a PIM is promising for processing CNNs. First, prior studies exploited approximate computing to reduce the computational costs. However, they only reduced the amount of the computation, thereby its performance is bottlenecked by the memory bandwidth due to an increased memory intensity. In addition, load imbalance between warps caused by approximation also inhibits the performance improvement. This dissertation proposes a PIM solution that reduces the amount of data movement and computation through the Approximate Data Comparison (ADC-PIM). Instead of determining the value similarity on the GPU, the ADC-PIM located on memory compares the similarity and transfers only the selected data to the GPU. The GPU performs convolution on the representative data transferred from the ADC-PIM, and reuses the calculated results based on the similarity information. To reduce the increase in memory latency due to the data comparison, a two-level PIM architecture that exploits both the DRAM bank and TSV stage is proposed. To ease the load balancing on the GPU, the ADC-PIM reorganizes data by assigning the representative data to proposer addresses that are computed based on the comparison result. Second, to solve the memory bottleneck caused by the high memory usage, non-convolutional layers are accelerated with PIM. Previous studies also accelerated the non-convolutional layers by PIM, but there was a limit to performance improvement because they simply assumed a situation in which the GPU and PIM operate sequentially. The proposed method accelerates the CNN training with a pipelined execution of GPU and PIM, focusing on the fact that the non-convolution operation is performed in units of channels of the output feature map. PIM performs non-convolutional operations on the output feature map where the GPU has completed the convolution operation. To balance the jobs between convolution and non-convolution in weight update and feature map gradient calculation that occur in the back propagation process, non-convolution job is properly distributed to each process. In addition, a memory scheduling algorithm based on bank ownership between the host and PIM is proposed to minimize the overall execution time in a situation where the host and PIM simultaneously access memory. Finally, a GPU-based PIM architecture for image processing application is proposed. Programmable GPU-based PIM is attractive because it enables the utilization of well-crafted software development kits (SDKs) such as CUDA and openCL. However, the large capacity of on-chip SRAM of GPU makes it difficult to mount a sufficient number of computing units in logic die. This dissertation proposes a GPU-based PIM architecture and well-matched optimization strategies considering both the characteristics of image applications and logic die constraints. Data allocation to the computing unit is addressed to maintain the data locality and data access pattern. By applying a prefetcher that leverages the pattern-aware data allocation, the number of active warps and the on-chip SRAM size of the PIM are significantly reduced. This enables the logic die constraints to be satisfied and a greater number of computing units to be integrated on a logic die.제 1 장 서 론 1 1.1 연구의 배경 1 1.2 연구의 내용 3 1.3 논문 구성 4 제 2 장 연구의 배경 지식 5 2.1 High Bandwidth Memory 5 2.2 Processing-In-Memory 6 2.3 GPU의 구조 및 동작 모델 7 제 3 장 PIM을 활용한 근사적 데이터 비교 및 근사 연산을 통한 Convolution 가속 9 3.1 관련 연구 10 3.1.1 CNN에서의 Approximate Computing 10 3.1.2 Processing In Memory를 활용한 CNN 가속 11 3.2 Motivation 13 3.2.1 GPU에서 Convolution 연산 시의 Approximation 기회 13 3.2.2 Approxiamte Convolution 연산에서 발생하는 문제점 14 3.3 제안하는 ADC-PIM Design 18 3.3.1 Overview 18 3.3.2 Data 간 유사도 비교 방법 19 3.3.3 ADC-PIM 아키텍처 21 3.3.4 Load Balancing을 위한 Data Reorganization 27 3.4 GPU에서의 Approximate Convolution 31 3.4.1 Instruction Skip을 통한 Approximate Convolution 31 3.4.2 Approximate Convolution을 위한 구조적 지원 32 3.5 실험 결과 및 분석 36 3.5.1 실험 환경 구성 36 3.5.2 제안하는 각 방법의 영향 분석 38 3.5.3 기존 연구와의 성능 비교 41 3.5.4 에너지 소모량 비교 44 3.5.5 Design Overhead 분석 44 3.5.6 정확도에 미치는 영향 46 3.6 본 장의 결론 47 제 4 장 Convolutional layer와 non-Convolutional Layer의 Pipeline 실행을 통한 CNN 학습 가속 48 4.1 관련 연구 48 4.1.1 Non-CONV Lasyer의 Memory Bottleneck 완화 48 4.1.2 Host와 PIM 간 Memory Scheduling 49 4.2 Motivation 51 4.2.1 CONV와 non-CONV의 동시 수행 시 성능 향상 기회 51 4.2.2 PIM 우선도에 따른 host 및 PIM request의 처리 효율성 변화 52 4.3 제안하는 host-PIM Memory Scheduling 알고리즘 53 4.3.1 host-PIM System Overview 53 4.3.2 PIM Duration Based Memory Scheduling 53 4.3.3 최적 PD_TH 값의 계산 방법 56 4.4 제안하는 CNN 학습 동작 Flow 62 4.4.1 CNN 학습 순전파 과정 62 4.4.2 CNN 학습 역전파 과정 63 4.5 실험 결과 및 분석 67 4.5.1 실험 환경 구성 67 4.5.2 Layer 당 수행 시간 변화 68 4.5.3 역전파 과정에서의 non-CONV job 배분 효과 70 4.5.4 전체 Network Level에서의 수행 시간 변화 72 4.5.5 제안하는 최적 PD_TH 추정 방법의 정확도 및 선택 알고리즘의 수렴 속도 74 4.6 본 장의 결론 75 제 5 장 Image processing의 데이터 접근 패턴을 활용한 PIM에 적합한 lightweight GPU 구조 76 5.1 관련 연구 77 5.1.1 Processing In Memory 77 5.1.2 GPU에서의 CTA Scheduling 78 5.1.3 GPU에서의 Prefetching 78 5.2 Motivation 79 5.2.1 PIM GPU system에서 Image Processing 알고리즘 처리 시 기존 GPU 구조의 비효율성 79 5.3 제안하는 GPU 기반 PIM System 82 5.3.1 Overview 82 5.3.2 Access Pattern을 고려한 CTA 할당 83 5.3.3 PIM GPU 구조 90 5.4 실험 결과 및 분석 94 5.4.1 실험 환경 구성 94 5.4.2 In-Depth Analysis 95 5.4.3 기존 연구와의 성능 비교 98 5.4.4 Cache Miss Rate 및 Memory Traffic 102 5.4.5 에너지 소모량 비교 103 5.4.6 PIM의 면적 및 전력 소모량 분석 105 5.5 본 장의 결론 107 제 6 장 결론 108 참고문헌 110 Abstract 118박

Towards Data Reliable, Low-Power, and Repairable Resistive Random Access Memories

A series of breakthroughs in memristive devices have demonstrated the potential of memristor arrays to serve as next generation resistive random access memories (ReRAM), which are fast, low-power, ultra-dense, and non-volatile. However, memristors' unique device characteristics also make them prone to several sources of error. Owing to the stochastic filamentary nature of memristive devices, various recoverable errors can affect the data reliability of a ReRAM. Permanent device failures further limit the lifetime of a ReRAM. This dissertation developed low-power solutions for more reliable and longer-enduring ReRAM systems. In this thesis, we first look into a data reliability issue known as write disturbance. Writing into a memristor in a crossbar could disturb the stored values in other memristors that are on the same memory line as the target cell. Such disturbance is accumulative over time which may lead to complete data corruption. To address this problem, we propose the use of two regular memristors on each word to keep track of the disturbance accumulation and trigger a refresh to restore the weakened data, once it becomes necessary. We also investigate the considerable variation in the write-time characteristics of individual memristors. With such variation, conventional fixed-pulse write schemes not only waste significant energy, but also cannot guarantee reliable completion of the write operations. We address such variation by proposing an adaptive write scheme that adjusts the width of the write pulses for each memristor. Our scheme embeds an online monitor to detect the completion of a write operation and takes into account the parasitic effect of line-shared devices in access-transistor-free memristive arrays. We further investigate the use of this method to shorten the test time of memory march algorithms by eliminating the need of a verifying read right after a write, which is commonly employed in the test sequences of march algorithms.Finally, we propose a novel mechanism to extend the lifetime of a ReRAM by protecting it against hard errors through the exploitation of a unique feature of bipolar memristive devices. Our solution proposes an unorthodox use of complementary resistive switches (a particular implementation of memristive devices) to provide an ``in-place spare'' for each memory cell at negligible extra cost. The in-place spares are then utilized by a repair scheme to repair memristive devices that have failed at a stuck-at-ON state at a page-level granularity. Furthermore, we explore the use of in-place spares in lieu of other memory reliability and yield enhancement solutions, such as error correction codes (ECC) and spare rows. We demonstrate that with the in-place spares, we can yield the same lifetime as a baseline ReRAM with either significantly fewer spare rows or a lighter-weight ECC, both of which can save on energy consumption and area

Associative Memristive Memory for Approximate Computing in GPUs

Using associative memories to enable computing-with-memory is a promising approach to improve energy efficiency. Associative memories can be tightly coupled with processing elements to restore and later recall function responses for a subset of input values. This approach avoids the actual function execution on the processing element to save on energy. The challenge, however, is to reduce the energy consumption of associative memory modules themselves. Here we address the challenge of designing ultra-low-power associative memories. We use memristive parts for memory implementation and demonstrate the energy saving potential of integrating associative memristive memory (AMM) into graphics processing units (GPUs). To reduce the energy consumption of AMM modules, we leverage approximate computing which benefits from application-level tolerance to errors: We employ voltage overscaling on AMM modules which deliberately relaxes its searching criteria to approximately match stored patterns within a 2 bit Hamming distance of the search pattern. This introduces some errors to the computation that are tolerable for target applications. We further reduce the energy consumption by employing purely resistive crossbar architectures for AMM modules. To evaluate the proposed architecture, we integrate AMM modules with floating point units in an AMD Southern Islands GPU and run four image processing kernels on an AMM-integrated GPU. Our experimental results show that employing AMM modules reduces energy consumption of running these kernels by 23%-45%, compared to a baseline GPU without AMM. The image processing kernels tolerate errors resulting from approximate search operations, maintaining an acceptable image quality, i.e., a PSNR above 30 dB