Get Out of the Valley: Power-Efficient Address Mapping for GPUs

GPU memory systems adopt a multi-dimensional hardware structure to provide the bandwidth necessary to support 100s to 1000s of concurrent threads. On the software side, GPU-compute workloads also use multi-dimensional structures to organize the threads. We observe that these structures can combine unfavorably and create significant resource imbalance in the memory subsystem causing low performance and poor power-efficiency. The key issue is that it is highly application-dependent which memory address bits exhibit high variability. To solve this problem, we first provide an entropy analysis approach tailored for the highly concurrent memory request behavior in GPU-compute workloads. Our window-based entropy metric captures the information content of each address bit of the memory requests that are likely to co-exist in the memory system at runtime. Using this metric, we find that GPU-compute workloads exhibit entropy valleys distributed throughout the lower order address bits. This indicates that efficient GPU-address mapping schemes need to harvest entropy from broad address-bit ranges and concentrate the entropy into the bits used for channel and bank selection in the memory subsystem. This insight leads us to propose the Page Address Entropy (PAE) mapping scheme which concentrates the entropy of the row, channel and bank bits of the input address into the bank and channel bits of the output address. PAE maps straightforwardly to hardware and can be implemented with a tree of XOR-gates. PAE improves performance by 1.31 x and power-efficiency by 1.25 x compared to state-of-the-art permutation-based address mapping

Intra-cluster coalescing to reduce GPU NoC pressure

GPUs continue to increase the number of streaming multiprocessors (SMs) to provide increasingly higher compute capabilities. To construct a scalable crossbar network-on-chip (NoC) that connects the SMs to the memory controllers, a cluster structure is introduced in modern GPUs in which several SMs are grouped together to share a network port. Because of network port sharing, clustered GPUs face severe NoC congestion, which creates a critical performance bottleneck. In this paper, we target redundant network traffic to mitigate GPU NoC congestion. In particular, we observe that in many GPU-compute applications, different SMs in a cluster access shared data. Issuing redundant requests to access the same memory location wastes valuable NoC bandwidth - we find on average 19.4% (and up to 48%) of the requests to be redundant. To reduce redundant NoC traffic, we propose intracluster coalescing (ICC) to merge memory requests from different SMs in a cluster. Our evaluation results show that ICC achieves an average performance improvement of 9.7% (and up to 33%) over a conventional design

Boustrophedonic Frames: Quasi-Optimal L2 Caching for Textures in GPUs

© 2023 Copyright held by the owner/author(s). This document is made available under the CC-BY-NC-ND 4.0 license http://creativecommons.org/licenses/by-nc-nd/4.0/ This document is the Accepted version of a Published Work that appeared in final form in 32nd International Conference on Parallel Architectures and Compilation Techniques (PACT), Viena, Austria, October 2023. To access the final edited and published work see https://doi.org/10.1109/PACT58117.2023.00019Literature is plentiful in works exploiting cache locality for GPUs. A majority of them explore replacement or bypassing policies. In this paper, however, we surpass this exploration by fabricating a formal proof for a no-overhead quasi-optimal caching technique for caching textures in graphics workloads. Textures make up a significant part of main memory traffic in mobile GPUs, which contributes to the total GPU energy consumption. Since texture accesses use a shared L2 cache, improving the L2 texture caching efficiency would decrease main memory traffic, thus improving energy efficiency, which is crucial for mobile GPUs. Our proposal reaches quasi-optimality by exploiting the frame-to-frame reuse of textures in graphics. We do this by traversing frames in a boustrophedonic1 manner w.r.t. the frame-to-frame tile order. We first approximate the texture access trace to a circular trace and then forge a formal proof for our proposal being optimal for such traces. We also complement the proof with empirical data that demonstrates the quasi-optimality of our no-cost proposal

CD-Xbar : a converge-diverge crossbar network for high-performance GPUs

Modern GPUs feature an increasing number of streaming multiprocessors (SMs) to boost system throughput. How to construct an efficient and scalable network-on-chip (NoC) for future high-performance GPUs is particularly critical. Although a mesh network is a widely used NoC topology in manycore CPUs for scalability and simplicity reasons, it is ill-suited to GPUs because of the many-to-few-to-many traffic pattern observed in GPU-compute workloads. Although a crossbar NoC is a natural fit, it does not scale to large SM counts while operating at high frequency. In this paper, we propose the converge-diverge crossbar (CD-Xbar) network with round-robin routing and topology-aware concurrent thread array (CTA) scheduling. CD-Xbar consists of two types of crossbars, a local crossbar and a global crossbar. A local crossbar converges input ports from the SMs into so-called converged ports; the global crossbar diverges these converged ports to the last-level cache (LLC) slices and memory controllers. CD-Xbar provides routing path diversity through the converged ports. Round-robin routing and topology-aware CTA scheduling balance network traffic among the converged ports within a local crossbar and across crossbars, respectively. Compared to a mesh with the same bisection bandwidth, CD-Xbar reduces NoC active silicon area and power consumption by 52.5 and 48.5 percent, respectively, while at the same time improving performance by 13.9 percent on average. CD-Xbar performs within 2.9 percent of an idealized fully-connected crossbar. We further demonstrate CD-Xbar's scalability, flexibility and improved performance perWatt (by 17.1 percent) over state-of-the-art GPU NoCs which are highly customized and non-scalable

Feluca : A two-stage graph coloring algorithm with color-centric paradigm on GPU

In this paper, we propose a two-stage high-performance graph coloring algorithm, called Feluca, aiming to address the above challenges. Feluca combines the recursion-based method with the sequential spread-based method. In the first stage, Feluca uses a recursive routine to color a majority of vertices in the graph. Then, it switches to the sequential spread method to color the remaining vertices in order to avoid the conflicts of the recursive algorithm. Moreover, the following techniques are proposed to further improve the graph coloring performance. i) A new method is proposed to eliminate the cycles in the graph; ii) a top-down scheme is developed to avoid the atomic operation originally required for color selection; and iii) a novel color-centric coloring paradigm is designed to improve the degree of parallelism for the sequential spread part. All these newly developed techniques, together with further GPU-specific optimizations such as coalesced memory access, comprise an efficient parallel graph coloring solution in Feluca. We have conducted extensive experiments on NVIDIA GPUs. The results show that Feluca can achieve 1.76 - 12.98x speedup over the state-of-the-art algorithms

Intra-cluster coalescing and distributed-block scheduling to reduce GPU NoC pressure

GPUs continue to boost the number of streaming multiprocessors (SMs) to provide increasingly higher compute capabilities. To construct a scalable crossbar network-on-chip (NoC) that connects the SMs to the memory controllers, a cluster structure is introduced in modern GPUs in which several SMs are grouped together to share a network port. Because of network port sharing, clustered GPUs face severe NoC congestion, which creates a critical performance bottleneck. In this paper, we target redundant network traffic to mitigate GPU NoC congestion. In particular, we observe that in many GPU-compute applications, different SMs in a cluster access shared data. Sending redundant requests to access the same memory location wastes valuable NoC bandwidth-we find on average 19 percent (and up to 48 percent) of the requests to be redundant. To remove redundant NoC traffic, we propose distributed-block scheduling, intra-cluster coalescing (ICC) and the coalesced cache (CC) to coalesce L1 cache misses within and across SMs in a cluster, respectively. Our evaluation results show that distributed-block scheduling, ICC and CC are complementary and improve both performance and energy consumption. We report an average performance improvement of 15 percent (and up to 67 percent) while at the same time reducing system energy by 6 percent (and up to 19 percent) and improving the energy-delay product (EDP) by 19 percent on average (and up to 53 percent), compared to state-of-the-art distributed CTA scheduling

Acceleration of CNN Computation on a PIM-enabled GPU system

학위논문(박사) -- 서울대학교대학원 : 공과대학 전기·정보공학부, 2022. 8. 이혁재.최근, convolutional neural network (CNN)은 image processing 및 computer vision 등에서 널리 사용되고 있다. CNN은 연산 집약적인 convolutional layer와 메모리 집약적인 fully connected layer, batch normalization layer 및 activation layer 등 다양한 layer로 구성된다. 일반적으로 CNN을 처리하기 위해 GPU가 널리 사용되지만, CNN은 연산 집약적인 동시에 메모리 집약적이기에 성능이 제한된다. 또한, 고화질의 image 및 video application의 사용은 GPU와 메모리 간의 data 이동에 의한 부담을 증가시킨다. Processing-in-memory는 메모리에 연산기를 탑재하여 data 이동에 의한 부담을 줄일 수 있어, host GPU와 PIM을 함께 사용하는 system은 CNN을 처리하기에 적합하다. 먼저 convolutional layer의 연산량을 감소시키기 위해, 근사 연산을 수행할 수 있다. 그러나 기존의 근사 연산은 host GPU로 data를 load 한 후 data 간 유사도를 파악하기에, GPU와 DRAM 간의 data 이동량을 줄이지는 못한다. 이는 메모리 intensity를 증가시켜 메모리 bottleneck을 유발한다. 게다가, 근사 연산으로 인해 warp 간 load imbalance 또한 발생하게 되어 성능이 저하된다. 이를 해결하기 위해, 본 논문에서는 data 간 근사 비교를 PIM에서 수행하는 방법을 제안한다. 제안하는 방법은 PIM에서 data간 유사도를 파악한 후, 대표 data와 유사도 정보만을 GPU로 전송한다. GPU는 대표 data에 대해서만 연산을 수행하고, 유사도 정보에 기반하여 해당 결과를 재사용하여 최종 결과를 생성한다. 이때, 메모리에서의 data 비교로 인한 latency 증가를 방지하기 위해 DRAM의 bank 단과 TSV 단을 모두 활용하는 2-level PIM 구조를 제안한다. 또한, 대표 data를 적당한 address에 재배치한 후 GPU로 전송하여 GPU에서의 별도 작업 없이 load balancing을 가능하게 한다. 다음으로, batch normalization 등 non-convolutional layer의 높은 메모리 사용량으로 인한 메모리 bottleneck 문제를 해결하기 위해 PIM에서 non-convolutional layer를 수행할 수 있다. 기존 연구에서는 PIM으로 non-convolutional layer를 가속하였지만, 단순히 GPU와 PIM이 순차적으로 동작하는 상황을 가정하여 성능 향상에 한계가 있었다. 제안하는 방법은 non-convolutional layer가 ouptut feature map의 channel 단위로 수행된다는 점에 착안하여 host와 PIM을 pipeline적으로 수행함으로써 CNN 학습을 가속한다. PIM은 host에서 convolution 연산이 끝난 output feature map의 channel에 대해 non-convolution 연산을 수행한다. 역전파 과정에서 발생하는 weight update와 feature map gradient 계산에서의 convolution과 non-convolution 간 job 균형을 위해, 적절하게 non-convolution job을 분배하여 성능을 향상시킨다. 이에 더해, host와 PIM이 동시에 memory에 access하는 상황에서 전체 수행 시간을 최소화하기 위해 bank 소유권 기반의 host와 PIM 간 memory scheduling 알고리즘을 제안한다. 마지막으로, image processing application 처리를 위해 logic die에 탑재 가능한 PIM GPU 구조를 제안한다. GPU 기반의 PIM은 CUDA 기반의 application을 수행할 수 있어 딥러닝 및 image application의 처리에 적합하지만, GPU의 큰 용량의 on-chip SRAM은 logic die에 충분한 수의 computing unit의 탑재를 어렵게 한다. 본 논문에서는 PIM에 적합한 최적의 lightweight GPU 구조와 함께 이를 활용하기 위한 최적화 기법을 제안한다. Image processing application의 메모리 접근 패턴과 data locality가 보존되도록 각 computing unit에 data를 할당하고, 예측 가능한 data의 할당을 기반으로 prefetcher를 탑재하여 lightweight한 구조임에도 충분한 수의 computing unit을 탑재하여 높은 성능을 확보한다.Recently, convolutional neural networks (CNN) have been widely used in image processing and computer vision. CNNs are composed of various layers such as computation-intensive convolutional layer and memory-intensive fully connected layer, batch normalization layer, and activation layer. GPUs are often used to accelerate the CNN, but performance is limited by high computational costs and memory usage of the convolution. Also, increasing demand for high resolution image applications increases the burden of data movement between GPU and memory. By performing computations on the memory, processing-in-memory (PIM) is expected to mitigate the overhead caused by data transfer. Therefore, a system that uses a PIM is promising for processing CNNs. First, prior studies exploited approximate computing to reduce the computational costs. However, they only reduced the amount of the computation, thereby its performance is bottlenecked by the memory bandwidth due to an increased memory intensity. In addition, load imbalance between warps caused by approximation also inhibits the performance improvement. This dissertation proposes a PIM solution that reduces the amount of data movement and computation through the Approximate Data Comparison (ADC-PIM). Instead of determining the value similarity on the GPU, the ADC-PIM located on memory compares the similarity and transfers only the selected data to the GPU. The GPU performs convolution on the representative data transferred from the ADC-PIM, and reuses the calculated results based on the similarity information. To reduce the increase in memory latency due to the data comparison, a two-level PIM architecture that exploits both the DRAM bank and TSV stage is proposed. To ease the load balancing on the GPU, the ADC-PIM reorganizes data by assigning the representative data to proposer addresses that are computed based on the comparison result. Second, to solve the memory bottleneck caused by the high memory usage, non-convolutional layers are accelerated with PIM. Previous studies also accelerated the non-convolutional layers by PIM, but there was a limit to performance improvement because they simply assumed a situation in which the GPU and PIM operate sequentially. The proposed method accelerates the CNN training with a pipelined execution of GPU and PIM, focusing on the fact that the non-convolution operation is performed in units of channels of the output feature map. PIM performs non-convolutional operations on the output feature map where the GPU has completed the convolution operation. To balance the jobs between convolution and non-convolution in weight update and feature map gradient calculation that occur in the back propagation process, non-convolution job is properly distributed to each process. In addition, a memory scheduling algorithm based on bank ownership between the host and PIM is proposed to minimize the overall execution time in a situation where the host and PIM simultaneously access memory. Finally, a GPU-based PIM architecture for image processing application is proposed. Programmable GPU-based PIM is attractive because it enables the utilization of well-crafted software development kits (SDKs) such as CUDA and openCL. However, the large capacity of on-chip SRAM of GPU makes it difficult to mount a sufficient number of computing units in logic die. This dissertation proposes a GPU-based PIM architecture and well-matched optimization strategies considering both the characteristics of image applications and logic die constraints. Data allocation to the computing unit is addressed to maintain the data locality and data access pattern. By applying a prefetcher that leverages the pattern-aware data allocation, the number of active warps and the on-chip SRAM size of the PIM are significantly reduced. This enables the logic die constraints to be satisfied and a greater number of computing units to be integrated on a logic die.제 1 장 서 론 1 1.1 연구의 배경 1 1.2 연구의 내용 3 1.3 논문 구성 4 제 2 장 연구의 배경 지식 5 2.1 High Bandwidth Memory 5 2.2 Processing-In-Memory 6 2.3 GPU의 구조 및 동작 모델 7 제 3 장 PIM을 활용한 근사적 데이터 비교 및 근사 연산을 통한 Convolution 가속 9 3.1 관련 연구 10 3.1.1 CNN에서의 Approximate Computing 10 3.1.2 Processing In Memory를 활용한 CNN 가속 11 3.2 Motivation 13 3.2.1 GPU에서 Convolution 연산 시의 Approximation 기회 13 3.2.2 Approxiamte Convolution 연산에서 발생하는 문제점 14 3.3 제안하는 ADC-PIM Design 18 3.3.1 Overview 18 3.3.2 Data 간 유사도 비교 방법 19 3.3.3 ADC-PIM 아키텍처 21 3.3.4 Load Balancing을 위한 Data Reorganization 27 3.4 GPU에서의 Approximate Convolution 31 3.4.1 Instruction Skip을 통한 Approximate Convolution 31 3.4.2 Approximate Convolution을 위한 구조적 지원 32 3.5 실험 결과 및 분석 36 3.5.1 실험 환경 구성 36 3.5.2 제안하는 각 방법의 영향 분석 38 3.5.3 기존 연구와의 성능 비교 41 3.5.4 에너지 소모량 비교 44 3.5.5 Design Overhead 분석 44 3.5.6 정확도에 미치는 영향 46 3.6 본 장의 결론 47 제 4 장 Convolutional layer와 non-Convolutional Layer의 Pipeline 실행을 통한 CNN 학습 가속 48 4.1 관련 연구 48 4.1.1 Non-CONV Lasyer의 Memory Bottleneck 완화 48 4.1.2 Host와 PIM 간 Memory Scheduling 49 4.2 Motivation 51 4.2.1 CONV와 non-CONV의 동시 수행 시 성능 향상 기회 51 4.2.2 PIM 우선도에 따른 host 및 PIM request의 처리 효율성 변화 52 4.3 제안하는 host-PIM Memory Scheduling 알고리즘 53 4.3.1 host-PIM System Overview 53 4.3.2 PIM Duration Based Memory Scheduling 53 4.3.3 최적 PD_TH 값의 계산 방법 56 4.4 제안하는 CNN 학습 동작 Flow 62 4.4.1 CNN 학습 순전파 과정 62 4.4.2 CNN 학습 역전파 과정 63 4.5 실험 결과 및 분석 67 4.5.1 실험 환경 구성 67 4.5.2 Layer 당 수행 시간 변화 68 4.5.3 역전파 과정에서의 non-CONV job 배분 효과 70 4.5.4 전체 Network Level에서의 수행 시간 변화 72 4.5.5 제안하는 최적 PD_TH 추정 방법의 정확도 및 선택 알고리즘의 수렴 속도 74 4.6 본 장의 결론 75 제 5 장 Image processing의 데이터 접근 패턴을 활용한 PIM에 적합한 lightweight GPU 구조 76 5.1 관련 연구 77 5.1.1 Processing In Memory 77 5.1.2 GPU에서의 CTA Scheduling 78 5.1.3 GPU에서의 Prefetching 78 5.2 Motivation 79 5.2.1 PIM GPU system에서 Image Processing 알고리즘 처리 시 기존 GPU 구조의 비효율성 79 5.3 제안하는 GPU 기반 PIM System 82 5.3.1 Overview 82 5.3.2 Access Pattern을 고려한 CTA 할당 83 5.3.3 PIM GPU 구조 90 5.4 실험 결과 및 분석 94 5.4.1 실험 환경 구성 94 5.4.2 In-Depth Analysis 95 5.4.3 기존 연구와의 성능 비교 98 5.4.4 Cache Miss Rate 및 Memory Traffic 102 5.4.5 에너지 소모량 비교 103 5.4.6 PIM의 면적 및 전력 소모량 분석 105 5.5 본 장의 결론 107 제 6 장 결론 108 참고문헌 110 Abstract 118박

Locality-aware CTA Clustering for modern GPUs

\u3cp\u3eCache is designed to exploit locality; however, the role of onchip L1 data caches on modern GPUs is often awkward. The locality among global memory requests from different SMs (Streaming Multiprocessors) is predominantly harvested by the commonly-shared L2 with long access latency; while the in-core locality, which is crucial for performance delivery, is handled explicitly by user-controlled scratchpad memory. In this work, we disclose another type of data locality that has been long ignored but with performance boosting potential - the inter-CTA locality. Exploiting such locality is rather challenging due to unclear hardware feasibility, unknown and inaccessible underlying CTA scheduler, and small incore cache capacity. To address these issues, we first conduct a thorough empirical exploration on various modern GPUs and demonstrate that inter-CTA locality can be harvested, both spatially and temporally, on L1 or L1/Tex unified cache. Through further quantification process, we prove the significance and commonality of such locality among GPU applications, and discuss whether such reuse is exploitable. By leveraging these insights, we propose the concept of CTAClustering and its associated software-based techniques to reshape the default CTA scheduling in order to group the CTAs with potential reuse together on the same SM. Our techniques require no hardware modification and can be directly deployed on existing GPUs. In addition, we incorporate these techniques into an integrated framework for automatic inter-CTA locality optimization.We evaluate our techniques using a wide range of popular GPU applications on all modern generations of NVIDIA GPU architectures. The results show that our proposed techniques significantly improve cache performance through reducing L2 cache transactions by 55%, 65%, 29%, 28% on average for Fermi, Kepler, Maxwell and Pascal, respectively, leading to an average of 1.46x, 1.48x, 1.45x, 1.41x (up to 3.8x, 3.6x, 3.1x, 3.3x) performance speedups for applications with algorithmrelated inter-CTA reuse.\u3c/p\u3

Fault Tolerant and Energy Efficient One-Sided Matrix Decompositions on Heterogeneous Systems with GPUs

Heterogeneous computing system with both CPUs and GPUs has become a class of widely used hardware architecture in supercomputers. As heterogeneous systems delivering higher computational performance, they are being built with an increasing number of complex components. This is anticipated that these systems will be more susceptible to hardware faults with higher power consumption. Numerical linear algebra libraries are used in a wide spectrum of high-performance scientific applications. Among numerical linear algebra operations, one-sided matrix decompositions can sometimes take a large portion of execution time or even dominate the whole scientific application execution. Due to the computational characteristic of one-sided matrix decompositions, they are very suitable for computation platforms such as heterogeneous systems with CPUs and GPUs. Many works have been done to implement and optimize one-sided matrix decompositions on heterogeneous systems with CPUs and GPUs. However, it is challenging to enable stable and high performance one-sided matrix decompositions running on computing platforms that are unreliable and high energy consumption. So, in this thesis, we aim to develop novel fault tolerance and energy efficiency optimizations for one-sided matrix decompositions on heterogeneous systems with CPUs and GPUs.To improve reliability and energy efficiency, extensive researches have been done on developing and optimizing fault tolerance methods and energy-saving strategies for one-sided matrix decompositions. However, current designs still have several limitations: (1) Little has been done on developing and optimizing fault tolerance method for one-sided matrix decompositions on heterogeneous systems with GPUs; (2) Limited by the protection coverage and strength, existing fault tolerance works provide insufficient protection when applied to one-sided matrix decompositions on heterogeneous systems with GPUs; (3) Lack the knowledge of algorithms, existing system level energy saving solutions cannot achieve the optimal energy savings due to potentially inaccurate and high-cost workload prediction they rely on when they are used in one-sided matrix decompositions; (4) It is challenging to apply both fault tolerance techniques and energy saving strategies to one-side matrix decompositions at the same time given that their current designs are not naturally compatible with each other.To address the first problem, based on the original (Algorithm Based Fault Tolerance) ABFT, we develop the first ABFT for matrix decomposition on heterogeneous systems with GPUs together with the novel storage errors protection and several optimization techniques specifically for GPUs. As for the second problem, we design a novel checksum scheme for ABFT that allows data stored in matrices to be encoded in two dimensions. This stronger checksum encoding mechanism enables much stronger protection including enhanced error propagation protection. In addition, we introduce a more efficient checking scheme. By prioritizing the checksum verification according to the sensitivity of matrix operations to soft errors with optimized checksum verification kernel for GPUs, we can achieve strong protect to matrix decompositions with comparable overhead. For the third problem, to improve energy efficiency for one-sided matrix decompositions, we introduce an algorithm-based energy-saving approach designed to maximize energy savings by utilizing algorithmic characteristics. Our approach can predict program execution behavior much more accurately, which is difficult for system level solutions for applications with variable execution characteristics. Experiments show that our approach can lead to much higher energy saving than existing works. Finally, for the fourth problem, we propose a novel energy saving approach for one-sided matrix decompositions on heterogeneous systems with GPUs. It allows energy saving strategies and fault tolerance techniques to be enabled at the same time without brings performance impact or extra energy cost