760 research outputs found
CUDA Unified Memory๋ฅผ ์ํ ๋ฐ์ดํฐ ๊ด๋ฆฌ ๋ฐ ํ๋ฆฌํ์นญ ๊ธฐ๋ฒ
ํ์๋
ผ๋ฌธ(๋ฐ์ฌ) -- ์์ธ๋ํ๊ต๋ํ์ : ๊ณต๊ณผ๋ํ ์ปดํจํฐ๊ณตํ๋ถ, 2022. 8. ์ด์ฌ์ง.Unified Memory (UM) is a component of CUDA programming model which provides a memory pool that has a single address space and can be accessed by both the host and the GPU. When UM is used, a CUDA program does not need to explicitly move data between the host and the device. It also allows GPU memory oversubscription by using CPU memory as a backing store. UM significantly lessens the burden of a programmer and provides great programmability. However, using UM solely does not guarantee good performance. To fully exploit UM and improve performance, the programmer needs to add user hints to the source code to prefetch pages that are going to be accessed during the CUDA kernel execution.
In this thesis, we propose three frameworks that exploits UM to improve the ease-of-programming while maximizing the application performance. The first framework is HUM, which hides host-to-device memory copy time of traditional CUDA program without any code modification. It overlaps the host-to-device memory copy with host computation or CUDA kernel computation by exploiting Unified Memory and fault mechanisms. The evaluation result shows that executing the applications under HUM is, on average, 1.21 times faster than executing them under original CUDA. The speedup is comparable to the average speedup 1.22 of the hand-optimized implementations for Unified Memory.
The second framework is DeepUM which exploits UM to allow GPU memory oversubscription for deep neural networks. While UM allows memory oversubscription using a page fault mechanism, page fault handling introduces enormous overhead. We use a correlation prefetching technique to solve the problem and hide the overhead. The evaluation result shows that DeepUM achieves comparable performance to the other state-of-the-art approaches. At the same time, our framework can run larger batch size that other methods fail to run.
The last framework is SnuRHAC that provides an illusion of a single GPU for the multiple GPUs in a cluster. Under SnuRHAC, a CUDA program designed to use a single GPU can utilize multiple GPUs in a cluster without any source code modification. SnuRHAC automatically distributes workload to multiple GPUs in a cluster and manages data across the nodes. To manage data efficiently, SnuRHAC extends Unified Memory and exploits its page fault mechanism. We also propose two prefetching techniques to fully exploit UM and to maximize performance. The evaluation result shows that while SnuRHAC significantly improves ease-of-programming, it shows scalable performance for the cluster environment depending on the application characteristics.Unified Memory (UM)๋ CUDA ํ๋ก๊ทธ๋๋ฐ ๋ชจ๋ธ์์ ์ ๊ณตํ๋ ๊ธฐ๋ฅ ์ค ํ๋๋ก ๋จ์ผ ๋ฉ๋ชจ๋ฆฌ ์ฃผ์ ๊ณต๊ฐ์ CPU์ GPU๊ฐ ๋์์ ์ ๊ทผํ ์ ์๋๋ก ํด์ค๋ค. ์ด์ ๋ฐ๋ผ, UM์ ์ฌ์ฉํ ๊ฒฝ์ฐ CUDA ํ๋ก๊ทธ๋จ์์ ๋ช
์์ ์ผ๋ก ํ๋ก์ธ์๊ฐ์ ๋ฐ์ดํฐ๋ฅผ ์ด๋์์ผ์ฃผ์ง ์์๋ ๋๋ค. ๋ํ, CPU ๋ฉ๋ชจ๋ฆฌ๋ฅผ backing store๋ก ์ฌ์ฉํ์ฌ GPU์ ๋ฉ๋ชจ๋ฆฌ ํฌ๊ธฐ ๋ณด๋ค ๋ ๋ง์ ์์ ๋ฉ๋ชจ๋ฆฌ๋ฅผ ํ์๋ก ํ๋ ํ๋ก๊ทธ๋จ์ ์คํํ ์ ์๋๋ก ํด์ค๋ค. ๊ฒฐ๊ณผ์ ์ผ๋ก, UM์ ํ๋ก๊ทธ๋๋จธ์ ๋ถ๋ด์ ํฌ๊ฒ ๋์ด์ฃผ๊ณ ์ฝ๊ฒ ํ๋ก๊ทธ๋๋ฐ ํ ์ ์๋๋ก ๋์์ค๋ค. ํ์ง๋ง, UM์ ์๋ ๊ทธ๋๋ก ์ฌ์ฉํ๋ ๊ฒ์ ์ฑ๋ฅ ์ธก๋ฉด์์ ์ข์ง ์๋ค. UM์ page fault mechanism์ ํตํด ๋์ํ๋๋ฐ page fault๋ฅผ ์ฒ๋ฆฌํ๊ธฐ ์ํด์๋ ๋ง์ ์ค๋ฒํค๋๊ฐ ๋ฐ์ํ๊ธฐ ๋๋ฌธ์ด๋ค. UM์ ์ฌ์ฉํ๋ฉด์ ์ต๋์ ์ฑ๋ฅ์ ์ป๊ธฐ ์ํด์๋ ํ๋ก๊ทธ๋๋จธ๊ฐ ์์ค ์ฝ๋์ ์ฌ๋ฌ ํํธ๋ ์์ผ๋ก CUDA ์ปค๋์์ ์ฌ์ฉ๋ ๋ฉ๋ชจ๋ฆฌ ์์ญ์ ๋ํ ํ๋ฆฌํ์น ๋ช
๋ น์ ์ฝ์
ํด์ฃผ์ด์ผ ํ๋ค.
๋ณธ ๋
ผ๋ฌธ์ UM์ ์ฌ์ฉํ๋ฉด์๋ ์ฌ์ด ํ๋ก๊ทธ๋๋ฐ๊ณผ ์ต๋์ ์ฑ๋ฅ์ด๋ผ๋ ๋๋ง๋ฆฌ ํ ๋ผ๋ฅผ ๋์์ ์ก๊ธฐ ์ํ ๋ฐฉ๋ฒ๋ค์ ์๊ฐํ๋ค. ์ฒซ์งธ๋ก, HUM์ ๊ธฐ์กด CUDA ํ๋ก๊ทธ๋จ์ ์์ค ์ฝ๋๋ฅผ ์์ ํ์ง ์๊ณ ํธ์คํธ์ ๋๋ฐ์ด์ค ๊ฐ์ ๋ฉ๋ชจ๋ฆฌ ์ ์ก ์๊ฐ์ ์ต์ํํ๋ค. ์ด๋ฅผ ์ํด, UM๊ณผ fault mechanism์ ์ฌ์ฉํ์ฌ ํธ์คํธ-๋๋ฐ์ด์ค ๊ฐ ๋ฉ๋ชจ๋ฆฌ ์ ์ก์ ํธ์คํธ ๊ณ์ฐ ํน์ CUDA ์ปค๋ ์คํ๊ณผ ์ค์ฒฉ์ํจ๋ค. ์คํ ๊ฒฐ๊ณผ๋ฅผ ํตํด HUM์ ํตํด ์ ํ๋ฆฌ์ผ์ด์
์ ์คํํ๋ ๊ฒ์ด ๊ทธ๋ ์ง ์๊ณ CUDA๋ง์ ์ฌ์ฉํ๋ ๊ฒ์ ๋นํด ํ๊ท 1.21๋ฐฐ ๋น ๋ฅธ ๊ฒ์ ํ์ธํ์๋ค. ๋ํ, Unified Memory๋ฅผ ๊ธฐ๋ฐ์ผ๋ก ํ๋ก๊ทธ๋๋จธ๊ฐ ์์ค ์ฝ๋๋ฅผ ์ต์ ํํ ๊ฒ๊ณผ ์ ์ฌํ ์ฑ๋ฅ์ ๋ด๋ ๊ฒ์ ํ์ธํ์๋ค.
๋๋ฒ์งธ๋ก, DeepUM์ UM์ ํ์ฉํ์ฌ GPU์ ๋ฉ๋ชจ๋ฆฌ ํฌ๊ธฐ ๋ณด๋ค ๋ ๋ง์ ์์ ๋ฉ๋ชจ๋ฆฌ๋ฅผ ํ์๋ก ํ๋ ๋ฅ ๋ฌ๋ ๋ชจ๋ธ์ ์คํํ ์ ์๊ฒ ํ๋ค. UM์ ํตํด GPU ๋ฉ๋ชจ๋ฆฌ๋ฅผ ์ด๊ณผํด์ ์ฌ์ฉํ ๊ฒฝ์ฐ CPU์ GPU๊ฐ์ ํ์ด์ง๊ฐ ๋งค์ฐ ๋น๋ฒํ๊ฒ ์ด๋ํ๋๋ฐ, ์ด๋ ๋ง์ ์ค๋ฒํค๋๊ฐ ๋ฐ์ํ๋ค. ๋๋ฒ์งธ ๋ฐฉ๋ฒ์์๋ correlation ํ๋ฆฌํ์นญ ๊ธฐ๋ฒ์ ํตํด ์ด ์ค๋ฒํค๋๋ฅผ ์ต์ํํ๋ค. ์คํ ๊ฒฐ๊ณผ๋ฅผ ํตํด DeepUM์ ๊ธฐ์กด์ ์ฐ๊ตฌ๋ ๊ฒฐ๊ณผ๋ค๊ณผ ๋น์ทํ ์ฑ๋ฅ์ ๋ณด์ด๋ฉด์ ๋ ํฐ ๋ฐฐ์น ์ฌ์ด์ฆ ํน์ ๋ ํฐ ํ์ดํผํ๋ผ๋ฏธํฐ๋ฅผ ์ฌ์ฉํ๋ ๋ชจ๋ธ์ ์คํํ ์ ์์์ ํ์ธํ์๋ค.
๋ง์ง๋ง์ผ๋ก, SnuRHAC์ ํด๋ฌ์คํฐ์ ์ฅ์ฐฉ๋ ์ฌ๋ฌ GPU๋ฅผ ๋ง์น ํ๋์ ํตํฉ๋ GPU์ฒ๋ผ ๋ณด์ฌ์ค๋ค. ๋ฐ๋ผ์, ํ๋ก๊ทธ๋๋จธ๋ ์ฌ๋ฌ GPU๋ฅผ ๋์์ผ๋ก ํ๋ก๊ทธ๋๋ฐ ํ์ง ์๊ณ ํ๋์ ๊ฐ์ GPU๋ฅผ ๋์์ผ๋ก ํ๋ก๊ทธ๋๋ฐํ๋ฉด ํด๋ฌ์คํฐ์ ์ฅ์ฐฉ๋ ๋ชจ๋ GPU๋ฅผ ํ์ฉํ ์ ์๋ค. ์ด๋ SnuRHAC์ด Unified Memory๋ฅผ ํด๋ฌ์คํฐ ํ๊ฒฝ์์ ๋์ํ๋๋ก ํ์ฅํ๊ณ , ํ์ํ ๋ฐ์ดํฐ๋ฅผ ์๋์ผ๋ก GPU๊ฐ์ ์ ์กํ๊ณ ๊ด๋ฆฌํด์ฃผ๊ธฐ ๋๋ฌธ์ด๋ค. ๋ํ, UM์ ์ฌ์ฉํ๋ฉด์ ๋ฐ์ํ ์ ์๋ ์ค๋ฒํค๋๋ฅผ ์ต์ํํ๊ธฐ ์ํด ๋ค์ํ ํ๋ฆฌํ์นญ ๊ธฐ๋ฒ์ ์๊ฐํ๋ค. ์คํ ๊ฒฐ๊ณผ๋ฅผ ํตํด SnuRHAC์ด ์ฝ๊ฒ GPU ํด๋ฌ์คํฐ๋ฅผ ์ํ ํ๋ก๊ทธ๋๋ฐ์ ํ ์ ์๋๋ก ๋์์ค ๋ฟ๋ง ์๋๋ผ, ์ ํ๋ฆฌ์ผ์ด์
ํน์ฑ์ ๋ฐ๋ผ ์ต์ ์ ์ฑ๋ฅ์ ๋ผ ์ ์์์ ๋ณด์ธ๋ค.1 Introduction 1
2 Related Work 7
3 CUDA Unified Memory 12
4 Framework for Maximizing the Performance of Traditional CUDA Program 17
4.1 Overall Structure of HUM 17
4.2 Overlapping H2Dmemcpy and Computation 19
4.3 Data Consistency and Correctness 23
4.4 HUM Driver 25
4.5 HUM H2Dmemcpy Mechanism 26
4.6 Parallelizing Memory Copy Commands 29
4.7 Scheduling Memory Copy Commands 31
5 Framework for Running Large-scale DNNs on a Single GPU 33
5.1 Structure of DeepUM 33
5.1.1 DeepUM Runtime 34
5.1.2 DeepUM Driver 35
5.2 Correlation Prefetching for GPU Pages 36
5.2.1 Pair-based Correlation Prefetching 37
5.2.2 Correlation Prefetching in DeepUM 38
5.3 Optimizations for GPU Page Fault Handling 42
5.3.1 Page Pre-eviction 42
5.3.2 Invalidating UM Blocks of Inactive PyTorch Blocks 43
6 Framework for Virtualizing a Single Device Image for a GPU Cluster 45
6.1 Overall Structure of SnuRHAC 45
6.2 Workload Distribution 48
6.3 Cluster Unified Memory 50
6.4 Additional Optimizations 57
6.5 Prefetching 58
6.5.1 Static Prefetching 58
6.5.2 Dynamic Prefetching 61
7 Evaluation 62
7.1 Framework for Maximizing the Performance of Traditional CUDA Program 62
7.1.1 Methodology 63
7.1.2 Results 64
7.2 Framework for Running Large-scale DNNs on a Single GPU 70
7.2.1 Methodology 70
7.2.2 Comparison with Naive UM and IBM LMS 72
7.2.3 Parameters of the UM Block Correlation Table 78
7.2.4 Comparison with TensorFlow-based Approaches 79
7.3 Framework for Virtualizing Single Device Image for a GPU Cluster 81
7.3.1 Methodology 81
7.3.2 Results 84
8 Discussions and Future Work 91
9 Conclusion 93
์ด๋ก 111๋ฐ
Comprehensive characterization of an open source document search engine
This work performs a thorough characterization and analysis of the open source Lucene search library. The article describes in detail the architecture, functionality, and micro-architectural behavior of the search engine, and investigates prominent online document search research issues. In particular, we study how intra-server index partitioning affects the response time and throughput, explore the potential use of low power servers for document search, and examine the sources of performance degradation ands the causes of tail latencies. Some of our main conclusions are the following: (a) intra-server index partitioning can reduce tail latencies but with diminishing benefits as incoming query traffic increases, (b) low power servers given enough partitioning can provide same average and tail response times as conventional high performance servers, (c) index search is a CPU-intensive cache-friendly application, and (d) C-states are the main culprits for performance degradation in document search.Web of Science162art. no. 1
Holistic Performance Analysis and Optimization of Unified Virtual Memory
The programming difficulty of creating GPU-accelerated high performance computing (HPC) codes has been greatly reduced by the advent of Unified Memory technologies that abstract the management of physical memory away from the developer. However, these systems incur substantial overhead that paradoxically grows for codes where these technologies are most useful. While these technologies are increasingly adopted for use in modern HPC frameworks and applications, the performance cost reduces the efficiency of these systems and turns away some developers from adoption entirely. These systems are naturally difficult to optimize due to the large number of interconnected hardware and software components that must be untangled to perform thorough analysis.
In this thesis, we take the first deep dive into a functional implementation of a Unified Memory system, NVIDIA UVM, to evaluate the performance and characteristics of these systems. We show specific hardware and software interactions that cause serialization between host and devices. We further provide a quantitative evaluation of fault handling for various applications under different scenarios, including prefetching and oversubscription. Through lower-level analysis, we find that the driver workload is dependent on the interactions among application access patterns, GPU hardware constraints, and Host OS components. These findings indicate that the cost of host OS components is significant and present across UM implementations. We also provide a proof-of-concept asynchronous approach to memory management in UVM that allows for reduced system overhead and improved application performance. This study provides constructive insight into future implementations and systems, such as Heterogeneous Memory Management
IMP: Indirect Memory Prefetcher
Machine learning, graph analytics and sparse linear algebra-based applications are dominated by irregular memory accesses resulting from following edges in a graph or non-zero elements in a sparse matrix. These accesses have little temporal or spatial locality, and thus incur long memory stalls and large bandwidth requirements. A traditional streaming or striding prefetcher cannot capture these irregular access patterns.
A majority of these irregular accesses come from indirect patterns of the form A[B[i]]. We propose an efficient hardware indirect memory prefetcher (IMP) to capture this access pattern and hide latency. We also propose a partial cacheline accessing mechanism for these prefetches to reduce the network and DRAM bandwidth pressure from the lack of spatial locality.
Evaluated on 7 applications, IMP shows 56% speedup on average (up to 2.3ร) compared to a baseline 64 core system with streaming prefetchers. This is within 23% of an idealized system. With partial cacheline accessing, we see another 9.4% speedup on average (up to 46.6%).Intel Science and Technology Center for Big Dat
TLB pre-loading for Java applications
The increasing memory requirement for today\u27s applications is causing more stress for the memory system. This side effect puts pressure into available caches, and specifically the TLB cache. TLB misses are responsible for a considerable ratio of the total memory latency, since an average of 10% of execution time is wasted on miss penalties. Java applications are not in a better position. Their attractive features increase the memory footprint. Generally, Java applications TLB miss rate tends to be multiples of miss rate for non-java applications. The high miss rate will cause the application to loose valuable execution time. Our experiments show that on average, miss penalty can constitute about 24% of execution time. Several hardware modifications were suggested to reduce TLB misses for general applications. However, to the best of our knowledge, there have been no similar efforts for Java applications. Here we propose a software-based prediction model that relies on information available to the virtual machine. The model uses the write barrier operation to predict TLB misses with an average 41% accuracy rate
Recommended from our members
Accurate modeling of core and memory locality for proxy generation targeting emerging applications and architectures
Designing optimal computer systems for improved performance and energy efficiency requires architects and designers to have a deep understanding of the end-user workloads. However, many end-users (e.g., large corporations, banks, defense organizations, etc.) are apprehensive to share their applications with designers due to the confidential nature of software code and data. In addition, emerging applications pose significant challenges to early design space exploration due to their long-running nature and the highly complex nature of their software stack that cannot be supported on many early performance models.
The above challenges can be overcome by using a proxy benchmark. A miniaturized proxy benchmark can be used as a substitute of the original workload to perform early computer performance evaluation. The process of generating a proxy benchmark consists of extracting a set of key statistics to summarize the behavior of end-user applications through profiling and using the collected statistics to synthesize a representative proxy benchmark. Using such proxy benchmarks can help designers to understand the behavior of end-userโs workloads in a reasonable time without the users having to disclose sensitive information about their workloads.
Prior proxy benchmarking schemes leverage micro-architecture independent metrics, derived from detailed simulation tools, to generate proxy benchmarks. However, many emerging workloads do not work reliably with many profiling or simulation tools, in which case it becomes impossible to apply prior proxy generation techniques to generate proxy benchmarks for such complex applications. Furthermore, these techniques model instruction pipeline-level locality in great detail, but abstract out memory locality modeling using simple stride-based models. This results in poor cloning accuracy especially for emerging applications, which have larger memory footprints and complex access patterns. A few detailed cache and memory locality modeling techniques have also been proposed in literature. However, these techniques either model limited locality metrics and suffer from poor cloning accuracy or are fairly accurate, but at the expense of significant metadata overhead. Finally, none of the prior proxy benchmarking techniques model both core and memory locality with high accuracy. As a result, they are not useful for studying system-level performance behavior. Keeping the above key limitations and shortcomings of prior work in mind, this dissertation presents several techniques that expand the frontiers of workload proxy benchmarking, thereby enabling computer designers to gain a better and faster understanding of end-user application behavior without compromising the privileged nature of software or data.
This dissertation first presents a core-level proxy benchmark generation methodology that leverages performance metrics derived from hardware performance counter measurements to create miniature proxy benchmarks targeting emerging big-data applications. The presented performance counter based characterization and associated extrapolation into generic parameters for proxy generation enables faster analysis (runs almost at native hardware speeds, unlike prior workload cloning proposals) and proxy generation for emerging applications that do not work with simulators or profiling tools. The generated proxy benchmarks are representative of the performance of the real-world big-data applications, including operating system and run-time effects, and yet converge to results quickly without needing any complex software stack support.
Next, to improve upon the accuracy and efficiency of prior memory proxy benchmarking techniques, this dissertation presents a novel memory locality modeling technique that leverages localized pattern detection to create miniature memory proxy benchmarks. The presented technique models memory reference locality by decomposing an applicationโs memory accesses into a set of independent streams (localized by using address region based localization property), tracking fine-grained patterns within the localized streams and, finally, chaining or interleaving accesses from different localized memory streams to create an ordered proxy memory access sequence. This dissertation further extends the workload cloning approach to Graphics Processing Units (GPUs) and presents a novel proxy generation methodology to model the inherent memory access locality of GPU applications, while also accounting for the GPUโs parallel execution model. The generated memory proxy benchmarks help to enable fast and efficient design space exploration of futuristic memory hierarchies.
Finally, this dissertation presents a novel technique to integrate accurate core and memory locality models to create system-level proxy benchmarks targeting emerging applications. This is a new capability that can facilitate efficient overall system (core, cache and memory subsystem) design-space exploration. This dissertation further presents a novel methodology that exploits the synthetic benchmark generation framework to create hypothetical workloads with performance behavior that does not currently exist. Such proxies can be generated to cover anticipated code trends and can represent futuristic workloads before the workloads even exist.Electrical and Computer Engineerin
- โฆ