13,841 research outputs found
Efficient Memory Management for GPU-based Deep Learning Systems
GPU (graphics processing unit) has been used for many data-intensive
applications. Among them, deep learning systems are one of the most important
consumer systems for GPU nowadays. As deep learning applications impose deeper
and larger models in order to achieve higher accuracy, memory management
becomes an important research topic for deep learning systems, given that GPU
has limited memory size. Many approaches have been proposed towards this issue,
e.g., model compression and memory swapping. However, they either degrade the
model accuracy or require a lot of manual intervention. In this paper, we
propose two orthogonal approaches to reduce the memory cost from the system
perspective. Our approaches are transparent to the models, and thus do not
affect the model accuracy. They are achieved by exploiting the iterative nature
of the training algorithm of deep learning to derive the lifetime and
read/write order of all variables. With the lifetime semantics, we are able to
implement a memory pool with minimal fragments. However, the optimization
problem is NP-complete. We propose a heuristic algorithm that reduces up to
13.3% of memory compared with Nvidia's default memory pool with equal time
complexity. With the read/write semantics, the variables that are not in use
can be swapped out from GPU to CPU to reduce the memory footprint. We propose
multiple swapping strategies to automatically decide which variable to swap and
when to swap out (in), which reduces the memory cost by up to 34.2% without
communication overhead
Efficient Memory Management for GPU-based Deep Learning Systems
GPU (graphics processing unit) has been used for many data-intensive
applications. Among them, deep learning systems are one of the most important
consumer systems for GPU nowadays. As deep learning applications impose deeper
and larger models in order to achieve higher accuracy, memory management
becomes an important research topic for deep learning systems, given that GPU
has limited memory size. Many approaches have been proposed towards this issue,
e.g., model compression and memory swapping. However, they either degrade the
model accuracy or require a lot of manual intervention. In this paper, we
propose two orthogonal approaches to reduce the memory cost from the system
perspective. Our approaches are transparent to the models, and thus do not
affect the model accuracy. They are achieved by exploiting the iterative nature
of the training algorithm of deep learning to derive the lifetime and
read/write order of all variables. With the lifetime semantics, we are able to
implement a memory pool with minimal fragments. However, the optimization
problem is NP-complete. We propose a heuristic algorithm that reduces up to
13.3% of memory compared with Nvidia's default memory pool with equal time
complexity. With the read/write semantics, the variables that are not in use
can be swapped out from GPU to CPU to reduce the memory footprint. We propose
multiple swapping strategies to automatically decide which variable to swap and
when to swap out (in), which reduces the memory cost by up to 34.2% without
communication overhead
vDNN: Virtualized Deep Neural Networks for Scalable, Memory-Efficient Neural Network Design
The most widely used machine learning frameworks require users to carefully
tune their memory usage so that the deep neural network (DNN) fits into the
DRAM capacity of a GPU. This restriction hampers a researcher's flexibility to
study different machine learning algorithms, forcing them to either use a less
desirable network architecture or parallelize the processing across multiple
GPUs. We propose a runtime memory manager that virtualizes the memory usage of
DNNs such that both GPU and CPU memory can simultaneously be utilized for
training larger DNNs. Our virtualized DNN (vDNN) reduces the average GPU memory
usage of AlexNet by up to 89%, OverFeat by 91%, and GoogLeNet by 95%, a
significant reduction in memory requirements of DNNs. Similar experiments on
VGG-16, one of the deepest and memory hungry DNNs to date, demonstrate the
memory-efficiency of our proposal. vDNN enables VGG-16 with batch size 256
(requiring 28 GB of memory) to be trained on a single NVIDIA Titan X GPU card
containing 12 GB of memory, with 18% performance loss compared to a
hypothetical, oracular GPU with enough memory to hold the entire DNN.Comment: Published as a conference paper at the 49th IEEE/ACM International
Symposium on Microarchitecture (MICRO-49), 201
SuperNeurons: Dynamic GPU Memory Management for Training Deep Neural Networks
Going deeper and wider in neural architectures improves the accuracy, while
the limited GPU DRAM places an undesired restriction on the network design
domain. Deep Learning (DL) practitioners either need change to less desired
network architectures, or nontrivially dissect a network across multiGPUs.
These distract DL practitioners from concentrating on their original machine
learning tasks. We present SuperNeurons: a dynamic GPU memory scheduling
runtime to enable the network training far beyond the GPU DRAM capacity.
SuperNeurons features 3 memory optimizations, \textit{Liveness Analysis},
\textit{Unified Tensor Pool}, and \textit{Cost-Aware Recomputation}, all
together they effectively reduce the network-wide peak memory usage down to the
maximal memory usage among layers. We also address the performance issues in
those memory saving techniques. Given the limited GPU DRAM, SuperNeurons not
only provisions the necessary memory for the training, but also dynamically
allocates the memory for convolution workspaces to achieve the high
performance. Evaluations against Caffe, Torch, MXNet and TensorFlow have
demonstrated that SuperNeurons trains at least 3.2432 deeper network than
current ones with the leading performance. Particularly, SuperNeurons can train
ResNet2500 that has basic network layers on a 12GB K40c.Comment: PPoPP '2018: 23nd ACM SIGPLAN Symposium on Principles and Practice of
Parallel Programmin
Scalable Distributed DNN Training using TensorFlow and CUDA-Aware MPI: Characterization, Designs, and Performance Evaluation
TensorFlow has been the most widely adopted Machine/Deep Learning framework.
However, little exists in the literature that provides a thorough understanding
of the capabilities which TensorFlow offers for the distributed training of
large ML/DL models that need computation and communication at scale. Most
commonly used distributed training approaches for TF can be categorized as
follows: 1) Google Remote Procedure Call (gRPC), 2) gRPC+X: X=(InfiniBand
Verbs, Message Passing Interface, and GPUDirect RDMA), and 3) No-gRPC: Baidu
Allreduce with MPI, Horovod with MPI, and Horovod with NVIDIA NCCL. In this
paper, we provide an in-depth performance characterization and analysis of
these distributed training approaches on various GPU clusters including the Piz
Daint system (6 on Top500). We perform experiments to gain novel insights along
the following vectors: 1) Application-level scalability of DNN training, 2)
Effect of Batch Size on scaling efficiency, 3) Impact of the MPI library used
for no-gRPC approaches, and 4) Type and size of DNN architectures. Based on
these experiments, we present two key insights: 1) Overall, No-gRPC designs
achieve better performance compared to gRPC-based approaches for most
configurations, and 2) The performance of No-gRPC is heavily influenced by the
gradient aggregation using Allreduce. Finally, we propose a truly CUDA-Aware
MPI Allreduce design that exploits CUDA kernels and pointer caching to perform
large reductions efficiently. Our proposed designs offer 5-17X better
performance than NCCL2 for small and medium messages, and reduces latency by
29% for large messages. The proposed optimizations help Horovod-MPI to achieve
approximately 90% scaling efficiency for ResNet-50 training on 64 GPUs.
Further, Horovod-MPI achieves 1.8X and 3.2X higher throughput than the native
gRPC method for ResNet-50 and MobileNet, respectively, on the Piz Daint
cluster.Comment: 10 pages, 9 figures, submitted to IEEE IPDPS 2019 for peer-revie
Cloud-based or On-device: An Empirical Study of Mobile Deep Inference
Modern mobile applications are benefiting significantly from the advancement
in deep learning, e.g., implementing real-time image recognition and
conversational system. Given a trained deep learning model, applications
usually need to perform a series of matrix operations based on the input data,
in order to infer possible output values. Because of computational complexity
and size constraints, these trained models are often hosted in the cloud. To
utilize these cloud-based models, mobile apps will have to send input data over
the network. While cloud-based deep learning can provide reasonable response
time for mobile apps, it restricts the use case scenarios, e.g. mobile apps
need to have network access. With mobile specific deep learning optimizations,
it is now possible to employ on-device inference. However, because mobile
hardware, such as GPU and memory size, can be very limited when compared to its
desktop counterpart, it is important to understand the feasibility of this new
on-device deep learning inference architecture. In this paper, we empirically
evaluate the inference performance of three Convolutional Neural Networks
(CNNs) using a benchmark Android application we developed. Our measurement and
analysis suggest that on-device inference can cost up to two orders of
magnitude greater response time and energy when compared to cloud-based
inference, and that loading model and computing probability are two performance
bottlenecks for on-device deep inferences.Comment: Accepted at The IEEE International Conference on Cloud Engineering
(IC2E) conference 201
- …