890 research outputs found
Recommended from our members
Preparing sparse solvers for exascale computing.
Sparse solvers provide essential functionality for a wide variety of scientific applications. Highly parallel sparse solvers are essential for continuing advances in high-fidelity, multi-physics and multi-scale simulations, especially as we target exascale platforms. This paper describes the challenges, strategies and progress of the US Department of Energy Exascale Computing project towards providing sparse solvers for exascale computing platforms. We address the demands of systems with thousands of high-performance node devices where exposing concurrency, hiding latency and creating alternative algorithms become essential. The efforts described here are works in progress, highlighting current success and upcoming challenges. This article is part of a discussion meeting issue 'Numerical algorithms for high-performance computational science'
TeAAL: A Declarative Framework for Modeling Sparse Tensor Accelerators
Over the past few years, the explosion in sparse tensor algebra workloads has
led to a corresponding rise in domain-specific accelerators to service them.
Due to the irregularity present in sparse tensors, these accelerators employ a
wide variety of novel solutions to achieve good performance. At the same time,
prior work on design-flexible sparse accelerator modeling does not express this
full range of design features, making it difficult to understand the impact of
each design choice and compare or extend the state-of-the-art.
To address this, we propose TeAAL: a language and compiler for the concise
and precise specification and evaluation of sparse tensor algebra
architectures. We use TeAAL to represent and evaluate four disparate
state-of-the-art accelerators--ExTensor, Gamma, OuterSPACE, and SIGMA--and
verify that it reproduces their performance with high accuracy. Finally, we
demonstrate the potential of TeAAL as a tool for designing new accelerators by
showing how it can be used to speed up Graphicionado--by on BFS and
on SSSP.Comment: 14 pages, 12 figure
Task-based Runtime Optimizations Towards High Performance Computing Applications
The last decades have witnessed a rapid improvement of computational capabilities in high-performance computing (HPC) platforms thanks to hardware technology scaling. HPC architectures benefit from mainstream advances on the hardware with many-core systems, deep hierarchical memory subsystem, non-uniform memory access, and an ever-increasing gap between computational power and memory bandwidth. This has necessitated continuous adaptations across the software stack to maintain high hardware utilization. In this HPC landscape of potentially million-way parallelism, task-based programming models associated with dynamic runtime systems are becoming more popular, which fosters developers’ productivity at extreme scale by abstracting the underlying hardware complexity.
In this context, this dissertation highlights how a software bundle powered by a task-based programming model can address the heterogeneous workloads engendered by HPC applications., i.e., data redistribution, geospatial modeling and 3D unstructured mesh deformation here. Data redistribution aims to reshuffle data to optimize some objective for an algorithm, whose objective can be multi-dimensional, such as improving computational load balance or decreasing communication volume or cost, with the ultimate goal of increasing the efficiency and therefore reducing the time-to-solution for the algorithm. Geostatistical modeling, one of the prime motivating applications for exascale computing, is a technique for predicting desired quantities from geographically distributed data, based on statistical models and optimization of parameters. Meshing the deformable contour of moving 3D bodies is an expensive operation that can cause huge computational challenges in fluid-structure interaction (FSI) applications. Therefore, in this dissertation, Redistribute-PaRSEC, ExaGeoStat-PaRSEC and HiCMA-PaRSEC are proposed to efficiently tackle these HPC applications respectively at extreme scale, and they are evaluated on multiple HPC clusters, including AMD-based, Intel-based, Arm-based CPU systems and IBM-based multi-GPU system. This multidisciplinary work emphasizes the need for runtime systems to go beyond their primary responsibility of task scheduling on massively parallel hardware system for servicing the next-generation scientific applications
Bridging Control-Centric and Data-Centric Optimization
With the rise of specialized hardware and new programming languages, code
optimization has shifted its focus towards promoting data locality. Most
production-grade compilers adopt a control-centric mindset - instruction-driven
optimization augmented with scalar-based dataflow - whereas other approaches
provide domain-specific and general purpose data movement minimization, which
can miss important control-flow optimizations. As the two representations are
not commutable, users must choose one over the other. In this paper, we explore
how both control- and data-centric approaches can work in tandem via the
Multi-Level Intermediate Representation (MLIR) framework. Through a combination
of an MLIR dialect and specialized passes, we recover parametric, symbolic
dataflow that can be optimized within the DaCe framework. We combine the two
views into a single pipeline, called DCIR, showing that it is strictly more
powerful than either view. On several benchmarks and a real-world application
in C, we show that our proposed pipeline consistently outperforms MLIR and
automatically uncovers new optimization opportunities with no additional
effort.Comment: CGO'2
Full Stack Optimization of Transformer Inference: a Survey
Recent advances in state-of-the-art DNN architecture design have been moving
toward Transformer models. These models achieve superior accuracy across a wide
range of applications. This trend has been consistent over the past several
years since Transformer models were originally introduced. However, the amount
of compute and bandwidth required for inference of recent Transformer models is
growing at a significant rate, and this has made their deployment in
latency-sensitive applications challenging. As such, there has been an
increased focus on making Transformer models more efficient, with methods that
range from changing the architecture design, all the way to developing
dedicated domain-specific accelerators. In this work, we survey different
approaches for efficient Transformer inference, including: (i) analysis and
profiling of the bottlenecks in existing Transformer architectures and their
similarities and differences with previous convolutional models; (ii)
implications of Transformer architecture on hardware, including the impact of
non-linear operations such as Layer Normalization, Softmax, and GELU, as well
as linear operations, on hardware design; (iii) approaches for optimizing a
fixed Transformer architecture; (iv) challenges in finding the right mapping
and scheduling of operations for Transformer models; and (v) approaches for
optimizing Transformer models by adapting the architecture using neural
architecture search. Finally, we perform a case study by applying the surveyed
optimizations on Gemmini, the open-source, full-stack DNN accelerator
generator, and we show how each of these approaches can yield improvements,
compared to previous benchmark results on Gemmini. Among other things, we find
that a full-stack co-design approach with the aforementioned methods can result
in up to 88.7x speedup with a minimal performance degradation for Transformer
inference
- …