12,369 research outputs found
Accelerating Large-Scale Data Analysis by Offloading to High-Performance Computing Libraries using Alchemist
Apache Spark is a popular system aimed at the analysis of large data sets,
but recent studies have shown that certain computations---in particular, many
linear algebra computations that are the basis for solving common machine
learning problems---are significantly slower in Spark than when done using
libraries written in a high-performance computing framework such as the
Message-Passing Interface (MPI).
To remedy this, we introduce Alchemist, a system designed to call MPI-based
libraries from Apache Spark. Using Alchemist with Spark helps accelerate linear
algebra, machine learning, and related computations, while still retaining the
benefits of working within the Spark environment. We discuss the motivation
behind the development of Alchemist, and we provide a brief overview of its
design and implementation.
We also compare the performances of pure Spark implementations with those of
Spark implementations that leverage MPI-based codes via Alchemist. To do so, we
use data science case studies: a large-scale application of the conjugate
gradient method to solve very large linear systems arising in a speech
classification problem, where we see an improvement of an order of magnitude;
and the truncated singular value decomposition (SVD) of a 400GB
three-dimensional ocean temperature data set, where we see a speedup of up to
7.9x. We also illustrate that the truncated SVD computation is easily scalable
to terabyte-sized data by applying it to data sets of sizes up to 17.6TB.Comment: Accepted for publication in Proceedings of the 24th ACM SIGKDD
International Conference on Knowledge Discovery and Data Mining, London, UK,
201
Abstract State Machines 1988-1998: Commented ASM Bibliography
An annotated bibliography of papers which deal with or use Abstract State
Machines (ASMs), as of January 1998.Comment: Also maintained as a BibTeX file at http://www.eecs.umich.edu/gasm
Structure Preserving Parallel Algorithms for Solving the Bethe-Salpeter Eigenvalue Problem
The Bethe-Salpeter eigenvalue problem is a dense structured eigenvalue
problem arising from discretized Bethe-Salpeter equation in the context of
computing exciton energies and states. A computational challenge is that at
least half of the eigenvalues and the associated eigenvectors are desired in
practice. We establish the equivalence between Bethe-Salpeter eigenvalue
problems and real Hamiltonian eigenvalue problems. Based on theoretical
analysis, structure preserving algorithms for a class of Bethe-Salpeter
eigenvalue problems are proposed. We also show that for this class of problems
all eigenvalues obtained from the Tamm-Dancoff approximation are overestimated.
In order to solve large scale problems of practical interest, we discuss
parallel implementations of our algorithms targeting distributed memory
systems. Several numerical examples are presented to demonstrate the efficiency
and accuracy of our algorithms
QuantumInformation.jl---a Julia package for numerical computation in quantum information theory
Numerical investigations are an important research tool in quantum
information theory. There already exists a wide range of computational tools
for quantum information theory implemented in various programming languages.
However, there is little effort in implementing this kind of tools in the Julia
language. Julia is a modern programming language designed for numerical
computation with excellent support for vector and matrix algebra, extended type
system that allows for implementation of elegant application interfaces and
support for parallel and distributed computing. QuantumInformation.jl is a new
quantum information theory library implemented in Julia that provides functions
for creating and analyzing quantum states, and for creating quantum operations
in various representations. An additional feature of the library is a
collection of functions for sampling random quantum states and operations such
as unitary operations and generic quantum channels.Comment: 32 pages, 8 figure
- …