61,375 research outputs found
LINVIEW: Incremental View Maintenance for Complex Analytical Queries
Many analytics tasks and machine learning problems can be naturally expressed
by iterative linear algebra programs. In this paper, we study the incremental
view maintenance problem for such complex analytical queries. We develop a
framework, called LINVIEW, for capturing deltas of linear algebra programs and
understanding their computational cost. Linear algebra operations tend to cause
an avalanche effect where even very local changes to the input matrices spread
out and infect all of the intermediate results and the final view, causing
incremental view maintenance to lose its performance benefit over
re-evaluation. We develop techniques based on matrix factorizations to contain
such epidemics of change. As a consequence, our techniques make incremental
view maintenance of linear algebra practical and usually substantially cheaper
than re-evaluation. We show, both analytically and experimentally, the
usefulness of these techniques when applied to standard analytics tasks. Our
evaluation demonstrates the efficiency of LINVIEW in generating parallel
incremental programs that outperform re-evaluation techniques by more than an
order of magnitude.Comment: 14 pages, SIGMO
Declarative Ajax Web Applications through SQL++ on a Unified Application State
Implementing even a conceptually simple web application requires an
inordinate amount of time. FORWARD addresses three problems that reduce
developer productivity: (a) Impedance mismatch across the multiple languages
used at different tiers of the application architecture. (b) Distributed data
access across the multiple data sources of the application (SQL database, user
input of the browser page, session data in the application server, etc). (c)
Asynchronous, incremental modification of the pages, as performed by Ajax
actions.
FORWARD belongs to a novel family of web application frameworks that attack
impedance mismatch by offering a single unifying language. FORWARD's language
is SQL++, a minimally extended SQL. FORWARD's architecture is based on two
novel cornerstones: (a) A Unified Application State (UAS), which is a virtual
database over the multiple data sources. The UAS is accessed via distributed
SQL++ queries, therefore resolving the distributed data access problem. (b)
Declarative page specifications, which treat the data displayed by pages as
rendered SQL++ page queries. The resulting pages are automatically
incrementally modified by FORWARD. User input on the page becomes part of the
UAS.
We show that SQL++ captures the semi-structured nature of web pages and
subsumes the data models of two important data sources of the UAS: SQL
databases and JavaScript components. We show that simple markup is sufficient
for creating Ajax displays and for modeling user input on the page as UAS data
sources. Finally, we discuss the page specification syntax and semantics that
are needed in order to avoid race conditions and conflicts between the user
input and the automated Ajax page modifications.
FORWARD has been used in the development of eight commercial and academic
applications. An alpha-release web-based IDE (itself built in FORWARD) enables
development in the cloud.Comment: Proceedings of the 14th International Symposium on Database
Programming Languages (DBPL 2013), August 30, 2013, Riva del Garda, Trento,
Ital
Stochastic Database Cracking: Towards Robust Adaptive Indexing in Main-Memory Column-Stores
Modern business applications and scientific databases call for inherently
dynamic data storage environments. Such environments are characterized by two
challenging features: (a) they have little idle system time to devote on
physical design; and (b) there is little, if any, a priori workload knowledge,
while the query and data workload keeps changing dynamically. In such
environments, traditional approaches to index building and maintenance cannot
apply. Database cracking has been proposed as a solution that allows on-the-fly
physical data reorganization, as a collateral effect of query processing.
Cracking aims to continuously and automatically adapt indexes to the workload
at hand, without human intervention. Indexes are built incrementally,
adaptively, and on demand. Nevertheless, as we show, existing adaptive indexing
methods fail to deliver workload-robustness; they perform much better with
random workloads than with others. This frailty derives from the inelasticity
with which these approaches interpret each query as a hint on how data should
be stored. Current cracking schemes blindly reorganize the data within each
query's range, even if that results into successive expensive operations with
minimal indexing benefit. In this paper, we introduce stochastic cracking, a
significantly more resilient approach to adaptive indexing. Stochastic cracking
also uses each query as a hint on how to reorganize data, but not blindly so;
it gains resilience and avoids performance bottlenecks by deliberately applying
certain arbitrary choices in its decision-making. Thereby, we bring adaptive
indexing forward to a mature formulation that confers the workload-robustness
previous approaches lacked. Our extensive experimental study verifies that
stochastic cracking maintains the desired properties of original database
cracking while at the same time it performs well with diverse realistic
workloads.Comment: VLDB201
Practical Sparse Matrices in C++ with Hybrid Storage and Template-Based Expression Optimisation
Despite the importance of sparse matrices in numerous fields of science,
software implementations remain difficult to use for non-expert users,
generally requiring the understanding of underlying details of the chosen
sparse matrix storage format. In addition, to achieve good performance, several
formats may need to be used in one program, requiring explicit selection and
conversion between the formats. This can be both tedious and error-prone,
especially for non-expert users. Motivated by these issues, we present a
user-friendly and open-source sparse matrix class for the C++ language, with a
high-level application programming interface deliberately similar to the widely
used MATLAB language. This facilitates prototyping directly in C++ and aids the
conversion of research code into production environments. The class internally
uses two main approaches to achieve efficient execution: (i) a hybrid storage
framework, which automatically and seamlessly switches between three underlying
storage formats (compressed sparse column, Red-Black tree, coordinate list)
depending on which format is best suited and/or available for specific
operations, and (ii) a template-based meta-programming framework to
automatically detect and optimise execution of common expression patterns.
Empirical evaluations on large sparse matrices with various densities of
non-zero elements demonstrate the advantages of the hybrid storage framework
and the expression optimisation mechanism.Comment: extended and revised version of an earlier conference paper
arXiv:1805.0338
Practical Sparse Matrices in C++ with Hybrid Storage and Template-Based Expression Optimisation
Despite the importance of sparse matrices in numerous fields of science,
software implementations remain difficult to use for non-expert users,
generally requiring the understanding of underlying details of the chosen
sparse matrix storage format. In addition, to achieve good performance, several
formats may need to be used in one program, requiring explicit selection and
conversion between the formats. This can be both tedious and error-prone,
especially for non-expert users. Motivated by these issues, we present a
user-friendly and open-source sparse matrix class for the C++ language, with a
high-level application programming interface deliberately similar to the widely
used MATLAB language. This facilitates prototyping directly in C++ and aids the
conversion of research code into production environments. The class internally
uses two main approaches to achieve efficient execution: (i) a hybrid storage
framework, which automatically and seamlessly switches between three underlying
storage formats (compressed sparse column, Red-Black tree, coordinate list)
depending on which format is best suited and/or available for specific
operations, and (ii) a template-based meta-programming framework to
automatically detect and optimise execution of common expression patterns.
Empirical evaluations on large sparse matrices with various densities of
non-zero elements demonstrate the advantages of the hybrid storage framework
and the expression optimisation mechanism.Comment: extended and revised version of an earlier conference paper
arXiv:1805.0338
Dynamic Graphs on the GPU
We present a fast dynamic graph data structure for the GPU. Our dynamic graph structure uses one hash table per vertex to store adjacency lists and achieves 3.4–14.8x faster insertion rates over the state of the art across a diverse set of large datasets, as well as deletion speedups up to 7.8x. The data structure supports queries and dynamic updates through both edge and vertex insertion and deletion. In addition, we define a comprehensive evaluation strategy based on operations, workloads, and applications that we believe better characterize and evaluate dynamic graph data structures
Maunakea Spectroscopic Explorer (MSE) - The Prime Focus Subsystems: Requirements and Interfaces
MSE will be a massively multiplexed survey telescope, including a segmented
primary mirror which feeds fibers at the prime focus, including an array of
approximately four thousand fibers, positioned precisely to feed banks of
spectrographs several tens of meters away. We describe the process of mapping
top-level requirements on MSE to technical specifications for subsystems
located at the MSE prime focus. This includes the overall top-level
requirements based on knowledge of similar systems at other telescopes and how
those requirements were converted into specifications so that the subsystems
could begin working on their Conceptual Design Phases. We then discuss the
verification of the engineering specifications and the compiling of lower-level
requirements and specifications into higher level performance budgets (e.g.
Image Quality). We also briefly discuss the interface specifications, their
effect on the performance of the system and the plan to manage them going
forward. We also discuss the opto-mechanical design of the telescope top end
assembly and refer readers to more details for instrumentation located at the
top end.Comment: 14 pages; Proceedings of SPIE Astronomical Telescopes +
Instrumentation 2018; Modeling, Systems Engineering, and Project Management
for Astronomy VII
- …