Efficient Iterative Processing in the SciDB Parallel Array Engine

Many scientific data-intensive applications perform iterative computations on array data. There exist multiple engines specialized for array processing. These engines efficiently support various types of operations, but none includes native support for iterative processing. In this paper, we develop a model for iterative array computations and a series of optimizations. We evaluate the benefits of an optimized, native support for iterative array processing on the SciDB engine and real workloads from the astronomy domain

Incremental eigenpair computation for graph Laplacian matrices: theory and applications

The smallest eigenvalues and the associated eigenvectors (i.e., eigenpairs) of a graph Laplacian matrix have been widely used for spectral clustering and community detection. However, in real-life applications, the number of clusters or communities (say, K) is generally unknown a priori. Consequently, the majority of the existing methods either choose K heuristically or they repeat the clustering method with different choices of K and accept the best clustering result. The first option, more often, yields suboptimal result, while the second option is computationally expensive. In this work, we propose an incremental method for constructing the eigenspectrum of the graph Laplacian matrix. This method leverages the eigenstructure of graph Laplacian matrix to obtain the Kth smallest eigenpair of the Laplacian matrix given a collection of all previously compute

A market-based transmission planning for HVDC grid—case study of the North Sea

There is significant interest in building HVDC transmission to carry out transnational power exchange and deliver cheaper electricity from renewable energy sources which are located far from the load centers. This paper presents a market-based approach to solve a long-term TEP for meshed VSC-HVDC grids that connect regional markets. This is in general a nonlinear, non-convex large-scale optimization problem with high computational burden, partly due to the many combinations of wind and load that become possible. We developed a two-step iterative algorithm that first selects a subset of operating hours using a clustering technique, and then seeks to maximize the social welfare of all regions and minimize the investment capital of transmission infrastructure subject to technical and economic constraints. The outcome of the optimization is an optimal grid design with a topology and transmission capacities that results in congestion revenue paying off investment by the end the project's economic lifetime. Approximations are made to allow an analytical solution to the problem and demonstrate that an HVDC pricing mechanism can be consistent with an AC counterpart. The model is used to investigate development of the offshore grid in the North Sea. Simulation results are interpreted in economic terms and show the effectiveness of our proposed two-step approach