Low-latency, query-driven analytics over voluminous multidimensional, spatiotemporal datasets

2017 Summer.Includes bibliographical references.Ubiquitous data collection from sources such as remote sensing equipment, networked observational devices, location-based services, and sales tracking has led to the accumulation of voluminous datasets; IDC projects that by 2020 we will generate 40 zettabytes of data per year, while Gartner and ABI estimate 20-35 billion new devices will be connected to the Internet in the same time frame. The storage and processing requirements of these datasets far exceed the capabilities of modern computing hardware, which has led to the development of distributed storage frameworks that can scale out by assimilating more computing resources as necessary. While challenging in its own right, storing and managing voluminous datasets is only the precursor to a broader field of study: extracting knowledge, insights, and relationships from the underlying datasets. The basic building block of this knowledge discovery process is analytic queries, encompassing both query instrumentation and evaluation. This dissertation is centered around query-driven exploratory and predictive analytics over voluminous, multidimensional datasets. Both of these types of analysis represent a higher-level abstraction over classical query models; rather than indexing every discrete value for subsequent retrieval, our framework autonomously learns the relationships and interactions between dimensions in the dataset (including time series and geospatial aspects), and makes the information readily available to users. This functionality includes statistical synopses, correlation analysis, hypothesis testing, probabilistic structures, and predictive models that not only enable the discovery of nuanced relationships between dimensions, but also allow future events and trends to be predicted. This requires specialized data structures and partitioning algorithms, along with adaptive reductions in the search space and management of the inherent trade-off between timeliness and accuracy. The algorithms presented in this dissertation were evaluated empirically on real-world geospatial time-series datasets in a production environment, and are broadly applicable across other storage frameworks

AC-RDVT: Acyclic Resource Distance Vector Routing Tables for Dynamic Grid Resource Discovery

Since the objective of grid is sharing the numerous and heterogeneous resources, resource discovery is a challenging issue. Recently appeared, Ontosum, is a resource discovery method based on semantically linked organizations and a routing algorithm Resource Distance Vector (RDV), has been presented to forward resource discovery queries into the clusters. Although this framework is efficient for large-scale grids and nodes are clustered automatically based on semantic attributes to constitute a semantically linked overlay network, but the dynamic behavior of grid isn’t considered. In this method, deceptive information is stored in RDV tables (RDVT) which cause some problems in routing process. In this paper, a method is proposed to improve the dynamism of RDV routing algorithm, so the consistency with grid environments is increased. The developed algorithm is assessed by investigating the success probability, number of hops and routing time of resource discovery.DOI:http://dx.doi.org/10.11591/ijece.v3i1.183

Assured information sharing for ad-hoc collaboration

Collaborative information sharing tends to be highly dynamic and often ad hoc among organizations. The dynamic natures and sharing patterns in ad-hoc collaboration impose a need for a comprehensive and flexible approach to reflecting and coping with the unique access control requirements associated with the environment. This dissertation outlines a Role-based Access Management for Ad-hoc Resource Shar- ing framework (RAMARS) to enable secure and selective information sharing in the het- erogeneous ad-hoc collaborative environment. Our framework incorporates a role-based approach to addressing originator control, delegation and dissemination control. A special trust-aware feature is incorporated to deal with dynamic user and trust management, and a novel resource modeling scheme is proposed to support fine-grained selective sharing of composite data. As a policy-driven approach, we formally specify the necessary pol- icy components in our framework and develop access control policies using standardized eXtensible Access Control Markup Language (XACML). The feasibility of our approach is evaluated in two emerging collaborative information sharing infrastructures: peer-to- peer networking (P2P) and Grid computing. As a potential application domain, RAMARS framework is further extended and adopted in secure healthcare services, with a unified patient-centric access control scheme being proposed to enable selective and authorized sharing of Electronic Health Records (EHRs), accommodating various privacy protection requirements at different levels of granularity

Parallel computations based on domain decompositions and integrated radial basis functions for fluid flow problems

The thesis reports a contribution to the development of parallel algorithms based on Domain Decomposition (DD) method and Compact Local Integrated Radial Basis Function (CLIRBF) method. This development aims to solve large scale fluid flow problems more efficiently by using parallel high performance computing (HPC). With the help of the DD method, one big problem can be separated into sub-problems and solved on parallel machines. In terms of numerical analysis, for each sub-problem, the overall condition number of the system matrix is significantly reduced. This is one of the main reasons for the stability, high accuracy and efficiency of parallel algorithms. The developed methods have been successfully applied to solve several benchmark problems with both rectangular and non-rectangular boundaries. In parallel computation, there is a challenge called Distributed Termination Detection (DTD) problem. DTD concerns the discovery whether all processes in a distributed system have finished their job. In a distributed system, this problem is not a trivial problem because there is neither a global synchronised clock nor a shared memory. Taking into account the specific requirement of parallel algorithms, a new algorithm is proposed and called the Bitmap DTD. This algorithm is designed to work with DD method for solving Partial Differential Equations (PDEs). The Bitmap DTD algorithm is inspired by the Credit/Recovery DTD class (or weight-throw). The distinguishing feature of this algorithm is the use of a bitmap to carry the snapshot of the system from process to process. The proposed algorithm possesses characteristics as follows. (i) It allows any process to detect termination (symmetry); (ii) it does not require any central control agent (decentralisation); (iii) termination detection delay is of the order of the diameter of the network; and (iv) the message complexity of the proposed algorithm is optimal. In the first attempt, the combination of the DD method and CLIRBF based collocation approach yields an effective parallel algorithm to solve PDEs. This approach has enabled not only the problem to be solved separately in each subdomain by a Central Processing Unit (CPU) but also compact local stencils to be independently treated. The present algorithm has achieved high throughput in solving large scale problems. The procedure is illustrated by several numerical examples including the benchmark lid-driven cavity flow problem. A new parallel algorithm is developed using the Control Volume Method (CVM) for the solution of PDEs. The goal is to develop an efficient parallel algorithm especially for problems with non-rectangular domains. When combined with CLIRBF approach, the resultant method can produce high-order accuracy and economical solution for problems with complex boundary. The algorithm is verified by solving two benchmark problems including the square lid-driven cavity flow and the triangular lid-driven cavity flow. In both cases, the accuracy is in great agreement with benchmark values. In terms of efficiency, the results show that the method has a very high efficiency profile and for some specific cases a super-linear speed-up is achieved. Although overlapping method yields a straightforward implementation and stable convergence, overlapping of sub-domains makes it less applicable for complex domains. The method even generates more computing overhead for each subdomain as the overlapping area grows. Hence, a parallel algorithm based on non-overlapping DD and CLIRBF has been developed for solving Navier-Stokes equations where a CLIRBF scheme is used to solve the problem in each subdomain. A relaxation factor is employed for the transmission conditions at the interface of sub-domains to ensure the convergence of the iterative method while the Bitmap DTD algorithm is used to achieve the global termination. The parallel algorithm is demonstrated through two fluid flow problems, namely the natural convection in concentric annuli (Boussinesq fluids) and the lid-driven cavity flow (viscous fluids). The results confirm the high efficiency of the present method in comparison with a sequential algorithm. A super-linear efficiency is also observed for a range of numbers of CPUs. Finally, when comparing the overlapping and non-overlapping parallel algorithms, it is found that the non-overlapping one is less stable. The numerical results show that the non-overlapping method is not able to converge for high Reynolds number while overlapping method reaches the same convergence profile as the sequential CLIRBF method. Thus, in this research when dealing with non-Newtonian fluids and large scale problems, the overlapping method is preferred to the nonoverlapping one. The flow of Oldroyd-B fluid through a planar contraction was considered as a benchmark problem. In this problem, the singularity of stress at the re-entrant corners always poses difficulty to numerical methods in obtaining stable solutions at high Weissenberg numbers. In this work, a high resolution simulation of the flow is obtained and the contour of streamline is shown to be in great agreement with other results

GraphCombEx: A Software Tool for Exploration of Combinatorial Optimisation Properties of Large Graphs

We present a prototype of a software tool for exploration of multiple combinatorial optimisation problems in large real-world and synthetic complex networks. Our tool, called GraphCombEx (an acronym of Graph Combinatorial Explorer), provides a unified framework for scalable computation and presentation of high-quality suboptimal solutions and bounds for a number of widely studied combinatorial optimisation problems. Efficient representation and applicability to large-scale graphs and complex networks are particularly considered in its design. The problems currently supported include maximum clique, graph colouring, maximum independent set, minimum vertex clique covering, minimum dominating set, as well as the longest simple cycle problem. Suboptimal solutions and intervals for optimal objective values are estimated using scalable heuristics. The tool is designed with extensibility in mind, with the view of further problems and both new fast and high-performance heuristics to be added in the future. GraphCombEx has already been successfully used as a support tool in a number of recent research studies using combinatorial optimisation to analyse complex networks, indicating its promise as a research software tool

Resource Management in a Peer to Peer Cloud Network for IoT

Software-Defined Internet of Things (SDIoT) is defined as merging heterogeneous objects in a form of interaction among physical and virtual entities. Large scale of data centers, heterogeneity issues and their interconnections have made the resource management a hard problem specially when there are different actors in cloud system with different needs. Resource management is a vital requirement to achieve robust networks specially with facing continuously increasing amount of heterogeneous resources and devices to the network. The goal of this paper is reviews to address IoT resource management issues in cloud computing services. We discuss the bottlenecks of cloud networks for IoT services such as mobility. We review Fog computing in IoT services to solve some of these issues. It provides a comprehensive literature review of around one hundred studies on resource management in Peer to Peer Cloud Networks and IoT. It is very important to find a robust design to efficiently manage and provision requests and available resources. We also reviewed different search methodologies to help clients find proper resources to answer their needs