1,235 research outputs found
Distributed-Memory Breadth-First Search on Massive Graphs
This chapter studies the problem of traversing large graphs using the
breadth-first search order on distributed-memory supercomputers. We consider
both the traditional level-synchronous top-down algorithm as well as the
recently discovered direction optimizing algorithm. We analyze the performance
and scalability trade-offs in using different local data structures such as CSR
and DCSC, enabling in-node multithreading, and graph decompositions such as 1D
and 2D decomposition.Comment: arXiv admin note: text overlap with arXiv:1104.451
Recommended from our members
MobileTrust: Secure Knowledge Integration in VANETs
Vehicular Ad hoc NETworks (VANET) are becoming popular due to the emergence of the Internet of Things and ambient intelligence applications. In such networks, secure resource sharing functionality is accomplished by incorporating trust schemes. Current solutions adopt peer-to-peer technologies that can cover the large operational area. However, these systems fail to capture some inherent properties of VANETs, such as fast and ephemeral interaction, making robust trust evaluation of crowdsourcing challenging. In this article, we propose MobileTrustâa hybrid trust-based system for secure resource sharing in VANETs. The proposal is a breakthrough in centralized trust computing that utilizes cloud and upcoming 5G technologies to provide robust trust establishment with global scalability. The ad hoc communication is energy-efficient and protects the system against threats that are not countered by the current settings. To evaluate its performance and effectiveness, MobileTrust is modelled in the SUMO simulator and tested on the traffic features of the small-size German city of Eichstatt. Similar schemes are implemented in the same platform to provide a fair comparison. Moreover, MobileTrust is deployed on a typical embedded system platform and applied on a real smart car installation for monitoring traffic and road-state parameters of an urban application. The proposed system is developed under the EU-founded THREAT-ARREST project, to provide security, privacy, and trust in an intelligent and energy-aware transportation scenario, bringing closer the vision of sustainable circular economy
A Taxonomy of Data Grids for Distributed Data Sharing, Management and Processing
Data Grids have been adopted as the platform for scientific communities that
need to share, access, transport, process and manage large data collections
distributed worldwide. They combine high-end computing technologies with
high-performance networking and wide-area storage management techniques. In
this paper, we discuss the key concepts behind Data Grids and compare them with
other data sharing and distribution paradigms such as content delivery
networks, peer-to-peer networks and distributed databases. We then provide
comprehensive taxonomies that cover various aspects of architecture, data
transportation, data replication and resource allocation and scheduling.
Finally, we map the proposed taxonomy to various Data Grid systems not only to
validate the taxonomy but also to identify areas for future exploration.
Through this taxonomy, we aim to categorise existing systems to better
understand their goals and their methodology. This would help evaluate their
applicability for solving similar problems. This taxonomy also provides a "gap
analysis" of this area through which researchers can potentially identify new
issues for investigation. Finally, we hope that the proposed taxonomy and
mapping also helps to provide an easy way for new practitioners to understand
this complex area of research.Comment: 46 pages, 16 figures, Technical Repor
BlobSeer: How to Enable Efficient Versioning for Large Object Storage under Heavy Access Concurrency
To accommodate the needs of large-scale distributed P2P systems, scalable
data management strategies are required, allowing applications to efficiently
cope with continuously growing, highly dis tributed data. This paper addresses
the problem of efficiently stor ing and accessing very large binary data
objects (blobs). It proposesan efficient versioning scheme allowing a large
number of clients to concurrently read, write and append data to huge blobs
that are fragmented and distributed at a very large scale. Scalability under
heavy concurrency is achieved thanks to an original metadata scheme, based on a
distributed segment tree built on top of a Distributed Hash Table (DHT). Our
approach has been implemented and experimented within our BlobSeer prototype on
the Grid'5000 testbed, using up to 175 nodes
Big Data and Large-scale Data Analytics: Efficiency of Sustainable Scalability and Security of Centralized Clouds and Edge Deployment Architectures
One of the significant shifts of the next-generation computing technologies will certainly be in
the development of Big Data (BD) deployment architectures. Apache Hadoop, the BD
landmark, evolved as a widely deployed BD operating system. Its new features include
federation structure and many associated frameworks, which provide Hadoop 3.x with the
maturity to serve different markets. This dissertation addresses two leading issues involved in
exploiting BD and large-scale data analytics realm using the Hadoop platform. Namely,
(i)Scalability that directly affects the system performance and overall throughput using
portable Docker containers. (ii) Security that spread the adoption of data protection practices
among practitioners using access controls. An Enhanced Mapreduce Environment (EME),
OPportunistic and Elastic Resource Allocation (OPERA) scheduler, BD Federation Access Broker
(BDFAB), and a Secure Intelligent Transportation System (SITS) of multi-tiers architecture for
data streaming to the cloud computing are the main contribution of this thesis study
Pando: Personal Volunteer Computing in Browsers
The large penetration and continued growth in ownership of personal
electronic devices represents a freely available and largely untapped source of
computing power. To leverage those, we present Pando, a new volunteer computing
tool based on a declarative concurrent programming model and implemented using
JavaScript, WebRTC, and WebSockets. This tool enables a dynamically varying
number of failure-prone personal devices contributed by volunteers to
parallelize the application of a function on a stream of values, by using the
devices' browsers. We show that Pando can provide throughput improvements
compared to a single personal device, on a variety of compute-bound
applications including animation rendering and image processing. We also show
the flexibility of our approach by deploying Pando on personal devices
connected over a local network, on Grid5000, a French-wide computing grid in a
virtual private network, and seven PlanetLab nodes distributed in a wide area
network over Europe.Comment: 14 pages, 12 figures, 2 table
SuperScaler: Supporting Flexible DNN Parallelization via a Unified Abstraction
With the growing model size, deep neural networks (DNN) are increasingly
trained over massive GPU accelerators, which demands a proper parallelization
plan that transforms a DNN model into fine-grained tasks and then schedules
them to GPUs for execution. Due to the large search space, the contemporary
parallelization plan generators often rely on empirical rules that couple
transformation and scheduling, and fall short in exploring more flexible
schedules that yield better memory usage and compute efficiency. This tension
can be exacerbated by the emerging models with increasing complexity in their
structure and model size. SuperScaler is a system that facilitates the design
and generation of highly flexible parallelization plans. It formulates the plan
design and generation into three sequential phases explicitly: model
transformation, space-time scheduling, and data dependency preserving. Such a
principled approach decouples multiple seemingly intertwined factors and
enables the composition of highly flexible parallelization plans. As a result,
SuperScaler can not only generate empirical parallelization plans, but also
construct new plans that achieve up to 3.5X speedup compared to
state-of-the-art solutions like DeepSpeed, Megatron and Alpa, for emerging DNN
models like Swin-Transformer and AlphaFold2, as well as well-optimized models
like GPT-3
- âŠ