HEC: Collaborative Research: SAM^2 Toolkit: Scalable and Adaptive Metadata Management for High-End Computing

The increasing demand for Exa-byte-scale storage capacity by high end computing applications requires a higher level of scalability and dependability than that provided by current file and storage systems. The proposal deals with file systems research for metadata management of scalable cluster-based parallel and distributed file storage systems in the HEC environment. It aims to develop a scalable and adaptive metadata management (SAM2) toolkit to extend features of and fully leverage the peak performance promised by state-of-the-art cluster-based parallel and distributed file storage systems used by the high performance computing community. There is a large body of research on data movement and management scaling, however, the need to scale up the attributes of cluster-based file systems and I/O, that is, metadata, has been underestimated. An understanding of the characteristics of metadata traffic, and an application of proper load-balancing, caching, prefetching and grouping mechanisms to perform metadata management correspondingly, will lead to a high scalability. It is anticipated that by appropriately plugging the scalable and adaptive metadata management components into the state-of-the-art cluster-based parallel and distributed file storage systems one could potentially increase the performance of applications and file systems, and help translate the promise and potential of high peak performance of such systems to real application performance improvements. The project involves the following components: 1. Develop multi-variable forecasting models to analyze and predict file metadata access patterns. 2. Develop scalable and adaptive file name mapping schemes using the duplicative Bloom filter array technique to enforce load balance and increase scalability 3. Develop decentralized, locality-aware metadata grouping schemes to facilitate the bulk metadata operations such as prefetching. 4. Develop an adaptive cache coherence protocol using a distributed shared object model for client-side and server-side metadata caching. 5. Prototype the SAM2 components into the state-of-the-art parallel virtual file system PVFS2 and a distributed storage data caching system, set up an experimental framework for a DOE CMS Tier 2 site at University of Nebraska-Lincoln and conduct benchmark, evaluation and validation studies

A Survey of Pipelined Workflow Scheduling: Models and Algorithms

International audienceA large class of applications need to execute the same workflow on different data sets of identical size. Efficient execution of such applications necessitates intelligent distribution of the application components and tasks on a parallel machine, and the execution can be orchestrated by utilizing task-, data-, pipelined-, and/or replicated-parallelism. The scheduling problem that encompasses all of these techniques is called pipelined workflow scheduling, and it has been widely studied in the last decade. Multiple models and algorithms have flourished to tackle various programming paradigms, constraints, machine behaviors or optimization goals. This paper surveys the field by summing up and structuring known results and approaches

Determining the Best Sensing Coverage for 2-Dimensional Acoustic Target Tracking

Distributed acoustic target tracking is an important application area of wireless sensor networks. In this paper we use algebraic geometry to formally model 2-dimensional acoustic target tracking and then prove its best degree of required sensing coverage. We present the necessary conditions for three sensing coverage to accurately compute the spatio-temporal information of a target object. Simulations show that 3-coverage accurately locates a target object only in 53% of cases. Using 4-coverage, we present two different methods that yield correct answers in almost all cases and have time and memory usage complexity of Θ(1). Analytic 4-coverage tracking is our first proposed method that solves a simultaneous equation system using the sensing information of four sensor nodes. Redundant answer fusion is our second proposed method that solves at least two sets of simultaneous equations of target tracking using the sensing information of two different sets of three sensor nodes, and fusing the results using a new customized formal majority voter. We prove that 4-coverage guarantees accurate 2-dimensional acoustic target tracking under ideal conditions

Construction of a real vehicular delay-tolerant network testbed

Vehicular Delay-Tolerant Networks (VDTNs) appear as innovative network architecture, able to outline communication challenges caused by issues like variable delays, disruption and intermittent connectivity once that it utilizes the store-carry-and-forward method to allow that in-transit messages (called bundles) can be delivered to the destination by hopping over the mobile vehicles even that an end-to-end path does not exist. Since messages are stored persistently in a buffer and forward to the next hop, a new communication infrastructure is created allowing low-cost asynchronous opportunistic communication under the most critical situations like variable delays and bandwidth constraints. VDTN introduces a layered architecture, acting as an overlay network over the link layer, aggregating incoming IP packets in data bundles (large IP packets), using out-of-band signaling, based on the separation of the control plane and planes. This dissertation presents and evaluates the performance of a real VDTN testbed, demonstrating the real applicability of this new vehicular communication approach. It includes an embedded VDTN testbed created to evaluate safety systems in a real-world scenario. It was used cars with laptops to realize terminal and relay nodes. A real testbed is very important because some complex issues presented in vehicular communication systems can be treated with more realism in real-world environments than in a laboratory environment. The experiments were performed on the internal streets of Brazilian Fiat Automobile manufacturing plant. Performance measurements and analysis were also conduct to verify the efficiency of the system. The results obtained show that safety applications and services can be executed with the actual proposal VDTN architecture in several environments, although notable interference as fading and characteristics of the radio channel, require the use of more modern, appropriate and robust technologies. Thus, the real deployment of VDTNs confirms that VDTNs can be seen as a very promising technology for vehicular communications.Fundação para a Ciência e a Tecnologia (FCT

SAMuS : service-oriented architecture for multisensor surveillance in smart homes

The design of a service-oriented architecture for multisensor surveillance in smart homes is presented as an integrated solution enabling automatic deployment, dynamic selection, and composition of sensors. Sensors are implemented as Web-connected devices, with a uniform Web API. RESTdesc is used to describe the sensors and a novel solution is presented to automatically compose Web APIs that can be applied with existing Semantic Web reasoners. We evaluated the solution by building a smart Kinect sensor that is able to dynamically switch between IR and RGB and optimizing person detection by incorporating feedback from pressure sensors, as such demonstrating the collaboration among sensors to enhance detection of complex events. The performance results show that the platform scales for many Web APIs as composition time remains limited to a few hundred milliseconds in almost all cases

hwloc: a Generic Framework for Managing Hardware Affinities in HPC Applications

International audienceThe increasing numbers of cores, shared caches and memory nodes within machines introduces a complex hardware topology. High-performance computing applications now have to carefully adapt their placement and behavior according to the underlying hierarchy of hardware resources and their software affinities. We introduce the Hardware Locality (hwloc) software which gathers hardware information about processors, caches, memory nodes and more, and exposes it to applications and runtime systems in a abstracted and portable hierarchical manner. hwloc may significantly help performance by having runtime systems place their tasks or adapt their communication strategies depending on hardware affinities. We show that hwloc can already be used by popular high-performance OpenMP or MPI software. Indeed, scheduling OpenMP threads according to their affinities or placing MPI processes according to their communication patterns shows interesting performance improvement thanks to hwloc. An optimized MPI communication strategy may also be dynamically chosen according to the location of the communicating processes in the machine and its hardware characteristics

Reliable Delay Constrained Multihop Broadcasting in VANETs

Vehicular communication is regarded as a major innovative feature for in-car technology. While improving road safety is unanimously considered the major driving factor for the deployment of Intelligent Vehicle Safety Systems, the challenges relating to reliable multi-hop broadcasting are exigent in vehicular networking. In fact, safety applications must rely on very accurate and up-to-date information about the surrounding environment, which in turn requires the use of accurate positioning systems and smart communication protocols for exchanging information. Communications protocols for VANETs must guarantee fast and reliable delivery of information to all vehicles in the neighbourhood, where the wireless communication medium is shared and highly unreliable with limited bandwidth. In this paper, we focus on mechanisms that improve the reliability of broadcasting protocols, where the emphasis is on satisfying the delay requirements for safety applications. We present the Pseudoacknowledgments (PACKs) scheme and compare this with existing methods over varying vehicle densities in an urban scenario using the network simulator OPNET

Fast and Space-Efficient Queues via Relaxation

Efficient message-passing implementations of shared data types are a vital component of practical distributed systems, enabling them to work on shared data in predictable ways, but there is a long history of results showing that many of the most useful types of access to shared data are necessarily slow. A variety of approaches attempt to circumvent these bounds, notably weakening consistency guarantees and relaxing the sequential specification of the provided data type. These trade behavioral guarantees for performance. We focus on relaxing the sequential specification of a first-in, first-out queue type, which has been shown to allow faster linearizable implementations than are possible for traditional FIFO queues without relaxation. The algorithms which showed these improvements in operation time tracked a complete execution history, storing complete object state at all n processes in the system, leading to n copies of every stored data element. In this paper, we consider the question of reducing the space complexity of linearizable implementations of shared data types, which provide intuitive behavior through strong consistency guarantees. We improve the existing algorithm for a relaxed queue, showing that it is possible to store only one copy of each element in a shared queue, while still having a low amortized time cost. This is one of several important steps towards making these data types practical in real world systems