Distributed storage and cloud computing: a test case

Since 2003 the computing farm hosted by the INFN Tier3 facility in Trieste supports the activities of many scientific communities. Hundreds of jobs from 45 different VOs, including those of the LHC experiments, are processed simultaneously. Given that normally the requirements of the different computational communities are not synchronized, the probability that at any given time the resources owned by one of the participants are not fully utilized is quite high. A balanced compensation should in principle allocate the free resources to other users, but there are limits to this mechanism. In fact, the Trieste site may not hold the amount of data needed to attract enough analysis jobs, and even in that case there could be a lack of bandwidth for their access. The Trieste ALICE and CMS computing groups, in collaboration with other Italian groups, aim to overcome the limitations of existing solutions using two approaches: sharing the data among all the participants taking full advantage of GARR-X wide area networks (10 GB/s) and integrating the resources dedicated to batch analysis with the ones reserved for dynamic interactive analysis, through modern solutions as cloud computing

Building a high-level architecture federated interoperable framework from legacy information systems

International audienceThis paper aims at improving the re-implementation of existing information systems when they are called to be involved in a system of systems, i.e. a federation of enterprise information systems that interoperate. The idea is reusing the local experiences coming from the previous development of the existing information system with the process of model discovery. To avoid redeveloping the entire system when the enterprise needs to cooperate with others, this approach proposes to create local interfaces to code and decode information. The interfaces are instantiated by using models discovered. The interfaces are developed in accordance with the high-level architecture (HLA) standard that proposes message interoperability and synchronisation mechanisms among distributed systems. First, this paper recalls the strong points of model-driven architecture (MDA)/architecture-driven modernisation methodologies for model transformation from conceptual level to implementation and the HLA standard. Then, a MDA and HLA framework is proposed to implement distributed enterprise components from the conceptual level through a federated enterprise interoperability approach. In addition, a model reversal methodology is developed under the framework to guide the re-implementation of legacy information systems to achieve desired interoperability with other systems. To extend the scope of the approach, implemented Web services are combined with HLA in order to facilitate the use of HLA in large distributed execution. This paper ends with an implementation example for validating the approach

BonFIRE: A multi-cloud test facility for internet of services experimentation

BonFIRE offers a Future Internet, multi-site, cloud testbed, targeted at the Internet of Services community, that supports large scale testing of applications, services and systems over multiple, geographically distributed, heterogeneous cloud testbeds. The aim of BonFIRE is to provide an infrastructure that gives experimenters the ability to control and monitor the execution of their experiments to a degree that is not found in traditional cloud facilities. The BonFIRE architecture has been designed to support key functionalities such as: resource management; monitoring of virtual and physical infrastructure metrics; elasticity; single document experiment descriptions; and scheduling. As for January 2012 BonFIRE release 2 is operational, supporting seven pilot experiments. Future releases will enhance the offering, including the interconnecting with networking facilities to provide access to routers, switches and bandwidth-on-demand systems. BonFIRE will be open for general use late 2012

Distributed Simulation in Manufacturing Using High Level Architecture

Manufacturing is a critical industry for all major economies. Every individual and industry depends on manufactured goods, which makes manufacturing crucial to the national economies. Competition is increasingly hard and globalization is leading to worldwide distribution of production, products and services, affecting all countries and economical regions. At the same time, markets are changing. Customers call for faster product changes and demand products, which are increasingly targeted to individual needs. Mass production is therefore replaced by customised and personalised production of individual products. Distributed simulation has the potential to become widely applicable for geographically-dispersed manufacturing environments, as is the case with desktop manufacturing or rapidly deployable micro-assembly stations. This thesis focuses on creating a generic framework that permits the distribution of manufacturing simulations, which was one of the goals of the MS2Value (Modeling and Simulation of Manufacturing Systems for Value Chains) project. Companies, nowadays, normally have their activities and resources geographically dispersed, which represents a challenge for the reusability and interconnection of their manufacturing simulation models. Different approaches have been taken by different communities like the research and military community, but no solution has been presented yet in the manufacturing field. The thesis work presented here proposes the use of the HLA (High Level Architecture) in combination with a simulation software as a solution to these problems. This proposal is demonstrated by an implementation of a distributed simulation using 3DCreate and an open source RTI (Runtime Infrastructure). /Kir1

Speedes: A Case Study Of Space Operations

This thesis describes the application of parallel simulation techniques to represent the structured functional parallelism present within the Space Shuttle Operations Flow using the Synchronous Parallel Environment for Emulation and Discrete-Event Simulation (SPEEDES), an object-oriented multi-computing architecture. SPEEDES is a unified parallel simulation environment, which allocates events over multiple processors to get simulation speed up. Its optimistic processing capability minimizes simulation lag time behind wall clock time, or multiples of real-time. SPEEDES accommodates an increase in process complexity with additional parallel computing nodes to allow sharing of processing loads. This thesis focuses on the process of translating a model of Space Shuttle Operations from a procedural oriented and single processor approach to one represented in a process-driven, object-oriented, and distributed processor approach. The processes are depicted by several classes created to represent the operations at the space center. The reference model used is the existing Space Shuttle Model created in ARENA by NASA and UCF in the year 2001. A systematic approach was used for this translation. A reduced version of the ARENA model was created, and then used as the SPEEDES prototype using C++. The prototype was systematically augmented to reflect the entire Space Shuttle Operations Flow. It was then verified, validated, and implemented

Beyond Nagios - Design of a cloud monitoring system

The paper describes a monitoring system specially designed for cloud infrastructures. The features that are relevant for such distributed application are -) scalability, that allows utilization in systems of thousands of nodes, -) flexibility, to be customized for a large number of applications, -) openness, to allow the coexistence of user and administration monitoring. We take as a starting point the Nagios monitoring system, that has been successfully used for Grid monitoring and is still used for clouds. We analyze its shortcomings when applied to cloud monitoring, and propose a new monitoring system, that we call Rocmon, that sums up Nagios experience with a cloud perspective. Like Nagios, Rocmon is plugin-oriented to be flexible. To be fully inter-operable and long-living, it uses standard tools: the OGF OCCI for the configuration interface, the REST paradigm to take advantage of Web tools, and HTML5 WebSockets for data transfers. The design is checked with an open source Ruby implementation featuring the most relevant aspects

A Collaborative Software Infrastructure based on the High Level Architecture and XML

A study is made of using the High Level Architecture (HLA) as foundation for distributed applications in the domain of Computer-Supported Collaborative Work. A plug-in, peer-to-peer infrastructure for such applications is proposed, aimed at facilitating development and management of collaborative software. Users of the framework collaborate in groups and sessions, described by a replicated state XML information model. A prototype infrastructure is developed, along with three prototype collaborative applications. Results of performance testing show that a transport system built on HLA compares reasonably well with a socket-based transport system. On the whole, results demonstrate feasibility of the infrastructure and of the objective of extending the HLA to non-simulation applications. Future work to adapt full-scale applications to the collaborative infrastructure is invited

Kestrel: Job Distribution and Scheduling using XMPP

A new distributed computing framework, named Kestrel, for Many-Task Computing (MTC) applications and implementing Virtual Organization Clusters (VOCs) is proposed. Kestrel is a lightweight, highly available system based on the Extensible Messaging and Presence Protocol (XMPP), and has been developed to explore XMPP-based techniques for improving MTC and VOC tolerance to faults due to scaling and intermittently connected heterogeneous resources. Kestrel provides a VOC with a special purpose scheduler for VOCs which can provide better scalability under certain workload assumptions, namely CPU bound processes and bag-of-task applications. Experimental results have shown that Kestrel is capable of operating a VOC of at least 1600 worker nodes with all nodes visible to the scheduler at once. When using multiple sites located in both North America and Europe, the latencies introduced to the round trip time of messages were on the order of 0.3 seconds. To offset the overhead of XMPP processing, a task execution time of 2 seconds is sufficient for a pool of 900 workers on a single site to operate at near 100% use. Requiring tasks that take on the order of 30 seconds to a minute to execute would compensate for increased latency during job dispatch across multiple sites. Kestrel\u27s architecture is rooted in pilot job frameworks heavily used in Grid computing, it is also modeled after the use of IRC by botnets to communicate between compromised machines and command and control servers. For Kestrel, the extensibility of XMPP has allowed development of protocols for identifying manager nodes, discovering the capabilities of worker agents, and for distributing tasks. The presence notifications provided by XMPP allow Kestrel to monitor the global state of the pool and to perform task dispatching based on worker availability. In this work it is argued that XMPP is by design a very good fit for cloud computing frameworks. It offers scalability, federation between servers and some autonomicity of the agents. During the summer of 2010, Kestrel was used and modified based on feedback from the STAR group at Brookhaven National Laboratories. STAR provided a virtual machine image with applications for simulating proton collisions using PYTHIA and GEANT3. A Kestrel-based virtual organization cluster, created on top of Clemson University\u27s Palmetto cluster, was able to provide over 400,000 CPU hours of computation over the course of a month using an average of 800 virtual machine instances every day, generating nearly seven terabytes of data and the largest PYTHIA production run that STAR ever achieved. Several architectural issues were encountered during the course of the experiment and were resolved by moving from the original JSON protocols used by Kestrel to native XMPP equivalents that offered better message delivery confirmation and integration with existing tools

A decentralized framework for cross administrative domain data sharing

Federation of messaging and storage platforms located in remote datacenters is an essential functionality to share data among geographically distributed platforms. When systems are administered by the same owner data replication reduces data access latency bringing data closer to applications and enables fault tolerance to face disaster recovery of an entire location. When storage platforms are administered by different owners data replication across different administrative domains is essential for enterprise application data integration. Contents and services managed by different software platforms need to be integrated to provide richer contents and services. Clients may need to share subsets of data in order to enable collaborative analysis and service integration. Platforms usually include proprietary federation functionalities and specific APIs to let external software and platforms access their internal data. These different techniques may not be applicable to all environments and networks due to security and technological restrictions. Moreover the federation of dispersed nodes under a decentralized administration scheme is still a research issue. This thesis is a contribution along this research direction as it introduces and describes a framework, called \u201cWideGroups\u201d, directed towards the creation and the management of an automatic federation and integration of widely dispersed platform nodes. It is based on groups to exchange messages among distributed applications located in different remote datacenters. Groups are created and managed using client side programmatic configuration without touching servers. WideGroups enables the extension of the software platform services to nodes belonging to different administrative domains in a wide area network environment. It lets different nodes form ad-hoc overlay networks on-the-fly depending on message destinations located in distinct administrative domains. It supports multiple dynamic overlay networks based on message groups, dynamic discovery of nodes and automatic setup of overlay networks among nodes with no server-side configuration. I designed and implemented platform connectors to integrate the framework as the federation module of Message Oriented Middleware and Key Value Store platforms, which are among the most widespread paradigms supporting data sharing in distributed systems