Configuration Management for Distributed Software Services

The paper describes the SysMan approach to interactive configuration management of distributed software components (objects). Domains are used to group objects to apply policy and for convenient naming of objects. Configuration Management involves using a domain browser to locate relevant objects within the domain service; creating new objects which form a distributed service; allocating these objects to physical nodes in the system and binding the interfaces of the objects to each other and to existing services. Dynamic reconfiguration of the objects forming a service can be accomplished using this tool. Authorisation policies specify which domains are accessible by which managers and which interfaces can be bound together. Keywords Domains, object creation, object binding, object allocation, graphical management interface. 1 INTRODUCTION The object-oriented approach brings considerable benefits to the design and implementation of software for distributed systems (Kramer 1992). Con..

The GENESIS platform, its distribution, and web services

The GENESIS project is developing an Open Source platform that supports co-operation and communication among software engineers belonging to distributed development teams involved in modeling, controlling, and measuring software development and maintenance processes. The GENESIS platform is made up of three main elements: a distributed workflow management system, a resource management system, and an artefact management system (OSCAR, developed at Durham). The platform is designed to be non-invasive and have a low barrier to entry (in terms of the effort required to begin using the system). This is accomplished, as far as possible, by adapting the platform to the workflow processes and tools already in place in an organisation. OSCAR (Open-Source Component Artefact Repository) is the artefact management system, used to store and retrieve any item produced by any member of a software engineering team. Traditional artefacts (documents and code, for example) as well as non-traditional items (such as informal annotations, mailing list postings, and personnel profiles) are managed by the system, which has the capability to maintain a rich set of relationships between the artefacts (for traceability and comprehension purposes). Each instance of OSCAR contains a software configuration management system (currently, a plugin is provided to use CVS). Currently, OSCAR is slightly distributed: the workflow management system can use more than one instance of a repository, but a single instance of OSCAR can use only one repository. There are a few known systems which provide some form of real distributed software configuration management, which, it is hoped, can be used to inspire further development of the distribution of OSCAR and its associated services. Initially, OSCAR and the rest of the GENESIS platform communicated using RMI, but a web service interface is currently under development. As an initial attempt at realising this, the RMI interface is simply wrapped to provide servlets. The web services approach allows for a single instance of an OSCAR repository to serve many projects, and for potential global distribution of a single instance of the GENESIS platform. Possible avenues for future work include: applications to e-learning and e-science (applying OSCAR to the Grid, in order to support educational and scientific collaboration); using OSCAR as a basis for supporting collaborative design work; instrumenting the tools in the GENESIS platform to provide data for studies of software engineering practices; and studying process models (for example, determining the difference between the ideal models defined in the literature and the real processes of software engineering)

Extensible Signaling Framework for Decentralized Network Management Applications

The management of network infrastructures has become increasingly complex over time, which is mainly attributed to the introduction of new functionality to support emerging services and applications. To address this important issue, research efforts in the last few years focused on developing Software-Defined Networking solutions. While initial work proposed centralized architectures, their scalability limitations have led researchers to investigate a distributed control plane, with controller placement algorithms and mechanisms for building a logically centralized network view, being examples of challenges addressed. A critical issue that has not been adequately addressed concerns the communication between distributed decision-making entities to ensure configuration consistency. To this end, this paper proposes a signaling framework that can allow the exchange of information in distributed management and control scenarios. The benefits of the proposed framework are illustrated through a realistic network resource management use case. Based on simulation, we demonstrate the flexibility and extensibility of our solution in meeting the requirements of distributed decision-making processes

Distributed Web Service Coordination for Collaboration Applications and Biological Workflows

In this dissertation work, we have investigated the main research thrust of decentralized coordination of workflows over web services. To address distributed workflow coordination, first we have developed “Web Coordination Bonds” as a capable set of dependency modeling primitives that enable each web service to manage its own dependencies. Web bond primitives are as powerful as extended Petri nets and have sufficient modeling and expressive capabilities to model workflow dependencies. We have designed and prototyped our “Web Service Coordination Management Middleware” (WSCMM) system that enhances current web services infrastructure to accommodate web bond enabled web services. Finally, based on core concepts of web coordination bonds and WSCMM, we have developed the “BondFlow” system that allows easy configuration distributed coordination of workflows. The footprint of the BonFlow runtime is 24KB and the additional third party software packages, SOAP client and XML parser, account for 115KB

A Novel Internet-of-Things Infrastructure to Support Self-Healing Distribution Systems

In this paper, we present a novel distributed software infrastructure to foster new services in smart grids with particular emphasis on supporting self-healing distribution systems. This infrastructure exploits the rising Internet-of-Things paradigms to build and manage an interoperable peer-to-peer network of our prototype smart meters, also presented in this paper. The proposed three-phase smart meter, called 3-SMA, is a low cost and open-source Internet-connected device that provides features for self-configuration. In addition, it selectively run onboard-algorithms for smart grid management depending on its deployment on the distribution network. Finally, we present the experimental results of Hardware-In-the-Loop simulations we performed

A decentralized framework for cross administrative domain data sharing

Federation of messaging and storage platforms located in remote datacenters is an essential functionality to share data among geographically distributed platforms. When systems are administered by the same owner data replication reduces data access latency bringing data closer to applications and enables fault tolerance to face disaster recovery of an entire location. When storage platforms are administered by different owners data replication across different administrative domains is essential for enterprise application data integration. Contents and services managed by different software platforms need to be integrated to provide richer contents and services. Clients may need to share subsets of data in order to enable collaborative analysis and service integration. Platforms usually include proprietary federation functionalities and specific APIs to let external software and platforms access their internal data. These different techniques may not be applicable to all environments and networks due to security and technological restrictions. Moreover the federation of dispersed nodes under a decentralized administration scheme is still a research issue. This thesis is a contribution along this research direction as it introduces and describes a framework, called \u201cWideGroups\u201d, directed towards the creation and the management of an automatic federation and integration of widely dispersed platform nodes. It is based on groups to exchange messages among distributed applications located in different remote datacenters. Groups are created and managed using client side programmatic configuration without touching servers. WideGroups enables the extension of the software platform services to nodes belonging to different administrative domains in a wide area network environment. It lets different nodes form ad-hoc overlay networks on-the-fly depending on message destinations located in distinct administrative domains. It supports multiple dynamic overlay networks based on message groups, dynamic discovery of nodes and automatic setup of overlay networks among nodes with no server-side configuration. I designed and implemented platform connectors to integrate the framework as the federation module of Message Oriented Middleware and Key Value Store platforms, which are among the most widespread paradigms supporting data sharing in distributed systems

DIRAC - Distributed Infrastructure with Remote Agent Control

This paper describes DIRAC, the LHCb Monte Carlo production system. DIRAC has a client/server architecture based on: Compute elements distributed among the collaborating institutes; Databases for production management, bookkeeping (the metadata catalogue) and software configuration; Monitoring and cataloguing services for updating and accessing the databases. Locally installed software agents implemented in Python monitor the local batch queue, interrogate the production database for any outstanding production requests using the XML-RPC protocol and initiate the job submission. The agent checks and, if necessary, installs any required software automatically. After the job has processed the events, the agent transfers the output data and updates the metadata catalogue. DIRAC has been successfully installed at 18 collaborating institutes, including the DataGRID, and has been used in recent Physics Data Challenges. In the near to medium term future we must use a mixed environment with different types of grid middleware or no middleware. We describe how this flexibility has been achieved and how ubiquitously available grid middleware would improve DIRAC.Comment: Talk from the 2003 Computing in High Energy and Nuclear Physics (CHEP03), La Jolla, Ca, USA, March 2003, 8 pages, Word, 5 figures. PSN TUAT00

Running user-provided virtual machines in batch-oriented computing clusters

The use of virtualization in HPC clusters can provide rich software environments, application isolation and efficient workload management mechanisms, but system-level virtualization introduces a software layer on the computing nodes that reduces performance and inhibits the direct use of hardware devices. We propose an unobtrusive user-level platform that allows the execution of virtual machines inside batch jobs without limiting the computing cluster’s ability to execute the most demanding applications. A per-user platform uses a static mode in which the VMs run entirely using the resources of a single batch job and a dynamic mode in which the VMs navigate at runtime between the continuously allocated jobs node time-slots. A dynamic mode is introduced to build complex scenarios with several VMs for personalized HPC environments or persistent services such as databases or web services based applications. Fault-tolerant system agents, integrated using group communication primitives, control the system and execute user commands and automatic scheduling decisions made by an optional monitoring function. The performance of compute intensive applications running on our system suffers negligible overhead compared to the native configuration. The performance of distributed applications is dependent on their communication patterns as the user-mode network overlay introduces a relevant communication overhead.FC

Dynamic Model-based Management of Service-Oriented Infrastructure.

Models are an effective tool for systems and software design. They allow software architects to abstract from the non-relevant details. Those qualities are also useful for the technical management of networks, systems and software, such as those that compose service oriented architectures. Models can provide a set of well-defined abstractions over the distributed heterogeneous service infrastructure that enable its automated management. We propose to use the managed system as a source of dynamically generated runtime models, and decompose management processes into a composition of model transformations. We have created an autonomic service deployment and configuration architecture that obtains, analyzes, and transforms system models to apply the required actions, while being oblivious to the low-level details. An instrumentation layer automatically builds these models and interprets the planned management actions to the system. We illustrate these concepts with a distributed service update operation

Software Defined Application Delivery Networking

In this thesis we present the architecture, design, and prototype implementation details of AppFabric. AppFabric is a next generation application delivery platform for easily creating, managing and controlling massively distributed and very dynamic application deployments that may span multiple datacenters. Over the last few years, the need for more flexibility, finer control, and automatic management of large (and messy) datacenters has stimulated technologies for virtualizing the infrastructure components and placing them under software-based management and control; generically called Software-defined Infrastructure (SDI). However, current applications are not designed to leverage this dynamism and flexibility offered by SDI and they mostly depend on a mix of different techniques including manual configuration, specialized appliances (middleboxes), and (mostly) proprietary middleware solutions together with a team of extremely conscientious and talented system engineers to get their applications deployed and running. AppFabric, 1) automates the whole control and management stack of application deployment and delivery, 2) allows application architects to define logical workflows consisting of application servers, message-level middleboxes, packet-level middleboxes and network services (both, local and wide-area) composed over application-level routing policies, and 3) provides the abstraction of an application cloud that allows the application to dynamically (and automatically) expand and shrink its distributed footprint across multiple geographically distributed datacenters operated by different cloud providers. The architecture consists of a hierarchical control plane system called Lighthouse and a fully distributed data plane design (with no special hardware components such as service orchestrators, load balancers, message brokers, etc.) called OpenADN . The current implementation (under active development) consists of ~10000 lines of python and C code. AppFabric will allow applications to fully leverage the opportunities provided by modern virtualized Software-Defined Infrastructures. It will serve as the platform for deploying massively distributed, and extremely dynamic next generation application use-cases, including: Internet-of-Things/Cyber-Physical Systems: Through support for managing distributed gather-aggregate topologies common to most Internet-of-Things(IoT) and Cyber-Physical Systems(CPS) use-cases. By their very nature, IoT and CPS use cases are massively distributed and have different levels of computation and storage requirements at different locations. Also, they have variable latency requirements for their different distributed sites. Some services, such as device controllers, in an Iot/CPS application workflow may need to gather, process and forward data under near-real time constraints and hence need to be as close to the device as possible. Other services may need more computation to process aggregated data to drive long term business intelligence functions. AppFabric has been designed to provide support for such very dynamic, highly diversified and massively distributed application use-cases. Network Function Virtualization: Through support for heterogeneous workflows, application-aware networking, and network-aware application deployments, AppFabric will enable new partnerships between Application Service Providers (ASPs) and Network Service Providers (NSPs). An application workflow in AppFabric may comprise of application services, packet and message-level middleboxes, and network transport services chained together over an application-level routing substrate. The Application-level routing substrate allows policy-based service chaining where the application may specify policies for routing their application traffic over different services based on application-level content or context. Virtual worlds/multiplayer games: Through support for creating, managing and controlling dynamic and distributed application clouds needed by these applications. AppFabric allows the application to easily specify policies to dynamically grow and shrink the application\u27s footprint over different geographical sites, on-demand. Mobile Apps: Through support for extremely diversified and very dynamic application contexts typical of such applications. Also, AppFabric provides support for automatically managing massively distributed service deployment and controlling application traffic based on application-level policies. This allows mobile applications to provide the best Quality-of-Experience to its users without This thesis is the first to handle and provide a complete solution for such a complex and relevant architectural problem that is expected to touch each of our lives by enabling exciting new application use-cases that are not possible today. Also, AppFabric is a non-proprietary platform that is expected to spawn lots of innovations both in the design of the platform itself and the features it provides to applications. AppFabric still needs many iterations, both in terms of design and implementation maturity. This thesis is not the end of journey for AppFabric but rather just the beginning