Self-management for large-scale distributed systems

Autonomic computing aims at making computing systems self-managing by using autonomic managers in order to reduce obstacles caused by management complexity. This thesis presents results of research on self-management for large-scale distributed systems. This research was motivated by the increasing complexity of computing systems and their management. In the first part, we present our platform, called Niche, for programming self-managing component-based distributed applications. In our work on Niche, we have faced and addressed the following four challenges in achieving self-management in a dynamic environment characterized by volatile resources and high churn: resource discovery, robust and efficient sensing and actuation, management bottleneck, and scale. We present results of our research on addressing the above challenges. Niche implements the autonomic computing architecture, proposed by IBM, in a fully decentralized way. Niche supports a network-transparent view of the system architecture simplifying the design of distributed self-management. Niche provides a concise and expressive API for self-management. The implementation of the platform relies on the scalability and robustness of structured overlay networks. We proceed by presenting a methodology for designing the management part of a distributed self-managing application. We define design steps that include partitioning of management functions and orchestration of multiple autonomic managers. In the second part, we discuss robustness of management and data consistency, which are necessary in a distributed system. Dealing with the effect of churn on management increases the complexity of the management logic and thus makes its development time consuming and error prone. We propose the abstraction of Robust Management Elements, which are able to heal themselves under continuous churn. Our approach is based on replicating a management element using finite state machine replication with a reconfigurable replica set. Our algorithm automates the reconfiguration (migration) of the replica set in order to tolerate continuous churn. For data consistency, we propose a majority-based distributed key-value store supporting multiple consistency levels that is based on a peer-to-peer network. The store enables the tradeoff between high availability and data consistency. Using majority allows avoiding potential drawbacks of a master-based consistency control, namely, a single-point of failure and a potential performance bottleneck. In the third part, we investigate self-management for Cloud-based storage systems with the focus on elasticity control using elements of control theory and machine learning. We have conducted research on a number of different designs of an elasticity controller, including a State-Space feedback controller and a controller that combines feedback and feedforward control. We describe our experience in designing an elasticity controller for a Cloud-based key-value store using state-space model that enables to trade-off performance for cost. We describe the steps in designing an elasticity controller. We continue by presenting the design and evaluation of ElastMan, an elasticity controller for Cloud-based elastic key-value stores that combines feedforward and feedback control

Self-Organized Routing For Wireless Micro-Sensor Networks

In this paper we develop an energy-aware self-organized routing algorithm for the networking of simple battery-powered wireless micro-sensors (as found, for example, in security or environmental monitoring applications). In these networks, the battery life of individual sensors is typically limited by the power required to transmit their data to a receiver or sink. Thus effective network routing algorithms allow us to reduce this power and extend both the lifetime and the coverage of the sensor network as a whole. However, implementing such routing algorithms with a centralized controller is undesirable due to the physical distribution of the sensors, their limited localization ability and the dynamic nature of such networks (given that sensors may fail, move or be added at any time and the communication links between sensors are subject to noise and interference). Against this background, we present a distributed mechanism that enables individual sensors to follow locally selfish strategies, which, in turn, result in the self-organization of a routing network with desirable global properties. We show that our mechanism performs close to the optimal solution (as computed by a centralized optimizer), it deals adaptively with changing sensor numbers and topology, and it extends the useful life of the network by a factor of three over the traditional approach

Self-Organizing Hierarchical Knowledge Discovery by an ARTMAP Image Fusion System

Classifying novel terrain or objects front sparse, complex data may require the resolution of conflicting information from sensors working at different times, locations, and scales, and from sources with different goals and situations. Information fusion methods can help resolve inconsistencies, as when evidence variously suggests that an object's class is car, truck, or airplane. The methods described here consider a complementary problem, supposing that information from sensors and experts is reliable though inconsistent, as when evidence suggests that an object's class is car, vehicle, and man-made. Underlying relationships among objects are assumed to be unknown to the automated system or the human user. The ARTMAP information fusion system used distributed code representations that exploit the neural network's capacity for one-to-many learning in order to produce self-organizing expert systems that discover hierarchical knowledge structures. The system infers multi-level relationships among groups of output classes, without any supervised labeling of these relationships.Air Force Office of Scientific Research (F49620-01-1-0397, AFOSR F49620-01-1-0423); Office of Naval Research (N00014-01-1-0624); National Imagery and Mapping Agency and the National Science Foundation for Siegfried Martens (NMA501-03-1-2030, DGE-0221680); Department of Homeland Securit

Self-Organizing Information Fusion and Hierarchical Knowledge Discovery: A New Framework Using Artmap Neural Networks

Classifying novel terrain or objects from sparse, complex data may require the resolution of conflicting information from sensors woring at different times, locations, and scales, and from sources with different goals and situations. Information fusion methods can help resolve inconsistencies, as when eveidence variously suggests that and object's class is car, truck, or airplane. The methods described her address a complementary problem, supposing that information from sensors and experts is reliable though inconsistent, as when evidence suggests that an object's class is car, vehicle, and man-made. Underlying relationships among classes are assumed to be unknown to the autonomated system or the human user. The ARTMAP information fusion system uses distributed code representations that exploit the neural network's capacity for one-to-many learning in order to produce self-organizing expert systems that discover hierachical knowlege structures. The fusion system infers multi-level relationships among groups of output classes, without any supervised labeling of these relationships. The procedure is illustrated with two image examples, but is not limited to image domain.Air Force Office of Scientific Research (F49620-01-1-0423); National Geospatial-Intelligence Agency (NMA 201-01-1-2016, NMA 501-03-1-2030); National Science Foundation (SBE-0354378, DGE-0221680); Office of Naval Research (N00014-01-1-0624); Department of Homeland Securit

Self-Stabilization in the Distributed Systems of Finite State Machines

The notion of self-stabilization was first proposed by Dijkstra in 1974 in his classic paper. The paper defines a system as self-stabilizing if, starting at any, possibly illegitimate, state the system can automatically adjust itself to eventually converge to a legitimate state in finite amount of time and once in a legitimate state it will remain so unless it incurs a subsequent transient fault. Dijkstra limited his attention to a ring of finite-state machines and provided its solution for self-stabilization. In the years following his introduction, very few papers were published in this area. Once his proposal was recognized as a milestone in work on fault tolerance, the notion propagated among the researchers rapidly and many researchers in the distributed systems diverted their attention to it. The investigation and use of self-stabilization as an approach to fault-tolerant behavior under a model of transient failures for distributed systems is now undergoing a renaissance. A good number of works pertaining to self-stabilization in the distributed systems were proposed in the yesteryears most of which are very recent. This report surveys all previous works available in the literature of self-stabilizing systems

Self-Configuring Socio-Technical Systems: Redesign at Runtime

Modern information systems are becoming more and more socio-technical systems, namely systems composed of human (social) agents and software (technical) systems operating together in a common environment. The structure of such systems has to evolve dynamically in response to the changes of the environment. When new requirements are introduced, when an actor leaves the system or when a new actor comes, the socio-technical structure needs to be redesigned and revised. In this paper, an approach to dynamic reconfiguration of a socio-technical system structure in response to internal or external changes is proposed. The approach is based on planning techniques for generating possible alternative configurations, and local strategies for their evaluation. The reconfiguration mechanism is presented, which makes the socio-technical system self-configuring, and the approach is discussed and analyzed on a simple case study

Self-concatenated code design and its application in power-efficient cooperative communications

In this tutorial, we have focused on the design of binary self-concatenated coding schemes with the help of EXtrinsic Information Transfer (EXIT) charts and Union bound analysis. The design methodology of future iteratively decoded self-concatenated aided cooperative communication schemes is presented. In doing so, we will identify the most important milestones in the area of channel coding, concatenated coding schemes and cooperative communication systems till date and suggest future research directions