A cloud queuing service with strong consistency and high availability,”

Message queues are widely used to connect loosely coupled components to form large-scale, highly distributed, and fault-tolerant applications. A number of vendors currently provide cloud-based queuing services that are designed to be elastic, scalable, and easy to use. However, unlike enterprise queuing systems, which provide strong queuing consistency and are suitable for many traditional enterprise workloads, these cloud-based queuing services offer reduced queuing consistency. They can deliver messages without loss, but they may deliver messages out of order or with duplications. This paper presents SilverDove Queuing Service (SDQS), a scalable cloud-based queuing service with stronger consistency than existing cloud-based queuing services. SDQS delivers messages without losses or duplications and provides in-order message delivery as an option. Built on top of IBM WebSphere A eXtreme Scale, i.e., an elastic in-memory object grid system, SDQS can be easily scaled up and down to accommodate a wide range of workloads. SDQS is able to provide high availability with either the no-order or the in-order message delivery option. We have performed a preliminary evaluation of SDQS with up to 70 nodes on a compute cloud platform, verifying its consistency offerings and providing insights into the tradeoff between performance and consistency. Introduction As computing reaches every corner of people's daily lives, including business informatics, personal entertainment, and real-world event sensing, the scale and complexity of software systems grow exponentially. Coming with this trend is the critical challenge to provide reliable, efficient, and flexible mechanisms for large-scale distributed computing components or applications to communicate with one another. One of the most widely adopted interapplication communication patterns is asynchronous message exchanges through message queues. A message queue decouples the two communicating parties from each other. It provides temporary storage when the destination application is busy or suffering from poor connectivity. It reduces the involvement of application developers with the complexity of handling the communication mechanisms. It also simplifies the development, deployment, and management of complex distributed applications that span multiple heterogeneous operating systems and network protocols. Exemplary applications that can benefit from message queues include workload dispatching/load balancing, MapReduce [1]-like pipelined processing, distributed workflow management, and information aggregation/dissemination, to name just a few. In addition to being used for asynchronous message exchange, message queues may also be used to support synchronous request-response communication patterns that are common in traditional enterprise application integration. A typical usage scenario is the supporting of Simple Object Access Protocol over Java** Message Service. With the advent of advanced virtualization technology, many enterprises are adopting cloud computing to reduce their capital and operational expenditure. Along with the trend comes the imperative to provide message queuing (MQ) as a common cloud service that can be consumed by Copyright 2011 by International Business Machines Corporation. Copying in printed form for private use is permitted without payment of royalty provided that (1) each reproduction is done without alteration and (2) the Journal reference and IBM copyright notice are included on the first page. The title and abstract, but no other portions, of this paper may be copied by any means or distributed royalty free without further permission by computer-based and other information-service systems. Permission to republish any other portion of this paper must be obtained from the Editor. multiple tenant organizations and applications. In order to fully realize the benefit of cloud computing and to accommodate the rapidly growing and highly dynamic workloads from today's data-intensive applications, a cloud-based MQ service should be highly reliable and highly elastic while providing good performance. A common approach adopted by many cloud-based queuing service providers focuses on lower cost, elasticity, and availability as the main design objectives while relaxing queuing semantics. Examples include Amazon's Simple Queue Service (SQS) On the other hand, traditional enterprise-class MQ systems, such as IBM WebSphere* MQ and Microsoft MSMQ, and Apache ActiveMQ**, offer excellent system performance and various other desirable features such as exactly once delivery, FIFO message ordering, and distributed transaction support. These messaging middlewares play an important role in applications that cannot tolerate message duplication or require strict FIFO delivery order. Many financial applications fall into this category. To enable these types of applications to be eventually migrated to a cloud environment and thus realize the cloud computing benefits, it is important to provide a highly available and scalable queuing service with strict queuing consistency. We note that with the strict queuing consistency offerings comes the inevitable tradeoffs that one would have to make in other metrics, such as performance or availability. Therefore, it is also highly desirable for a queuing service to provide applications with options for selecting only the necessary queuing consistency level so that the service can maximize the level of other nonfunctional qualities. Toward this end, this paper presents the design and evaluation of a cloud-based queuing service code-named SilverDove Queuing Service (SDQS). SDQS is built on top of IBM WebSphere eXtreme Scale [4], which is an elastic in-memory data grid system. Although WebSphere eXtreme Scale provides high storage consistency, it is a challenging task to provide high queuing consistency on top of it. SDQS overcomes the challenge by using a queue indexing approach and a visibility-timeout mechanism. With no failures in network transport and client applications, SDQS is able to provide exactly once message delivery, i.e., no loss or duplication, while offering two kinds of message order options to end users, which can be selected when establishing queues in the system. This represents stronger consistency than existing cloud-based offerings, approaching that of a traditional enterprise queuing product. Additionally, SDQS provides high availability and elasticity. We point out that end-to-end strong consistencies under all failure conditions are not currently provided by SDQS. Please refer to the ensuing sections for more detailed discussions. The main contribution of this paper is twofold. First, it presents a cloud-based scalable queuing service, which to our best knowledge is the first in its kind capable of providing exactly once message delivery in FIFO order. Second, through evaluation, it quantitatively compares SDQS to other representative queuing services and validates its queuing consistency and high availability. We note that SDQS is a research prototype designed to investigate the possibility of providing strong queuing consistency with high availability. It does not represent the official product plan or roadmap of IBM in the area of cloud-based messaging. The remainder of this paper is organized as follows. Background knowledge and related work are introduced in the next section, which is followed by a description of the system architecture and design. The following section provides the details of the implementation, evaluation methodology, and results. The final section concludes this paper. Background and related wor

UniquID: A Quest to Reconcile Identity Access Management and the Internet of Things

The Internet of Things (IoT) has caused a revolutionary paradigm shift in computer networking. After decades of human-centered routines, where devices were merely tools that enabled human beings to authenticate themselves and perform activities, we are now dealing with a device-centered paradigm: the devices themselves are actors, not just tools for people. Conventional identity access management (IAM) frameworks were not designed to handle the challenges of IoT. Trying to use traditional IAM systems to reconcile heterogeneous devices and complex federations of online services (e.g., IoT sensors and cloud computing solutions) adds a cumbersome architectural layer that can become hard to maintain and act as a single point of failure. In this paper, we propose UniquID, a blockchain-based solution that overcomes the need for centralized IAM architectures while providing scalability and robustness. We also present the experimental results of a proof-of-concept UniquID enrolment network, and we discuss two different use-cases that show the considerable value of a blockchain-based IAM.Comment: 15 pages, 10 figure

BaseFs - Basically Acailable, Soft State, Eventually Consistent Filesystem for Cluster Management

A peer-to-peer distributed filesystem for community cloud management. https://github.com/glic3rinu/basef

Platforms and Protocols for the Internet of Things

Building a general architecture for the Internet of Things (IoT) is a very complex task, exacerbated by the extremely large variety of devices, link layer technologies, and services that may be involved in such a system. In this paper, we identify the main blocks of a generic IoT architecture, describing their features and requirements, and analyze the most common approaches proposed in the literature for each block. In particular, we compare three of the most important communication technologies for IoT purposes, i.e., REST, MQTT, and AMQP, and we also analyze three IoT platforms: openHAB, Sentilo, and Parse. The analysis will prove the importance of adopting an integrated approach that jointly addresses several issues and is able to flexibly accommodate the requirements of the various elements of the system. We also discuss a use case which illustrates the design challenges and the choices to make when selecting which protocols and technologies to use

Datacenter Traffic Control: Understanding Techniques and Trade-offs

Datacenters provide cost-effective and flexible access to scalable compute and storage resources necessary for today's cloud computing needs. A typical datacenter is made up of thousands of servers connected with a large network and usually managed by one operator. To provide quality access to the variety of applications and services hosted on datacenters and maximize performance, it deems necessary to use datacenter networks effectively and efficiently. Datacenter traffic is often a mix of several classes with different priorities and requirements. This includes user-generated interactive traffic, traffic with deadlines, and long-running traffic. To this end, custom transport protocols and traffic management techniques have been developed to improve datacenter network performance. In this tutorial paper, we review the general architecture of datacenter networks, various topologies proposed for them, their traffic properties, general traffic control challenges in datacenters and general traffic control objectives. The purpose of this paper is to bring out the important characteristics of traffic control in datacenters and not to survey all existing solutions (as it is virtually impossible due to massive body of existing research). We hope to provide readers with a wide range of options and factors while considering a variety of traffic control mechanisms. We discuss various characteristics of datacenter traffic control including management schemes, transmission control, traffic shaping, prioritization, load balancing, multipathing, and traffic scheduling. Next, we point to several open challenges as well as new and interesting networking paradigms. At the end of this paper, we briefly review inter-datacenter networks that connect geographically dispersed datacenters which have been receiving increasing attention recently and pose interesting and novel research problems.Comment: Accepted for Publication in IEEE Communications Surveys and Tutorial

Survey and Analysis of Production Distributed Computing Infrastructures

This report has two objectives. First, we describe a set of the production distributed infrastructures currently available, so that the reader has a basic understanding of them. This includes explaining why each infrastructure was created and made available and how it has succeeded and failed. The set is not complete, but we believe it is representative. Second, we describe the infrastructures in terms of their use, which is a combination of how they were designed to be used and how users have found ways to use them. Applications are often designed and created with specific infrastructures in mind, with both an appreciation of the existing capabilities provided by those infrastructures and an anticipation of their future capabilities. Here, the infrastructures we discuss were often designed and created with specific applications in mind, or at least specific types of applications. The reader should understand how the interplay between the infrastructure providers and the users leads to such usages, which we call usage modalities. These usage modalities are really abstractions that exist between the infrastructures and the applications; they influence the infrastructures by representing the applications, and they influence the ap- plications by representing the infrastructures