Kafka versus RabbitMQ

Publish/subscribe is a distributed interaction paradigm well adapted to the deployment of scalable and loosely coupled systems. Apache Kafka and RabbitMQ are two popular open-source and commercially-supported pub/sub systems that have been around for almost a decade and have seen wide adoption. Given the popularity of these two systems and the fact that both are branded as pub/sub systems, two frequently asked questions in the relevant online forums are: how do they compare against each other and which one to use? In this paper, we frame the arguments in a holistic approach by establishing a common comparison framework based on the core functionalities of pub/sub systems. Using this framework, we then venture into a qualitative and quantitative (i.e. empirical) comparison of the common features of the two systems. Additionally, we also highlight the distinct features that each of these systems has. After enumerating a set of use cases that are best suited for RabbitMQ or Kafka, we try to guide the reader through a determination table to choose the best architecture given his/her particular set of requirements.Comment: 25 single-column pages, 7 figures, 5 table

Palvelun skaalaus moniasiakasympäristössä

Opinnäytetyössä toteutettiin jyväskyläläiselle ohjelmistoalalalla toimivalle Codecenter Oy:lle Java-ohjelmointikielellä skaalautuva muunnospalvelu moniasiakasympäristöön, jolla muunnettiin XML-tiedostoja eri muotoihin. Palvelun toteutuksessa hyödynnettiin VMwaren vFabricalustaa sekä avoimen lähdekoodin ratkaisuja, joista tärkein oli Spring Framework. Toteutettu palvelu koostui erillisestä hallintasovelluksesta ja viestikäsittelijästä. Työssä vertailtiin vertikaalista ja horisontaalista skaalausta keskenään teoriatasolla. Muunnospalvelu toteutettiin noudattaen horisontaalisen skaalauksen periaatteita. Työssä käsiteltiin muistinvaraisten tietokantojen ja viestijonojen käyttämistä osana skaalausratkaisuja. Muistinvaraisena tietokantana käytettiin vFabric GemFirea, joka mahdollisti useita eri lähestysmistapoja muistinvaraisen kannan muodostamiseen. Viestijonototeutuksena taas käytettiin RabbitMQ:ta. Muunnosten tekeminen suoritettiin Apache ServiceMix -alustalle toteutettua käsittelijäkomponenttia käyttäen. Tällöin mahdollistettiin muunnospalvelun joustava integroiminen muiden järjestelmien kanssa. ServiceMix ja sen kanssa käytetty Apache Camel tarjosivat yhdessä mittavan joukon erilaisia työkaluja integraatioita varten. Tutkimustuloksena saatiin tietoa horisontaalisen skaalauksen soveltamisesta oikean sovelluksen toteutuksessa sekä tietoa vFabric-alustan käytöstä. Horisontaalisen skaalauksen keskeisimpänä periaatteena esitelty useisiin osiin jaottelu osoittautui käytännössä hyvin toimivaksi ratkaisuksi. vFabric-alusta tarjosi hyvät työkalut skaalautuvan palvelun toteuttamiseen.This bachelor's thesis describes the implementation process of a scalable transformation service given as an assignment by an IT company called Codecenter Oy. The service was aimed at a multiuser environment. The transformation service was used for transforming XML files into other formats. VMware's vFabric platform was used for implementing scalability. Other open source tools were also used, including Spring Framework. Vertical and horizontal scaling were compared; but the main focus, however, was on horizontal scaling. The use of an in-memory database and message queues were the two most important concepts. vFabric GemFire was used as an implementation for an inmemory database and RabbitMQ as a messaging queue implementation. Apache ServiceMix and Apache Camel were used together for providing the best way of performing transformations. They also provided a number of tools for integrating the service with other systems. The thesis demonstrates how to apply the theory of horizontal scaling in practice. The results were promising and prove how splitting was the best way to implement horizontal scaling. vFabric platform was the most important solutions provider for scalability

DISCO: Distributed Multi-domain SDN Controllers

Modern multi-domain networks now span over datacenter networks, enterprise networks, customer sites and mobile entities. Such networks are critical and, thus, must be resilient, scalable and easily extensible. The emergence of Software-Defined Networking (SDN) protocols, which enables to decouple the data plane from the control plane and dynamically program the network, opens up new ways to architect such networks. In this paper, we propose DISCO, an open and extensible DIstributed SDN COntrol plane able to cope with the distributed and heterogeneous nature of modern overlay networks and wide area networks. DISCO controllers manage their own network domain and communicate with each others to provide end-to-end network services. This communication is based on a unique lightweight and highly manageable control channel used by agents to self-adaptively share aggregated network-wide information. We implemented DISCO on top of the Floodlight OpenFlow controller and the AMQP protocol. We demonstrated how DISCO's control plane dynamically adapts to heterogeneous network topologies while being resilient enough to survive to disruptions and attacks and providing classic functionalities such as end-point migration and network-wide traffic engineering. The experimentation results we present are organized around three use cases: inter-domain topology disruption, end-to-end priority service request and virtual machine migration

QoS-enabled middleware for smart grids

Emerging smart grid systems must be able to react quickly and predictably, adapting their operation to changing energy supply and demand, by controlling energy consuming and energy storage devices. An intrinsic problem with smart grids is that energy produced from in-house renewable sources is affected by fluctuating weather factors. The applications driving smart grids operation must rely on a solid communication network that is secure, highly scalable, and always available. Thus, any communication infrastructure for smart grids should support its potential of producing high quantities of real-time data, with the goal of reacting to state changes by actuating on devices in real-time, while providing Quality of Service (QoS)

Analysis of Communications Protocols for Future Networks

Import 03/11/2016Tato bakalářská práce je věnována komunikačním protokolům budoucnosti, které se zaměřují a přizpůsobují v současné době modernímu a stále se rozšiřujícímu trendu Internetu věcí. Snaží se zasvětit do problematiky Internetu věcí a objasnit z jakého důvodu je nutné zkoumat komunikační protokoly k tomu určené. Popisuje srovnání aktuálních a nejčastěji používaných komunikačních protokolů, jejich technologii a princip činnosti, včetně možnosti jejich využití. Cílem této práce je zanalyzovat vybrané protokoly, porovnat jejich implementaci, podrobit je zátěžovým testům a z výsledných měření pomocí nasbíraných dat vyhodnotit, který z protokolů lze určit, jako perspektivní pro budoucí sítě.This thesis is devoted to the communication protocols of the future that are focused and adapted to the present, modern and ever-increasing trend of Internet of Things. It tries to devote to the issue of Internet of things and explain why it is necessary to examine the communications protocols designated for this purpose. It describes a comparison of current and most commonly used communication protocols, their technology and principle of operation, including the possibility of their use. The aim of this work is to analyze the specific protocols, compare their implementation, subjecting them to stress tests and the resulting measurements using collected data to assess which of the protocols can be identified as promising for the future network.460 - Katedra informatikyvelmi dobř

R-MOM: A Component-Based Framework for Interoperable and Adaptive Asynchronous Middleware Systems

International audienceSystems of systems (SoS) are composed of subsystems such as Distributed, Information Technology, Real-Time and Embedded systems. Among distributed systems, Message- Oriented Middleware (MOM) is used by SoS in order to share status information from system elements (component, service, etc.). Often several different MOM technologies are used in one SoS, then interoperability between these MOM is a requirement. In this paper, we present R-MOM, a component-based framework for interoperable and adaptive asynchronous middleware systems. R-MOM provides a reflective component architecture where one MOM functionality is embedded into one component which is modifiable at run-time. Loosely-coupling between reflective components permits to get a fined-personalization of MOM functionalities, such as protocol, encoding rule, Quality of Services (QoS) processing, data production/consumption, description, routing and filtering. Interoperability between integrated protocol functionalities is a consequence of architecture design. R-MOM interoperates with different kinds of MOM, from distributed message queues (Java Message Service, Advanced Message Queueing Protocol, 0MQ) to content-based publish/ subscribe systems (OMG's Data Distribution Service). This paper describes the architectural concepts of the R-MOM framework, discusses its implementation, and evaluates its interoperability capability.Les Systèmes de Systèmes (SoS) sont composés de sous-systèmes tels que les systèmes de type distribué, IT, temps-réel et embarqué. Parmi les systèmes distribués, les intergiciels orientés message (MOM) sont utilisés par les SoS dans le but de partager des informations de statut des éléments du systèmes (composant, service, etc...). Souvent, plusieurs technologies MOM sont utilisées dans un même SoS, et interopérabilité entre MOM devient nécessaire. Dans ce papier, nous présentons R-MOM, un canevas basé sur l'approche composant pour les intergiciels interopérables, adaptatifs et asynchrones. R-MOM offre une architecture à composants réflexif, où une fonctionalité d'un MOM est embarquée dans un composant qui est modifiable durant son exécution. Le couplage lâche entre les composants réflexif permet de jouir d'une personnalisation fine sur les fonctionalités MOM visées, comme le protocole, la règle d'encodage, le support des qualités de service, la production et consommation, la description, le routage et le filtrage des données. L'interopérabilité entre les fonctionnalités protocolaire intégrées est une conséquence du choix de conception de l'architecture. R-MOM est interopérable avec différent types de MOM, depuis les queues de message distribué (Java Message Service, Advanced Message Queueing Protocol, 0MQ) jusqu'aux systèmes de publication/souscription basées sur le contenu (Service de Distribution de Données de l'OMG). Ce papier papier décrit les concepts d'architecture du canevas R-MOM, discute son implémentation et évalue sa capacité d'interopérabilité