Both researchers and industry players are facing the same obstacles when entering the big data field. Deploying and testing distributed data technologies requires a big up-front investment of both time and knowledge. Existing cloud automation solutions are not well suited for managing complex distributed data solutions. This paper proposes a distributed service orchestration architecture to better handle the complex orchestration logic needed in these cases. A novel service-engine based approach is proposed to cope with the versatility of the individual components. A hybrid integration approach bridges the gap between cloud modeling languages, automation artifacts, image-based schedulers and PaaS solutions. This approach is integrated in the distributed data experimentation platform Tengu, making it more flexible and robust

De Turck, Filip

Sebrechts, Merlijn

Van Seghbroeck, Gregory

Vanhove, Thomas

Volckaert, Bruno

Wauters, Tim

English

Merlijn Sebrechts

Thomas Vanhove

Gregory Van Seghbroeck

Tim Wauters

Bruno Volckaert

Filip De Turck

Crossref

Distributed Service Orchestration: Eventually Consistent Cloud Operation and Integration

Ghent University Academic Bibliography

Distributed Service Orchestration: Eventually Consistent Cloud Operation andIntegrationMerlijn Sebrechts, Thomas Vanhove, Gregory Van Seghbroeck, Tim Wauters, Bruno Volckaert and Filip De TurckDepartment of Information Technology, Ghent University - iMindsTech Lane Ghent Science Park Campus A, 15, 9052 Ghent, BelgiumEmail: merlijn.sebrechts@intec.ugent.beAbstract—Both researchers and industry players are facingthe same obstacles when entering the big data field. Deployingand testing distributed data technologies requires a big up-front investment of both time and knowledge. Existing cloudautomation solutions are not well suited for managing complexdistributed data solutions. This paper proposes a distributedservice orchestration architecture to better handle the complexorchestration logic needed in these cases. A novel service-engine based approach is proposed to cope with the versatilityof the individual components. A hybrid integration approachbridges the gap between cloud modeling languages, automationartifacts, image-based schedulers and PaaS solutions. Thisapproach is integrated in the distributed data experimentationplatform Tengu, making it more flexible and robust.I. INTRODUCTIONBig data is a fast-moving research field that has resulted inan explosion of distributed data technologies. The distributednature and the complexity of these data technologies createsa lot of operational overhead. Deploying, managing andmonitoring big data solutions is a very time-consuming andknowledge-intensive task. Hiring the right people to managesuch an infrastructure is hard because of the data skills short-age [1][2]. The DevOps movement has spawned a number oftools that help reduce the operational overhead and empowersmall teams to manage complex cloud infrastructure.There are a number of operational challenges specificto distributed data technologies that are not addressed inthe state of the art cloud automation tools. The first issuelies in the orchestration of multiple components. The cen-tralized and monolithic nature of most orchestration toolscannot handle the complex orchestration logic needed fordeploying and managing distributed technologies. The sec-ond challenge resides in the management of an individualcomponent. The current state-machine based approach forautomating the management of an individual component isnot flexible enough to meet the needs of adaptable services.The last issue addressed in this paper lies in the integrationbetween model-based orchestrators and existing automationartifacts. There is a need for integrating a variety of existingPaaS and SaaS solutions, configuration management toolsand model-based orchestrators.This paper further explores these three challenges andpropose a novel architecture to tackle them. Section IIdescribes where existing state of the art techniques fall short.The distributed service orchestrator is explained in sectionIII. Section IV goes into details about the Service Agentand Service Engine. Section V proposes the novel hybridintegration approach. Finally, Section VI reflects on the workand gives a peek into future work.II. STATE OF THE ARTA. OrchestrationWith IT infrastructure growing ever-more complex, thedependencies between individual components are becominghard to manage. One of the key properties of an orchestratoris that it enables isolation of components and autonomicmanagement of dependencies between these components.The conventional way of orchestrating large clouds includesa central orchestrator such as seen in the OASIS standardTopology and Orchestration Specification for Cloud Appli-cations (TOSCA) [3]. Research at Google has shown thatorchestrating and scheduling a datacenter from a centralmonolith becomes too complex to maintain [4]. The Omegaand Mesos [5] meta-schedulers are proposed as a solutionfor this complexity, implementing a decentralized schedulingapproach. Their orchestrating capabilities, however, are notas extensive as needed for cloud modeling languages suchas TOSCA or Ubuntu Juju1. The decentralized orchestratorproposed in this paper solves the complexity issues of acentralized approach while still maintaining the full featuresof a cloud modeling language.B. State machines and lifecyclesIndividual components in TOSCA are described using alifecycle described by a state machine [6]. This approachhas the disadvantage that it is quite inflexible becauseall possible states and transitions have to be known atdesign time. Moreover, the state machine of a componentcan become very complex when a component can have anumber of different relationships with other components.Each possible combination of those relationships needs itsown state. The amount of states needed to model sucha tool grows exponentially in function of the number of1https://jujucharms.com/possibilities. This is especially relevant in the distributeddata field where components such as an Extract, Transform,Load (ETL) tool can have a number of different relationshipsto a multitude of data sources and to many different datastores. The service engine proposed in this paper addressesthe complexity problem of state machines.C. IntegrationIntegration between model-driven cloud management andconfiguration management is key to automate every aspectof the management of a service [7]. Wettinger et al. proposetwo approaches to integrate model-driven cloud manage-ment and configuration management: direct integration andtransparent integration. For direct integration, the serviceorchestrator has to have knowledge about the particularconfiguration management tool. Transparent integration canbe accomplished using generic ‘script’ artifacts that trans-late generic calls of the orchestrator into configuration-management-specific calls. Wettinger et al. note that al-though the transparent integration approach is more portable,the direct integration makes it possible to fully leverage allfeatures of the configuration management tool. To get thefull benefits of both approaches, Wettinger et al. propose acombined approach. This however has the downside that itrequires some code duplication and that it requires rigoroustesting to ensure that both methods will yield the same result.This paper proposes a hybrid approach that combines thebest of direct and transparent integration without the addedcomplexity and duplicated code.III. DISTRIBUTED ORCHESTRATIONTo tackle the issues encountered in centralized, monolithicorchestrators, this paper proposes a novel decentralizedservice orchestration architecture. This architecture enablesvery complex orchestration logic without the complexity of amonolith. Moreover, this decentralized approach also tack-les the scalability and fault-tolerance issues of centralizedService orchestration layerService KBService engineConfiguration managerServiceService AgentService KBService engineConfiguration managerServiceContainerFigure 1. Decentralized service orchestration architecture.orchestrators. The orchestration layer consists of a numberof service agents and a number of peer-to-peer relationshipsbetween them as shown in Figure 1. Each service agent isresponsible for a particular service. Using the relationships,service agents communicate to achieve higher-level goals.An example is a Apache Hadoop2 cluster. The namenode,datanode and slaves each have their own service agent. Theservice agents communicate with each other to setup andconfigure the Apache Hadoop cluster.Service agents operate event-driven and transactional, re-sulting in an eventually consistent model [8]. Service agentscommunicate by sending relationship events to each other.Each relationship event updates the knowledgebase sharedbetween the two services. When a service agent receives arelationship event, it queues the event on a FIFO queue.One by one, each relationship event gets applied to theknowledgebase and is processed by the service engine in atransactional manner. The service engine can call a config-uration manager to update the service or send events to thepeers of the service agent. The event-driven approach allowsfor complex dependencies. Service agents can negotiate witheach other on topics such as how the services should beconfigured and what libraries should be installed.IV. THE SERVICE ENGINEThe actual management of the service and the communi-cation with other service agents happens in the service en-gine. In a conventional configuration management approach,the operator requests a certain state and the configurationmanager takes the correct actions to get into that state. Thisproactive approach requires the desired state to be known be-forehand. In the presented approach, actions handle changesin the state of the service and its surroundings. The serviceagent monitors changes and executes the required actions(handlers) to react to those changes. Each handler modifiesa small part of the service state, causing other handlers to beexecuted. When no more handlers are executed, the servicehas reached a consistent state, which causes the service agentto go back into monitoring mode until it detects anotherchange. When a handler fails, a rollback of the transactionis issued and the service engine retries the processing ofthe relationship event. This convergence-based approach, asproposed by Wettinger et al. [9] eliminates the need foridempotency and convergence, and is more fault tolerant.The state of the service is represented as a set of facts ina local knowledge base. The heart of the service agent is theservice engine. The service engine contains a set of handlersand decides which handlers should be executed based onthe facts in the knowledge base and the preconditions of thehandlers. A handler can add facts to the knowledge base,communicate with related service agents and change theservice state. There are four types of preconditions that a2https://hadoop.apache.org/...ConfigurationManager1.3.4.1. execute handler2. call config manager3. update knowledgebase4. retest queue5. queue handlers2.4. 5....KnowledgebaseFigure 2. A service engine run. After the handler updates the knowl-edgebase, the queue is retested and handlers with false preconditions areremoved. Handlers that react to events are automatically removed from thequeue after execution. Handlers whose preconditions are met, are addedto the queue. A handler that has an event precondition that is not active(greyed out) cannot be added to the queue.handler can specify: a relationship fact, a relationship event,an internal fact and an internal event.The difference between events and facts is that an eventis only relevant during a single moment in time while a factremains true until a handler changes it. An example of anevent is a file change. When a file changes, all handlers thatreact to that event get queued to execute. After all events arequeued, the event becomes inactive again. An example of afact is the state of a package: installed or uninstalled. Thismeans that if a handler reacts to two facts and one event,it will only get queued if both facts are true at the momentthe event is fired. Queued handlers are retested after eachhandler run to ensure that their preconditions based on factsare still true as seen in Figure 2. Preconditions based onevents are not retested.The difference between internal and relationship is thatrelationship events are processed as a transaction. When arelationship change or event happens, the service enginestarts a run. Relationship changes that happen during such arun get queued until the run completes. From the outside per-spective, this run happens during a single moment in time.This means that the relationship information is guaranteedto be consistent, but not current. However, the queuing ofrelationship events guarantees eventual consistency over allservice agents.This eventually consistent model works great if the goalis to let the infrastructure converge to a globally consistentstate. Where this falls short is when the actions of a serviceagent depend on the real-time current state of another serviceagent. An example of this would be a inter-cluster data copybetween two Hadoop clusters using distcp. For this tool towork, both clusters need to be ready when the distcp isstarted. Since the state a service agent sees is not necessarilythe current state, but could have been a current state in thepast, a service agent cannot reliably orchestrate the startof a distcp. Future work will expand this architecture withdistributed locking capabilities to tackle this challenge.V. HYBRID INTEGRATIONThe distributed orchestrator allows for a novel approach tointegrate cloud modeling languages, configuration manage-ment tools, image-based orchestrators and PaaS solutions.This approach can be called hybrid integration since itcombines the best of direct- and transparent integrationwithout the downside of duplicated code and without theadded complexity. This solution is portable, yet able to fullyleverage advanced features of the configuration managementtool. Service agents communicate using generic language-independent interfaces, making the implementation of a ser-vice agent transparent to other service agents. The internalsof a service agent, namely a service engine, can howeverdirectly integrate into configuration management tools. Thisallows them to leverage the full capabilities of the specifictool. Existing automation artifacts can be easily encapsulatedin a service agent allowing for reuse of the expertise inexisting code. Figure 3 shows an example of a Chef3 scriptencapsulated in a service agent. This service agent cancommunicate with other service agents independent of theunderlying configuration management tool.This approach is tested by applying this architecture toTengu, a distributed data experimentation platform [10].3https://www.chef.io/Chef ZeroPython ScriptKafkaWSO² ESBService engine Service engineService orchestration layer3. set IP and port     Chef Databag1. get IP and port2. send IP and port4. retrieve IP and port5. update serviceFigure 3. The distributed service orchestration layer handles the com-munication between service agents. The service engine translates thereceived information into a format the Chef Zero configuration managercan read: a Chef Databag. Chef then updates service configuration to enablecommunication from the ESB to Kafka.Tengu is built to give researchers an easy way to spin updistributed data environments. The first version of Tenguwas built using the configuration management tool Chef.The complexity of the dependencies between individualservices caused Tengu to be very inflexible. Adding a serviceorchestration layer to Tengu, turned it into a flexible andcomposable big data experimentation platform4.The hybrid integration approach also allows for integra-tion with image-based deployments such as containers andvirtual machines, cloud schedulers, PaaS and SaaS solutions.The biggest drawback of image-based deployment is the lackof flexibility. This can be addressed by creating a serviceagent that deploys a prebuilt image. After the deployment,the service engine changes the configuration of the deployedimage to accommodate the needs of related service agents.When a related service agent needs information about thedeployed service, the service engine uses introspection togather that information from the image. Integration withPaaS and SaaS solutions is similarly straightforward. Theservice agent can communicate to a PaaS or SaaS solutionusing an API. A service agent that has an API client asback-end can be interchanged with a service agent thathas a configuration management tool as back-end if bothservice agents implement the same relationship interface.This allows users to choose the solution that fits their use-case best, be it self hosted, SaaS or PaaS. As an additionalbonus, this further limits vendor lock-in problems.VI. CONCLUSION AND FUTURE WORKThe distributed service orchestrator presented in this pa-per tackles three problems. Its distributed nature reducesits complexity while maintaining the ability to orchestratecomplex distributed interdependent services. The serviceengines allow for orchestrating very versatile service agentswithout the need for complex state machines. Lastly, thehybrid integration makes deep integration possible betweenthe cloud modeling language and configuration managementtools, image based deployers and PaaS solutions. Futurework will further expand on and evaluate this architecture.This architecture will be extended with multi-tenant capa-bilities, a distributed monitoring framework, distributed rea-soning about higher-level objectives, and real-time lockingof resources.ACKNOWLEDGMENTThe research described in this paper is partially fundedby the IWT/VLAIO agency through the DeCoMAdS projectgrant.REFERENCES[1] Gartner, “Gartner Survey Highlights Challenges to HadoopAdoption,” May 2015. [Online]. Available: https://www.gartner.com/newsroom/id/30517174https://github.com/IBCNServices/tengu-charms[2] TEKsystems, “Lowered Expectations for IT Budgets2015 | TEKsystems,” Dec. 2014. [Online]. Avail-able: https://www.teksystems.com/en/resources/news-press/2014/teksystems-annual-it-forecast-2015?&year=2014[3] OASIS, “Topology and Orchestration Specification for CloudApplications (TOSCA) Version 1.0, Committee Specification01,” 2013.[4] M. Schwarzkopf, A. Konwinski, M. Abd-El-Malek, andJ. Wilkes, “Omega: Flexible, Scalable Schedulers for LargeCompute Clusters,” in Proceedings of the 8th ACM EuropeanConference on Computer Systems, ser. EuroSys ’13. NewYork, NY, USA: ACM, 2013, pp. 351–364.[5] B. Hindman, A. Konwinski, M. Zaharia, A. Ghodsi, A. D.Joseph, R. Katz, S. Shenker, and I. Stoica, “Mesos: A Plat-form for Fine-grained Resource Sharing in the Data Center,”in Proceedings of the 8th USENIX Conference on NetworkedSystems Design and Implementation, ser. NSDI’11. Berkeley,CA, USA: USENIX Association, 2011, pp. 295–308.[6] P. Lipton, S. Moser, D. Palma, M. Rutkowski,and T. Spatzier, “TOSCA Simple Profile in YAMLVersion 1.0,” Aug. 2015. [Online]. Available: https://docs.oasis-open.org/tosca/TOSCA-Simple-Profile-YAML/v1.0/csprd01/TOSCA-Simple-Profile-YAML-v1.0-csprd01.html# Toc430015763[7] J. Wettinger, M. Behrendt, T. Binz, U. Breitenbcher, G. Bre-iter, F. Leymann, S. Moser, I. Schwertle, and T. Spatzier,“Integrating Configuration Management with Model-drivenCloud Management based on TOSCA.” in CLOSER, 2013,pp. 437–446.[8] P. Bailis and A. Ghodsi, “Eventual Consistency Today: Lim-itations, Extensions, and Beyond,” Commun. ACM, vol. 56,no. 5, pp. 55–63, May 2013.[9] J. Wettinger, U. Breitenbcher, and F. Leymann,“Compensation-Based vs. Convergent DeploymentAutomation for Services Operated in the Cloud,” inService-Oriented Computing, ser. Lecture Notes in ComputerScience, X. Franch, A. K. Ghose, G. A. Lewis, and S. Bhiri,Eds. Springer Berlin Heidelberg, Nov. 2014, no. 8831, pp.336–350, dOI: 10.1007/978-3-662-45391-9 23.[10] T. Vanhove, G. Van Seghbroeck, T. Wauters, F. De Turck,B. Vermeulen, and P. Demeester, “Tengu: An ExperimentationPlatform for Big Data Applications,” in 2015 IEEE 35thInternational Conference on Distributed Computing SystemsWorkshops (ICDCSW), Jun. 2015, pp. 42–47.

Distributed service orchestration : eventually consistent cloud operation and integration

https://biblio.ugent.be/publication/8521043/file/8521045.pdf

Distributed service orchestration : eventually consistent cloud operation and integration

Abstract

Similar works

Full text

Available Versions

Crossref

Ghent University Academic Bibliography