Towards a Cloud Native Big Data Platform using MiCADO

In the big data era, creating self-managing scalable platforms for running big data applications is a fundamental task. Such self-managing and self-healing platforms involve a proper reaction to hardware (e.g., cluster nodes) and software (e.g., big data tools) failures, besides a dynamic resizing of the allocated resources based on overload and underload situations and scaling policies. The distributed and stateful nature of big data platforms (e.g., Hadoop-based cluster) makes the management of these platforms a challenging task. This paper aims to design and implement a scalable cloud native Hadoop-based big data platform using MiCADO, an open-source, and a highly customisable multi-cloud orchestration and auto-scaling framework for Docker containers, orchestrated by Kubernetes. The proposed MiCADO-based big data platform automates the deployment and enables an automatic horizontal scaling (in and out) of the underlying cloud infrastructure. The empirical evaluation of the MiCADO-based big data platform demonstrates how easy, efficient, and fast it is to deploy and undeploy Hadoop clusters of different sizes. Additionally, it shows how the platform can automatically be scaled based on user-defined policies (such as CPU-based scaling)

MiCADO – Towards a Microservice-based Cloud Application-level Dynamic Orchestrator

In order to satisfy end-user requirements, many scientific and commercial applications require access to dynamically adjustable infrastructure resources. Cloud computing has the potential to provide these dynamic capabilities. However, utilising these capabilities from application code is not trivial and requires application developers to understand low-level technical details of clouds. This paper investigates how a generic framework can be developed that supports the dynamic orchestration of cloud applications both at deployment and at run-time. The advantages and challenges of designing such framework based on microservices is analysed, and a generic framework, called MiCADO – (Microservices-based Cloud Application-level Dynamic Orchestrator) is proposed. A first prototype implementation of MiCADO to support data intensive commercial web applications is also presented

Unified Management of Applications on Heterogeneous Clouds

La diversidad con la que los proveedores cloud ofrecen sus servicios, definiendo sus propias interfaces y acuerdos de calidad y de uso, dificulta la portabilidad y la interoperabilidad entre proveedores, lo que incurre en el problema conocido como el bloqueo del vendedor. Dada la heterogeneidad que existe entre los distintos niveles de abstracción del cloud, como IaaS y PaaS, hace que desarrollar aplicaciones agnósticas que sean independientes de los proveedores y los servicios en los que se van a desplegar sea aún un desafío. Esto también limita la posibilidad de migrar los componentes de aplicaciones cloud en ejecución a nuevos proveedores. Esta falta de homogeneidad también dificulta el desarrollo de procesos para operar las aplicaciones que sean robustos ante los errores que pueden ocurrir en los distintos proveedores y niveles de abstracción. Como resultado, las aplicaciones pueden quedar ligadas a los proveedores para las que fueron diseñadas, limitando la capacidad de los desarrolladores para reaccionar ante cambios en los proveedores o en las propias aplicaciones. En esta tesis se define trans-cloud como una nueva dimensión que unifica la gestión de distintos proveedores y niveles de servicios, IaaS y PaaS, bajo una misma API y hace uso del estándar TOSCA para describir aplicaciones agnósticas y portables, teniendo procesos automatizados, por ejemplo para el despliegue. Por otro lado, haciendo uso de las topologías estructuradas de TOSCA, trans-cloud propone un algoritmo genérico para la migración de componentes de aplicaciones en ejecución. Además, trans-cloud unifica la gestión de los errores, permitiendo tener procesos robustos y agnósticos para gestionar el ciclo de vida de las aplicaciones, independientemente de los proveedores y niveles de servicio donde se estén ejecutando. Por último, se presentan los casos de uso y los resultados de los experimentos usados para validar cada una de estas propuestas

Algorithms for advance bandwidth reservation in media production networks

Media production generally requires many geographically distributed actors (e.g., production houses, broadcasters, advertisers) to exchange huge amounts of raw video and audio data. Traditional distribution techniques, such as dedicated point-to-point optical links, are highly inefficient in terms of installation time and cost. To improve efficiency, shared media production networks that connect all involved actors over a large geographical area, are currently being deployed. The traffic in such networks is often predictable, as the timing and bandwidth requirements of data transfers are generally known hours or even days in advance. As such, the use of advance bandwidth reservation (AR) can greatly increase resource utilization and cost efficiency. In this paper, we propose an Integer Linear Programming formulation of the bandwidth scheduling problem, which takes into account the specific characteristics of media production networks, is presented. Two novel optimization algorithms based on this model are thoroughly evaluated and compared by means of in-depth simulation results

Describing and Processing Topology and Quality of Service Parameters of Applications in the Cloud

Typical cloud applications require high-level policy driven orchestration to achieve efficient resource utilisation and robust security to support different types of users and user scenarios. However, the efficient and secure utilisation of cloud resources to run applications is not trivial. Although there have been several efforts to support the coordinated deployment, and to a smaller extent the run-time orchestration of applications in the Cloud, no comprehensive solution has emerged until now that successfully leverages applications in an efficient, secure and seamless way. One of the major challenges is how to specify and manage Quality of Service (QoS) properties governing cloud applications. The solution to address these challenges could be a generic and pluggable framework that supports the optimal and secure deployment and run-time orchestration of applications in the Cloud. A specific aspect of such a cloud orchestration framework is the need to describe complex applications incorporating several services. These application descriptions must specify both the structure of the application and its QoS parameters, such as desired performance, economic viability and security. This paper proposes a cloud technology agnostic approach to application descriptions based on existing standards and describes how these application descriptions can be processed to manage applications in the Cloud

PaaS-BDP a multi-cloud architectural pattern for big data processing on a platform-as-a-service model

Copyright © 2018 by SCITEPRESS – Science and Technology Publications, Lda. All rights reserved. This paper presents a contribution to the fields of Big Data Analytics and Software Architecture, namely an emerging and unifying architectural pattern for big data processing in the cloud from a cloud consumer’s perspective. PaaS-BDP (Platform-as-a-Service for Big Data) is an architectural pattern based on resource pooling and the use of a unified programming model for building big data processing pipelines capable of processing both batch and stream data. It uses container cluster technology on a PaaS service model to overcome common shortfalls of current big data solutions offered by major cloud providers such as low portability, lack of interoperability and the risk of vendor lock-in

Characterizing and providing interoperability to function as a service platforms

Dissertação para obtenção do Grau de Mestre em Engenharia Informática e de ComputadoresA computação sem servidor abstrai o controlo da infraestrutura dos programadores e executa código a pedido com escalonamento automático onde apenas se é cobrado pela quantidade de recursos consumidos. Um dos serviços mais populares da computação sem servidor é a Função como Serviço (Function-as-a-Service ou FaaS), onde os programadores são muitas vezes confrontados com requisitos específicos dos prestadores de serviços de nuvem. Requisitos de assinatura das funções, e o uso de bibliotecas exclusivas ao prestador de serviços, foram identificados como sendo as principais causas de problemas de portabilidade das aplicações FaaS. O controlo reduzido da infraestrutura e a elevada dependência para com o prestador de serviços dá origem a diversos problemas de aprisionamento tecnológico. Neste trabalho, introduzimos o QuickFaaS, uma ferramenta para desktop de interoperabilidade multi-cloud com foco principal no desenvolvimento de funções agnósticas à nuvem e na criação das mesmas na respetiva plataforma. O QuickFaaS permite melhorar substancialmente a produtividade, flexibilidade e agilidade no desenvolvimento de soluções sem servidor para múltiplos prestadores de serviços, sem o requisito de instalar software adicional. A abordagem agnóstica à nuvem irá permitir que os programadores reutilizem as suas funções em diferentes prestadores de serviços sem terem a necessidade de reescrever código. A solução visa a minimizar o aprisionamento tecnológico nas plataformas FaaS através do aumento da portabilidade das funções sem servidor, incentivando assim programadores e organizações a apostarem em diferentes prestadores de serviços em troca de um benefício funcional.Serverless computing hides infrastructure management from developers and runs code on-demand automatically scaled and billed during code’s execution time. One of the most popular serverless backend services is called Function-as-a-Service (FaaS), in which developers are many times confronted with cloud-specific requirements. Function signature requirements, and the usage of custom libraries that are unique to cloud providers, were identified as the two main reasons for portability issues in FaaS applications. Such reduced control over the infrastructure and tight-coupling with cloud services amplifies various vendor lock-in problems. In this work, we introduce QuickFaaS, a multi-cloud interoperability desktop tool targeting cloud-agnostic functions development and FaaS deployments. QuickFaaS substantially improves developers’ productivity, flexibility and agility when creating serverless solutions to multiple cloud providers, without requiring the installation of extra software. The proposed cloud-agnostic approach enables developers to reuse their serverless functions in different cloud providers with no need to rewrite code. The solution aims to minimize vendor lock-in in FaaS platforms by increasing the portability of serverless functions, which will, therefore, encourage developers and organizations to target different providers in exchange for a functional benefit.N/