SLA-Driven Governance of RESTful Systems

The Software as a Service (SaaS) paradigm has become entrenched in the industry as a deployment model, bringing flexibility to the customers and a recurring revenue to the business. The main architectural paradigm of SaaS systems is the service-oriented one since it provides numerous advantages in terms of elasticity, fault tolerance, and flexible architectural design. Currently, the RESTful paradigm, a layer of abstraction on the server created by defining resources and entities that can be accessed by means of a URI, is the preferred choice for the construction of SaaS, as it promotes the deployment, isolation and integration of microservices through APIs. Nowadays, APIs are regarded as a new form of business product and ever more organizations are publicly opening up access to their APIs as a way to create new business opportunities. In the same way, other organizations also consume a number of third-party APIs as part of their business. We henceforth define the concept of a RESTful System as an information system following the RESTful paradigm to shape the integration model between both its own components as well as other information systems. Furthermore, understanding governance as the way in which a component is directed and controlled, in RESTful Systems, those components will be the RESTful APIs and what we aim to control or regulate is their behavior (i.e., how an API is being consumed or provided). As APIs are increasingly regarded as business products, a crucial activity is to describe the set of plans (i.e., the pricing) that depicts the functionality and performance being offered to clients. API providers usually define certain limitations in each instance of a plan (e.g., quotas and rates); for example, a free plan might be limited to having one hundred monthly requests, and a professional plan to have five hundred monthly requests. However, although API providers use the Service Level Agreement (SLA) concept to delimit the functionality and guarantees to which they commit to their customers, there is no standard model used by API providers for modeling API pricing (including the plans and limitations). Although some providers do model the information regarding the API pricing and API limitations with an ad hoc approach, there is no widely accepted model in the industry. Wherefore answering questions regarding API limitations (e.g., determining whether or not a certain pricing is valid) is still a manual or non-interoperable process coming along with some inconveniences (being tedious, time-consuming, error-prone, etc.). Understating governance as to how a system is directed and controlled, we translate this concept to meet the SLA-driven approach: we consider the SLA (i.e., API pricing) as the element that will drive the directions, policies and rules to deliver and maintain the RESTful System. Adding the SLA to the idea of governance of RESTful systems leads to the main hypothesis of this dissertation: there is no well-established model for describing API pricings)in RESTful systems, which is hindering the automatic SLA-Driven governance. We claim the main goal of this thesis to be: the creation of an expressive, fully-fledged specification of SLAs for RESTful APIs endorsed with an open ecosystem of tools aimed at the SLA-Driven Governance of RESTful systems. The results of this endeavor are twofold: (I) Creation of a sufficiently expressive specification for the description of API pricings and the analysis of their validity. This comprises: (i) conducting an analysis of real-world APIs to evaluate the characteristics of the API pricings and limitations; (ii) identifying the relevance of SLAs in APIs in both academic and industrial scenarios; (iii) proposing a comprehensive model for describing API pricings; (iv) defining analysis operations for common questions regarding the validity in API pricings and limitations; (v) performing an evaluation of the model in real-world APIs. (II) Implementation of an ecosystem of tools to support the SLA-Driven governance of RESTful APIs. This includes: (i) developing a set of API governance tools; (ii) implementing a validity analysis operation; (iii) performing a validation of the tools and operations in realistic scenarios. In this thesis, we present the Governify4APIs ecosystem as the set comprised of (i) a model aimed at describing API pricings that is closely aligned with industry standards in APIs (OpenAPI Specification) and (ii) a set of companion tools for enacting the automatic governance using our specification, ranging from low-level validation tasks to SaaS solutions based on our model. Governify4APIs is, therefore, a fully-fledged specification, aligned with the mainstream standards and intended to enable an SLA-Driven Governance of RESTful Systems.El paradigma del software como servicio (SaaS) se ha afianzado en la industria como modelo de despliegue, aportando flexibilidad a los clientes y unos ingresos constantes a las organizaciones. El principal paradigma arquitectónico de los sistemas SaaS es la arquitectura orientada a servicios, ya que proporciona numerosas ventajas en términos de elasticidad, tolerancia a fallos y diseño flexible. RESTful, una capa de abstracción sobre el servidor creada mediante la definición de recursos y entidades a las que se puede acceder mediante una URI, es la opción preferida para la construcción de SaaS, ya que promueve el despliegue, el aislamiento y la integración de microservicios a través de APIs. Hoy en día, las APIs se consideran una nueva forma de producto empresarial y cada vez más organizaciones abren públicamente el acceso a sus APIs como forma de crear nuevas oportunidades de negocio. Del mismo modo, otras organizaciones también consumen una serie de APIs de terceros como parte de su negocio. A partir de ahora definimos el concepto de Sistema RESTful como un sistema de información que sigue el paradigma RESTful para conformar el modelo de integración tanto entre sus propios componentes como con otros sistemas de información. Además, entendiendo gobierno como la forma en que se dirige y controla un componente, en los sistemas RESTful, esos componentes serán las APIs RESTful y lo que pretendemos controlar o regular es su comportamiento (es decir, cómo se está consumiendo o proporcionando una API). Dado que las APIs están, cada vez más, siendo consideradas como productos comerciales, una actividad crucial es describir el conjunto de planes (es decir, el pricing) que describe la funcionalidad y el rendimiento que se ofrece a los clientes. Los proveedores de API suelen definir ciertas limitaciones en cada instancia de un plan (por ejemplo, quotas y rates); por ejemplo, un plan gratuito podría estar limitado a tener cien peticiones mensuales, y un plan profesional a tener quinientas peticiones mensuales. Sin embargo, aunque los proveedores de APIs utilizan el concepto de Acuerdo de Nivel de Servicio (SLA) para delimitar la funcionalidad y las garantías a las que se comprometen con sus clientes, no existe ningún modelo estándar usado por los proveedores para modelar el pricing de las API (incluyendo los planes y limitaciones). Aunque algunos proveedores modelan la información relativa a los pricings y las limitaciones de las APIs con un enfoque ad hoc, no existe un modelo ampliamente aceptado en el sector. Por lo tanto, responder a las preguntas relativas a las limitaciones de la APIs (por ejemplo, determinar si un determinado pricing es válido o no) sigue siendo un proceso manual o no interoperable, cosa que conlleva algunos inconvenientes (es tedioso, consume tiempo, es propenso a errores, etc.). Entendiendo el gobierno como la forma de dirigir y controlar un sistema, podemos traducir este concepto teniendo en cuenta el SLA, esto es, consideramos este elemento como aquel sobre el que se realiza la dirección, políticas y reglas para entregar y mantener el sistema RESTful. Añadir el concepto SLA a esa idea de gobierno de sistemas RESTful nos lleva a la hipótesis principal de esta tesis: no existe un modelo bien establecido para describir los SLAs (o pricing) en los sistemas RESTful, lo que está dificultando el gobierno automático. Es, por tanto, el objetivo principal de esta tesis la creación de una especificación expresiva y completa de SLAs para APIs RESTful, respaldada por un ecosistema abierto de herramientas orientadas al gobierno de sistemas RESTful dirigido por SLAs. Los resultados principales han sido: (I) Creación de una especificación suficientemente expresiva para la descripción de los pricings de la API y el análisis de su validez. Esto comprende: (i) realizar un análisis de APIs del mundo real para evaluar las características de los pricings y limitaciones de las APIs; (ii) identificar la relevancia de los SLAs en las APIs tanto en escenarios académicos como industriales; (iii) proponer un modelo completo para describir los pricings de las APIs; (iv) definir operaciones de análisis para preguntas comunes sobre la validez en los pricings y limitaciones de las APIs; (v) realizar una evaluación del modelo en APIs del mundo real. (II) Implementación de un ecosistema de herramientas para apoyar la gobernanza SLA-Driven de las APIs RESTful. Esto incluye: (i) desarrollar un conjunto de herramientas de gobierno de APIs; (ii) implementar una operación de análisis de validez; (iii) realizar una validación de las herramientas y operaciones en escenarios realistas. En esta tesis, presentamos el ecosistema Governify4APIs como el conjunto compuesto por (i) un modelo destinado a describir los pricings de las APIs y alineado estrechamente con los estándares de la industria (OpenAPI) y (ii) un conjunto de herramientas complementarias para el gobierno automático utilizando este modelo, que van desde tareas de validación hasta soluciones SaaS. Por lo tanto, Governify4APIs es una especificación acompañada de todo lo necesario, alineada con los estándares industriales y destinada a permitir un gobierno de sistemas RESTful dirigidos por SLAs

Model-driven round-trip engineering of REST APIs

Les API web s'han convertit cada vegada més en un actiu clau per a les empreses, que n'han promogut la implementació i la integració en les seves activitats quotidianes. A la pràctica, la majoria d'aquestes API web són "REST-like", que significa que s'adhereixen parcialment a l'estil arquitectònic conegut com transferència d'estat representacional ('representational state transfer', REST en anglés). De fet, REST és un paradigma de disseny i no proposa cap estàndard. Com a conseqüència, tant desenvolupar com consumir API REST són tasques difícils i costoses per als proveïdors i clients de l'API. L'objectiu d'aquesta tesi és facilitar el disseny, la implementació, la composició i el consum de les API REST, basant-se en tècniques d'enginyeria dirigida per models ('model-driven engineering', MDE en anglés). Aquesta tesi proposa les contribucions següents: EMF-REST, APIDiscoverer, APITester, APIGenerator, i APIComposer. Aquestes contribucions constitueixen un ecosistema que avança l'estat de la qüestió al camp de l'enginyeria de programari automàtica per al desenvolupament i el consum de les API REST.Las API Web se han convertido en una pieza fundamental para un gran número de compañías, que han promovido su implementación e integración en las actividades cotidianas del negocio. En la práctica, estas API Web son "REST-like", lo que significa que se adhieren parcialmente al estilo arquitectónico conocido como transferencia de estado representacional ('representational state transfer', REST en inglés). De hecho, REST es un paradigma de diseño y no propone ningún estándar. Por ello, tanto el desarrollo como el consumo de API REST son tareas difíciles y que demandan mucho tiempo de los proveedores y los clientes de API. El objetivo de esta tesis es facilitar el diseño, la implementación, la composición y el consumo de API REST, apoyándose en el desarrollo de software dirigido por modelos (DSDM). Esta tesis propone las siguientes contribuciones: EMF-REST, APIDiscoverer, APITester, APIGenerator y APIComposer. Estas contribuciones constituyen un ecosistema que avanza el estado de la cuestión en el área de la ingeniería del software referida a la automatización de las tareas relacionadas con el desarrollo y consumo de API REST.Web APIs have become an increasingly key asset for businesses, and their implementation and integration in companies' daily activities has thus been on the rise. In practice, most of these Web APIs are "REST-like", meaning that they adhere partially to the Representational State Transfer (REST) architectural style. In fact, REST is a design paradigm and does not propose any standard, so developing and consuming REST APIs end up being challenging and time-consuming tasks for API providers and clients. Therefore, the aim of this thesis is to facilitate the design, implementation, composition and consumption of REST APIs by relying on Model-Driven Engineering (MDE). Likewise, it offers the following contributions: EMF-REST, APIDiscoverer, APITester, APIGenerator and APIComposer. Together, these contributions make up an ecosystem which advances the state of the art of automated software engineering for REST APIs

Scalable and Quality-Aware Training Data Acquisition for Conversational Cognitive Services

Dialog Systems (or simply bots) have recently become a popular human-computer interface for performing user's tasks, by invoking the appropriate back-end APIs (Application Programming Interfaces) based on the user's request in natural language. Building task-oriented bots, which aim at performing real-world tasks (e.g., booking flights), has become feasible with the continuous advances in Natural Language Processing (NLP), Artificial Intelligence (AI), and the countless number of devices which allow third-party software systems to invoke their back-end APIs. Nonetheless, bot development technologies are still in their preliminary stages, with several unsolved theoretical and technical challenges stemming from the ambiguous nature of human languages. Given the richness of natural language, supervised models require a large number of user utterances paired with their corresponding tasks -- called intents. To build a bot, developers need to manually translate APIs to utterances (called canonical utterances) and paraphrase them to obtain a diverse set of utterances. Crowdsourcing has been widely used to obtain such datasets, by paraphrasing the initial utterances generated by the bot developers for each task. However, there are several unsolved issues. First, generating canonical utterances requires manual efforts, making bot development both expensive and hard to scale. Second, since crowd workers may be anonymous and are asked to provide open-ended text (paraphrases), crowdsourced paraphrases may be noisy and incorrect (not conveying the same intent as the given task). This thesis first surveys the state-of-the-art approaches for collecting large training utterances for task-oriented bots. Next, we conduct an empirical study to identify quality issues of crowdsourced utterances (e.g., grammatical errors, semantic completeness). Moreover, we propose novel approaches for identifying unqualified crowd workers and eliminating malicious workers from crowdsourcing tasks. Particularly, we propose a novel technique to promote the diversity of crowdsourced paraphrases by dynamically generating word suggestions while crowd workers are paraphrasing a particular utterance. Moreover, we propose a novel technique to automatically translate APIs to canonical utterances. Finally, we present our platform to automatically generate bots out of API specifications. We also conduct thorough experiments to validate the proposed techniques and models

Integration of Web APIs and Linked Data Using SPARQL Micro-Services - Application to Biodiversity Use Cases

International audienceIn recent years, Web APIs have become a de facto standard for exchanging machine-readable data on the Web. Despite this success, however, they often fail in making resource descriptions interoperable due to the fact that they rely on proprietary vocabularies that lack formal semantics.The Linked Data principles similarly seek the massive publication of data on the Web, yet with the specific goal of ensuring semantic interoperability.Given their complementary goals, it is commonly admitted that cross-fertilization could stem from the automatic combination of Linked Data and Web APIs. Towards this goal, in this paper we leverage the micro-service architectural principles to define a SPARQL Micro-Service architecture, aimed at querying Web APIs using SPARQL. A SPARQL micro-service is a lightweight SPARQL endpoint that provides access to a small, resource-centric, virtual graph. In this context, we argue that full SPARQL Query expressiveness can be supported efficiently without jeopardizing servers availability.Furthermore, we demonstrate how this architecture can be used to dynamically assign dereferenceable URIs to Web API resources that do not have URIs beforehand, thus literally “bringing” Web APIs into the Web of Data. We believe that the emergence of an ecosystem of SPARQL micro-services published by independent providers would enable Linked Data-based applications to easily glean pieces of data from a wealth of distributed, scalable, and reliable services. We describe a working prototype implementation and we finally illustrate the use of SPARQL micro-services in the context of two real-life use cases related to the biodiversity domain, developed in collaboration with the French National Museum of Natural History

Knowledge-driven architecture composition

Service interoperability for embedded devices is a mandatory feature for dynamically changing Internet-of-Things and Industry 4.0 software platforms. Service interoperability is achieved on a technical, syntactic, and semantic level. If service interoperability is achieved on all layers, plug and play functionality known from USB storage sticks or printer drivers becomes feasible. As a result, micro batch size production, individualized automation solution, or job order production become affordable. However, interoperability at the semantic layer is still a problem for the maturing class of IoT systems. Current solutions to achieve semantic integration of IoT devices’ heterogeneous services include standards, machine-understandable service descriptions, and the implementation of software adapters. Standardization bodies such as the VDMA tackle the problem by providing a reference software architecture and an information meta model for building up domain standards. For instance, the universal machine technology interface (UMATI) facilitates the data exchange between machines, components, installations, and their integration into a customerand user-specific IT ecosystem for mechanical engineering and plant construction worldwide. Automated component integration approaches fill the gap of software interfaces that are not relying on a global standard. These approaches translate required into provided software interfaces based on the needed architectural styles (e.g., client-server, layered, publish-subscribe, or cloud-based) using additional component descriptions. Interoperability at the semantic layer is achieved by relying on a shared domain vocabulary (e.g., an ontology) and service description (e.g., SAWSDL) used by all devices involved. If these service descriptions are available and machine-understandable knowledge of how to integrate software components on the functional and behavioral level is available, plug and play scenarios are feasible. Both standards and formal service descriptions cannot be applied effectively to IoT systems as they rely on the assumption that the semantic domain is completely known when they are noted down. This assumption is hard to believe as an increasing number of decentralized developed and connected IoT devices will exist (i.e., 30.73 billion in 2020 and 75.44 billion in 2025). If standards are applied in IoT systems, they must be updated continuously, so they contain the most recent domain knowledge agreed upon centrally and ahead of application. Although formal descriptions of concrete integration contexts can happen in a decentralized manner, they still rely on the assumption that the knowledge once noted down is complete. Hence, if an interoperable service from a new device is available that has not been considered in the initial integration context, the formal descriptions must be updated continuously. Both the formalization effort and keeping standards up to date result in too much additional engineering effort. Consequently, practitioners rely on implementing software adapters manually. However, this dull solution hardly scales with the increasing number of IoT devices. In this work, we introduce a novel engineering method that explicitly allows for an incomplete semantic domain description without losing the ability for automated IoT system integration. Dropping the completeness claim requires the management of incomplete integration knowledge. By sharing integration knowledge centrally, we assist the system integrator in automating software adapter generation. In addition to existing approaches, we enable semantic integration for services by making integration knowledge reusable. We empirically show with students that integration effort can be lowered in a home automation context

Evaluation of Arrowhead Framework in Condition Monitoring Application

The technological advancement in the field of electronics and information technology is changing how industrial automation systems are built. This phenomenon is commonly referred to as the fourth industrial revolution. However, before this prophecy on the change can manifest, new architectural solutions are needed to fully leverage the abilities brought by cheaper sensors, more advanced communication technology and more powerful processing units. The Arrowhead Framework tries to tackle this problem by providing means for Service-oriented architecture via System-of-Systems approach, where so-called application systems consume services provided by so-called core systems, which provide means for service discovery, service registration and service authorization. The goal of the thesis was to evaluate The Arrowhead Framework by developing a demo application on the edge-cloud setup used in the condition monitoring system of vibrating screens manufactured by Metso. The demo applications objective was to ease the configuration and installation of industrial Linux PC’s at the edge of the network. The methodological model for the evaluation was based on the design science research process (DSRP), which provides a model for research of IT artefacts. As a result, the Arrowhead Framework’s core features were found helpful in the problem domain, and suitable for small-scale test setup. However, the implementation of the framework was found to be low quality and lacking features from a production-ready software artefact. The found shortcomings were reported as feedback for the ongoing development process of the framework

Generating mock skeletons for lightweight Web service testing : a thesis presented in partial fulfilment of the requirements for the degree of Doctor of Philosophy in Computer Science at Massey University, Manawatū New Zealand

Modern application development allows applications to be composed using lightweight HTTP services. Testing such an application requires the availability of services that the application makes requests to. However, continued access to dependent services during testing may be restrained, making adequate testing a significant and non-trivial engineering challenge. The concept of Service Virtualisation is gaining popularity for testing such applications in isolation. It is a practise to simulate the behaviour of dependent services by synthesising responses using semantic models inferred from recorded traffic. Replacing services with their respective mocks is, therefore, useful to address their absence and move on application testing. In reality, however, it is unlikely that fully automated service virtualisation solutions can produce highly accurate proxies. Therefore, we recommend using service virtualisation to infer some attributes of HTTP service responses. We further acknowledge that engineers often want to fine-tune this. This requires algorithms to produce readily interpretable and customisable results. We assume that if service virtualisation is based on simple logical rules, engineers would have the potential to understand and customise rules. In this regard, Symbolic Machine Learning approaches can be investigated because of the high provenance of their results. Accordingly, this thesis examines the appropriateness of symbolic machine learning algorithms to automatically synthesise HTTP services' mock skeletons from network traffic recordings. We consider four commonly used symbolic techniques: the C4.5 decision tree algorithm, the RIPPER and PART rule learners, and the OCEL description logic learning algorithm. The experiments are performed employing network traffic datasets extracted from a few different successful, large-scale HTTP services. The experimental design further focuses on the generation of reproducible results. The chosen algorithms demonstrate the suitability of training highly accurate and human-readable semantic models for predicting the key aspects of HTTP service responses, such as the status and response headers. Having human-readable logics would make interpretation of the response properties simpler. These mock skeletons can then be easily customised to create mocks that can generate service responses suitable for testing

Collection, Transformation, and Integration of Data from the Web Services Domain

V současné době existuje několik repozitářů a datových modelů, které poskytují popisy webových služeb. Diplomová práce řeší problém transformace popisů webových služeb z několika datových modelů do jednoho sjednoceného datového modelu. Práce analyzuje existující datasety a datové modely pro webové služby, vytváří mapování mezi různými datovými modely, automatizuje sběr, transformace a integrace datových modelů webových služeb do jednotného datového modelu, ověřuje a vyhodnocuje výsledky extrakce.Currently, there are several repositories and data models that provide descriptions for Web APIs. The diploma thesis tackles the problem of transforming descriptions of Web APIs from several data models into one unified data model. It analyzes existing datasets and data models for Web APIs, establishes mappings between different data models, collects, transforms and integrates Web APIs data models into the unified data model, validates and evaluates extraction results

Propuesta de un estándar para solucionar los problemas de interoperabilidad semántica entre sistemas de tratamiento de datos de Seguridad Social

Los movimientos de población, especialmente los movimientos migratorios, son un fenómeno de gran impacto social y económico. Garantizar el ejercicio de los derechos en materia de Seguridad Social de las personas que se mueven por el mundo exige la puesta en marcha de una serie de acuerdos entre las instituciones competentes en diferentes países. El que esos acuerdos tengan una efectividad real exige que se realicen intercambios de datos entre sistemas de información totalmente heterogéneos. En ese contexto, las técnicas de interoperabilidad juegan un papel clave para que estos sistemas sean capaces de entenderse. La tesis trata los problemas de interoperabilidad en este ámbito, muy especialmente los relacionados con los aspectos semánticos. Partiendo de un análisis crítico de la situación actual y de las principales recomendaciones de los organismos internacionales, se propone una solución basada en la estandarización de un conjunto de metadatos que acompañen a cada uno de los datos intercambiados con el fin de evitar cualquier ambigüedad o interpretación equívoca. Como punto de partida para la propuesta de estándar se ha elegido Dublin Core, por ajustarse especialmente a las necesidades detectadas y por su amplia implantación en todos los ámbitos, incluido el de la administración púbica. Tal como se ha hecho en otros dominios, se ha adaptado este conjunto de elementos, modificando varios de ellos y añadiendo tres extensiones. Por otra parte, se propone un esquema de arquitectura de intercambio de datos entre las instituciones de Seguridad social basada en servicios web que demuestra la viabilidad y aplicabilidad de la especificación propuesta. Un aspecto importante es que el conjunto de metadatos se basa en una serie de vocabularios controlados que permiten definir y acotar sus posibles valores. Dichos vocabularios tienen diferente grado de desarrollo. Su revisión y extensión se propone como futura línea de investigación que dé continuidad a lo propuesto en esta tesis

Fundamental Approaches to Software Engineering

This open access book constitutes the proceedings of the 24th International Conference on Fundamental Approaches to Software Engineering, FASE 2021, which took place during March 27–April 1, 2021, and was held as part of the Joint Conferences on Theory and Practice of Software, ETAPS 2021. The conference was planned to take place in Luxembourg but changed to an online format due to the COVID-19 pandemic. The 16 full papers presented in this volume were carefully reviewed and selected from 52 submissions. The book also contains 4 Test-Comp contributions