IOStack: Software-Defined Object Storage

The complexity and scale of today’s cloud storage systems is growing fast. In response to these challenges, Software- Defined Storage (SDS) has recently become a prime candidate to simplify storage management in the cloud. This article presents IOStack: The first SDS architecture for object stores (OpenStack Swift). At the control plane, the provisioning of SDS services to tenants is made according to a set of policies managed via a high-level DSL. Policies may target storage automation and/or specific SLA objectives. At the data plane, policies define the enforcement of SDS services, namely filters, on a tenant’s requests. Moreover, IOStack is a framework to build a variety of filters, ranging from general-purpose computations close to the data to specialized data management mechanisms. Our experiments illustrate that IOStack enables easy and effective policy-based provisioning, which can significantly improve the operation of a multi-tenant object store.This work has been funded by the European Union through project H2020 “IOStack: Software-Defined Storage for Big Data” (644182) and by the Spanish Ministry of Science and Innovation through project “Servicios Cloud y Redes Comunitarias” (TIN-2013-47245-C2-2-R).Peer ReviewedPostprint (author's final draft

IOStack: Software-Defined Object Storage

The complexity and scale of today’s cloud storage systems is growing fast. In response to these challenges, Software- Defined Storage (SDS) has recently become a prime candidate to simplify storage management in the cloud. This article presents IOStack: The first SDS architecture for object stores (OpenStack Swift). At the control plane, the provisioning of SDS services to tenants is made according to a set of policies managed via a high-level DSL. Policies may target storage automation and/or specific SLA objectives. At the data plane, policies define the enforcement of SDS services, namely filters, on a tenant’s requests. Moreover, IOStack is a framework to build a variety of filters, ranging from general-purpose computations close to the data to specialized data management mechanisms. Our experiments illustrate that IOStack enables easy and effective policy-based provisioning, which can significantly improve the operation of a multi-tenant object store.This work has been funded by the European Union through project H2020 “IOStack: Software-Defined Storage for Big Data” (644182) and by the Spanish Ministry of Science and Innovation through project “Servicios Cloud y Redes Comunitarias” (TIN-2013-47245-C2-2-R).Peer Reviewe

D4.6 REUSABLE MODEL & ANALYTICAL TOOLS: SOFTWARE PROTOTYPE 3

This document is the third and last software demonstrator deliverable of PolicyCLOUD at M34, October 2022, of the project and is intended for the reviewers of the software deliverables. This deliverable provides the description of the final software demonstrator for the components of the Integrated Data Acquisition and Analytics (DAA) Layer, which provides the analytical capabilities of the PolicyCLOUD platform. The components include the DAA API Gateway (responsible for the overall orchestration and the layer API), the built-in analytical tools such as the new Trend Analysis tool as well as the analytical tools already presented in D4.4 [6]: Data cleaning and interoperability, Situational Knowledge, Opinion Mining & Sentiment Analysis and Social Dynamics & Behavioural Data analysis, and the Operational Data Repository. The full integration of the DAA API Gateway (responsible for the overall orchestration and the layer API) was demonstrated during the review of June 2021 for two use cases. As of the publication of this deliverable, this has been extended in multiple ways: Additional ingest analytics capabilities such as Trend Analysis were added as detailed in section 2.1.3 and 2.1.4 The new integration of the Politika framework with PolicyCloud as detailed in section 2.2.5 of D4.5 [3] and as further detailed in sections 2.1.5.4 and 2.1.6 of this deliverable. The integrated Politika framework was itself used for two use cases, as detailed in section 2.2.6.2 The novel seamless architecture which was introduced in D4.4 [6] and which presents single logical datasets to users that can be explored with SQL had further been enhanced in the past year by performance enhancements of the SQL JOIN as detailed in section 4.4.4 of D4.5. The impact of using the seamless technology is minimal, as the users do not even know in what storage tier their datasets are stored. The only impact is a change of the SQL statement that now makes use of atable function that accepts standard SQL statements though, as detailed in section 2.2.8.3.This deliverable is submitted to the EC, not yet approved

D6.9 INTEGRATION OF RESULTS: POLICYCLOUD COMPLETE ENVIRONMENT M36

This deliverable has been released in December 2022, at M36 of the project, and its main objective is to specify the final integration results between the PolicyCLOUD components. This deliverable will follow the methodology of D6.2 and D6.8 that were respectively submitted in M12 (December 2020) and M24 (December 2021) which have two main pillars: Define common practices for integration and validation of the outcomes of the project Detail the cloud environment the project will make use of to demonstrate the results Regarding the former, GitLab will be the base code repository for the project, where the project already owns an organizational account. Over GitLab [1], the trunk-based development branching policy has been applied, as we considered it the most suitable policy given the project characteristics. Also, GitLab’s issue reporting tool has been adopted, as it is fully integrated with GitLab’s features. The test bed to support the demonstrators has been deployed over EGI’s (EGI) infrastructure where flexibility is one of the critical features. This deliverable abstractly incorporates all the changes and implementations that WP2, WP3, WP4 and WP5 had made during the second year of the project. More details about the components and the actual implementation can be found in the related WP deliverables [7] [8] [9]. In detail, the schemas of the data have been finalized so the standard version that we defined initiated the data import to the repository of PolicyCLOUD. Moreover, the infrastructure (IaaS) and the platform deployment (PaaS/ Serverless) have been restructured and reshaped based on the latest needs of the components. EGI deployed the new flavour of PolicyCLOUD to the Openstack Infrastructure and IBM made the proper changes to the Openwhisk middleware for the serverless and other services. The related WP deliverables highlight detailed information and instructions for each component change that in total orchestrate the PolicyCLOUD engine.This deliverable is submitted to the EC, not yet approved

D2.7 CONCEPTUAL MODEL & REFERENCE ARCHITECTURE

The third and final version of the PolicyCLOUD Conceptual Model & Reference Architecture (originally submitted as Deliverable D2.2 in September 2020 [20] with the second version submitted as D2.6 in June 2021 [21]) is presented in this document. The PolicyCLOUD Conceptual Model presents the overall project concept along 2 main axes. Along the first data axis PolicyCLOUD delivers Cloud Gateways and APIs to access data sources and adapt to their interfaces so as to simplify interaction and data collection from any source. Along the second main axis, the Policies Management Framework of PolicyCLOUD allows the definition of forward-looking policies as well as their dynamic adaptation and refocusing to the population they are applied on. Based on the project’s offerings along the main two axes of the Concept, five main building blocks (in a layered manner) define its Architecture: (1) The Cloud Based Environment and Data Acquisition, (2) Data Analytics, (3) the Policies Management Framework, (4) the Policy Development Toolkit and (5) The Marketplace. The architecture also includes a Data Governance Model, Protection and Privacy Enforcement and the Ethical Framework as depicted in Figure 2. The architecture allows for integrated data acquisition and analytics. It also allows data fusion with processing and initial analytics (see 7.6.5) as well as seamless analytics (see 7.6.6) on hybrid data at rest. Integration in PolicyCLOUD follows three directions: (i) architecture integration, (ii) integration with the cloud infrastructure and (iii) integration with Use Case scenarios through the implementation of end-to- end scenarios. Additional integration activities take place along the two frameworks of PolicyCLOUD, (a) the Data Governance model, protection and privacy enforcement mechanism and (b) the Ethical and Legal Compliance framework. For end-to-end data path analysis we have used two Use Case scenarios: (i) the scenario of Use Case 1: “Radicalization incidents” and the scenario of Use Case 2: “Visualization of negative and positive opinions on social networks for different products”. The new updates in this final document provide the following: Analysis of how External Frameworks can be integrated with PolicyCLOUD (section 7.6.11.4); Presentation of the overall Conceptual View and architecture of the Data Marketplace (section 7.9.1); Outline of the mechanisms developed for initialising the Policy Development Toolkit with Policy Model components and the visualization of results (section 7.8.3); Analysis of the Ethical and Legal Compliance Framework positive interventions to the PolicyCLOUD architecture, including the addition of specific fields/parameters to the registration Application Programming Interfaces to be populated with details regarding each individual analytics tool and dataset/data source (section 7.5); Presentation of the integration of the Data Governance model, protection and privacy enforcement mechanisms with the Policy Development Toolkit, the cloud gateways and the marketplace (section 7.10.2), and within the same context, the integration of EGI-Check-in with Keycloak including the integration of the Data Governance model, protection and privacy enforcement mechanisms with the Kubernetes cluster. The document also addresses the Reviewers’ comments to the previous version of the deliverable (Deliverable D2.6), included in the second review report. In order to address these comments, additional updates of Deliverable D2.7 include: (i) links to specific user/stakeholder requirements (D2.5), (ii) descriptions and implementation details for the two remaining pilot Use Cases (Sofia and London) and (iii) reference to EOSC and to the role of the Conceptual Model & Reference Architecture document for the identification of the relevant services and of their providers, and description of the onboarding process based on Deliverable D3.4 [22].This deliverable is submitted to the EC, not yet approved