Dimensionerings- en werkverdelingsalgoritmen voor lambda grids

Grids bestaan uit een verzameling reken- en opslagelementen die geografisch verspreid kunnen zijn, maar waarvan men de gezamenlijke capaciteit wenst te benutten. Daartoe dienen deze elementen verbonden te worden met een netwerk. Vermits veel wetenschappelijke applicaties gebruik maken van een Grid, en deze applicaties doorgaans grote hoeveelheden data verwerken, is het noodzakelijk om een netwerk te voorzien dat dergelijke grote datastromen op betrouwbare wijze kan transporteren. Optische transportnetwerken lenen zich hier uitstekend toe. Grids die gebruik maken van dergelijk netwerk noemt men lambda Grids. Deze thesis beschrijft een kader waarin het ontwerp en dimensionering van optische netwerken voor lambda Grids kunnen beschreven worden. Ook wordt besproken hoe werklast kan verdeeld worden op een Grid eens die gedimensioneerd is. Een groot deel van de resultaten werd bekomen door simulatie, waarbij gebruik gemaakt wordt van een eigen Grid simulatiepakket dat precies focust op netwerk- en Gridelementen. Het ontwerp van deze simulator, en de daarbijhorende implementatiekeuzes worden dan ook uitvoerig toegelicht in dit werk

Survey and Analysis of Production Distributed Computing Infrastructures

This report has two objectives. First, we describe a set of the production distributed infrastructures currently available, so that the reader has a basic understanding of them. This includes explaining why each infrastructure was created and made available and how it has succeeded and failed. The set is not complete, but we believe it is representative. Second, we describe the infrastructures in terms of their use, which is a combination of how they were designed to be used and how users have found ways to use them. Applications are often designed and created with specific infrastructures in mind, with both an appreciation of the existing capabilities provided by those infrastructures and an anticipation of their future capabilities. Here, the infrastructures we discuss were often designed and created with specific applications in mind, or at least specific types of applications. The reader should understand how the interplay between the infrastructure providers and the users leads to such usages, which we call usage modalities. These usage modalities are really abstractions that exist between the infrastructures and the applications; they influence the infrastructures by representing the applications, and they influence the ap- plications by representing the infrastructures

The Ultralight project: the network as an integrated and managed resource for data-intensive science

Looks at the UltraLight project which treats the network interconnecting globally distributed data sets as a dynamic, configurable, and closely monitored resource to construct a next-generation system that can meet the high-energy physics community's data-processing, distribution, access, and analysis needs

Evolution towards Smart Optical Networking: Where Artificial Intelligence (AI) meets the World of Photonics

Smart optical networks are the next evolution of programmable networking and programmable automation of optical networks, with human-in-the-loop network control and management. The paper discusses this evolution and the role of Artificial Intelligence (AI)

Towards a cloud enabler : from an optical network resource provisioning system to a generalized architecture for dynamic infrastructure services provisioning

This work was developed during a period where most of the optical management and provisioning system where manual and proprietary. This work contributed to the evolution of the state of the art of optical networks with new architectures and advanced virtual infrastructure services. The evolution of optical networks, and internet globally, have been very promising during the last decade. The impact of mobile technology, grid, cloud computing, HDTV, augmented reality and big data, among many others, have driven the evolution of optical networks towards current service technologies, mostly based on SDN (Software Defined Networking) architectures and NFV(Network Functions Virtualisation). Moreover, the convergence of IP/Optical networks and IT services, and the evolution of the internet and optical infrastructures, have generated novel service orchestrators and open source frameworks. In fact, technology has evolved that fast that none could foresee how important Internet is for our current lives. Said in other words, technology was forced to evolve in a way that network architectures became much more transparent, dynamic and flexible to the end users (applications, user interfaces or simple APIs). This Thesis exposes the work done on defining new architectures for Service Oriented Networks and the contribution to the state of the art. The research work is divided into three topics. It describes the evolution from a Network Resource Provisioning System to an advanced Service Plane, and ends with a new architecture that virtualized the optical infrastructure in order to provide coordinated, on-demand and dynamic services between the application and the network infrastructure layer, becoming an enabler for the new generation of cloud network infrastructures. The work done on defining a Network Resource Provisioning System established the first bases for future work on network infrastructure virtualization. The UCLP (User Light Path Provisioning) technology was the first attempt for Customer Empowered Networks and Articulated Private Networks. It empowered the users and brought virtualization and partitioning functionalities into the optical data plane, with new interfaces for dynamic service provisioning. The work done within the development of a new Service Plane allowed the provisioning of on-demand connectivity services from the application, and in a multi-domain and multi-technology scenario based on a virtual network infrastructure composed of resources from different infrastructure providers. This Service Plane facilitated the deployment of applications consuming large amounts of data under deterministic conditions, so allowing the networks behave as a Grid-class resource. It became the first on-demand provisioning system that at lower levels allowed the creation of one virtual domain composed from resources of different providers. The last research topic presents an architecture that consolidated the work done in virtualisation while enhancing the capabilities to upper layers, so fully integrating the optical network infrastructure into the cloud environment, and so providing an architecture that enabled cloud services by integrating the request of optical network and IT infrastructure services together at the same level. It set up a new trend into the research community and evolved towards the technology we use today based on SDN and NFV. Summing up, the work presented is focused on the provisioning of virtual infrastructures from the architectural point of view of optical networks and IT infrastructures, together with the design and definition of novel service layers. It means, architectures that enabled the creation of virtual infrastructures composed of optical networks and IT resources, isolated and provisioned on-demand and in advance with infrastructure re-planning functionalities, and a new set of interfaces to open up those services to applications or third parties.Aquesta tesi es va desenvolupar durant un període on la majoria de sistemes de gestió de xarxa òptica eren manuals i basats en sistemes propietaris. En aquest sentit, la feina presentada va contribuir a l'evolució de l'estat de l'art de les xarxes òptiques tant a nivell d’arquitectures com de provisió d’infraestructures virtuals. L'evolució de les xarxes òptiques, i d'Internet a nivell mundial, han estat molt prometedores durant l'última dècada. L'impacte de la tecnologia mòbil, la computació al núvol, la televisió d'alta definició, la realitat augmentada i el big data, entre molts altres, han impulsat l'evolució cap a xarxes d’altes prestacions amb nous serveis basats en SDN (Software Defined Networking) i NFV (Funcions de xarxa La virtualització). D'altra banda, la convergència de xarxes òptiques i els serveis IT, junt amb l'evolució d'Internet i de les infraestructures òptiques, han generat nous orquestradors de serveis i frameworks basats en codi obert. La tecnologia ha evolucionat a una velocitat on ningú podria haver predit la importància que Internet està tenint en el nostre dia a dia. Dit en altres paraules, la tecnologia es va veure obligada a evolucionar d'una manera on les arquitectures de xarxa es fessin més transparent, dinàmiques i flexibles vers als usuaris finals (aplicacions, interfícies d'usuari o APIs simples). Aquesta Tesi presenta noves arquitectures de xarxa òptica orientades a serveis. El treball de recerca es divideix en tres temes. Es presenta un sistema de virtualització i aprovisionament de recursos de xarxa i la seva evolució a un pla de servei avançat, per acabar presentant el disseny d’una nova arquitectura capaç de virtualitzar la infraestructura òptica i IT i proporcionar serveis de forma coordinada, i sota demanda, entre l'aplicació i la capa d'infraestructura de xarxa òptica. Tot esdevenint un facilitador per a la nova generació d'infraestructures de xarxa en el núvol. El treball realitzat en la definició del sistema de virtualització de recursos va establir les primeres bases sobre la virtualització de la infraestructura de xarxa òptica en el marc de les “Customer Empowered Networks” i “Articulated Private Networks”. Amb l’objectiu de virtualitzar el pla de dades òptic, i oferir noves interfícies per a la provisió de serveis dinàmics de xarxa. En quant al pla de serveis presentat, aquest va facilitat la provisió de serveis de connectivitat sota demanda per part de l'aplicació, tant en entorns multi-domini, com en entorns amb múltiples tecnologies. Aquest pla de servei, anomenat Harmony, va facilitar el desplegament de noves aplicacions que consumien grans quantitats de dades en condicions deterministes. En aquest sentit, va permetre que les xarxes es comportessin com un recurs Grid, i per tant, va esdevenir el primer sistema d'aprovisionament sota demanda que permetia la creació de dominis virtuals de xarxa composts a partir de recursos de diferents proveïdors. Finalment, es presenta l’evolució d’un pla de servei cap una arquitectura global que consolida el treball realitzat a nivell de convergència d’infraestructures (òptica + IT) i millora les capacitats de les capes superiors. Aquesta arquitectura va facilitar la plena integració de la infraestructura de xarxa òptica a l'entorn del núvol. En aquest sentit, aquest resultats van evolucionar cap a les tendències actuals de SDN i NFV. En resum, el treball presentat es centra en la provisió d'infraestructures virtuals des del punt de vista d’arquitectures de xarxa òptiques i les infraestructures IT, juntament amb el disseny i definició de nous serveis de xarxa avançats, tal i com ho va ser el servei de re-planificació dinàmicaPostprint (published version

Future of networking is the future of Big Data, The

2019 Summer.Includes bibliographical references.Scientific domains such as Climate Science, High Energy Particle Physics (HEP), Genomics, Biology, and many others are increasingly moving towards data-oriented workflows where each of these communities generates, stores and uses massive datasets that reach into terabytes and petabytes, and projected soon to reach exabytes. These communities are also increasingly moving towards a global collaborative model where scientists routinely exchange a significant amount of data. The sheer volume of data and associated complexities associated with maintaining, transferring, and using them, continue to push the limits of the current technologies in multiple dimensions - storage, analysis, networking, and security. This thesis tackles the networking aspect of big-data science. Networking is the glue that binds all the components of modern scientific workflows, and these communities are becoming increasingly dependent on high-speed, highly reliable networks. The network, as the common layer across big-science communities, provides an ideal place for implementing common services. Big-science applications also need to work closely with the network to ensure optimal usage of resources, intelligent routing of requests, and data. Finally, as more communities move towards data-intensive, connected workflows - adopting a service model where the network provides some of the common services reduces not only application complexity but also the necessity of duplicate implementations. Named Data Networking (NDN) is a new network architecture whose service model aligns better with the needs of these data-oriented applications. NDN's name based paradigm makes it easier to provide intelligent features at the network layer rather than at the application layer. This thesis shows that NDN can push several standard features to the network. This work is the first attempt to apply NDN in the context of large scientific data; in the process, this thesis touches upon scientific data naming, name discovery, real-world deployment of NDN for scientific data, feasibility studies, and the designs of in-network protocols for big-data science

Integrating multiple clusters for compute-intensive applications

Multicluster grids provide one promising solution to satisfying the growing computational demands of compute-intensive applications. However, it is challenging to seamlessly integrate all participating clusters in different domains into a single virtual computational platform. In order to fully utilize the capabilities of multicluster grids, computer scientists need to deal with the issue of joining together participating autonomic systems practically and efficiently to execute grid-enabled applications. Driven by several compute-intensive applications, this theses develops a multicluster grid management toolkit called Pelecanus to bridge the gap between user\u27s needs and the system\u27s heterogeneity. Application scientists will be able to conduct very large-scale execution across multiclusters with transparent QoS assurance. A novel model called DA-TC (Dynamic Assignment with Task Containers) is developed and is integrated into Pelecanus. This model uses the concept of a task container that allows one to decouple resource allocation from resource binding. It employs static load balancing for task container distribution and dynamic load balancing for task assignment. The slowest resources become useful rather than be bottlenecks in this manner. A cluster abstraction is implemented, which not only provides various cluster information for the DA-TC execution model, but also can be used as a standalone toolkit to monitor and evaluate the clusters\u27 functionality and performance. The performance of the proposed DA-TC model is evaluated both theoretically and experimentally. Results demonstrate the importance of reducing queuing time in decreasing the total turnaround time for an application. Experiments were conducted to understand the performance of various aspects of the DA-TC model. Experiments showed that our model could significantly reduce turnaround time and increase resource utilization for our targeted application scenarios. Four applications are implemented as case studies to determine the applicability of the DA-TC model. In each case the turnaround time is greatly reduced, which demonstrates that the DA-TC model is efficient for assisting application scientists in conducting their research. In addition, virtual resources were integrated into the DA-TC model for application execution. Experiments show that the execution model proposed in this thesis can work seamlessly with multiple hybrid grid/cloud resources to achieve reduced turnaround time