A Preemption-Based Meta-Scheduling System for Distributed Computing

This research aims at designing and building a scheduling framework for distributed computing systems with the primary objectives of providing fast response times to the users, delivering high system throughput and accommodating maximum number of applications into the systems. The author claims that the above mentioned objectives are the most important objectives for scheduling in recent distributed computing systems, especially Grid computing environments. In order to achieve the objectives of the scheduling framework, the scheduler employs arbitration of application-level schedules and preemption of executing jobs under certain conditions. In application-level scheduling, the user develops a schedule for his application using an execution model that simulates the execution behavior of the application. Since application-level scheduling can seriously impede the performance of the system, the scheduling framework developed in this research arbitrates between different application-level schedules corresponding to different applications to provide fair system usage for all applications and balance the interests of different applications. In this sense, the scheduling framework is not a classical scheduling system, but a meta-scheduling system that interacts with the application-level schedulers. Due to the large system dynamics involved in Grid computing systems, the ability to preempt executing jobs becomes a necessity. The meta-scheduler described in this dissertation employs well defined scheduling policies to preempt and migrate executing applications. In order to provide the users with the capability to make their applications preemptible, a user-level check-pointing library called SRS (Stop-Restart Software) was also developed by this research. The SRS library is different from many user-level check-pointing libraries since it allows reconfiguration of applications between migrations. This reconfiguration can be achieved by changing the processor configuration and/or data distribution. The experimental results provided in this dissertation demonstrates the utility of the metascheduling framework for distributed computing systems. And lastly, the metascheduling framework was put to practical use by building a Grid computing system called GradSolve. GradSolve is a flexible system and it allows the application library writers to upload applications with different capabilities into the system. GradSolve is also unique with respect to maintaining traces of the execution of the applications and using the traces for subsequent executions of the application

Optimal Management of community Demand Response

More than one-third of the electricity produced globally is consumed by the residential sectors [1], with nearly 17% of CO2 emissions, are coming from residential buildings according to reports from 2018 [2] [3]. In order to cope with increase in electricity demand and consumption, while considering the environmental impacts, electricity providers are seeking to implement solutions to help them balance the supply with the electricity demand while mitigating emissions. Thus, increasing the number of conventional generation units and using unreliable renewable source of energy is not a viable investment. That’s why, in recent years research attention has shifted to demand side solutions [4]. This research investigates the optimal management for an urban residential community, that can help in reducing energy consumption and peak and CO2 emissions. This will help to put an agreement with the grid operator for an agreed load shape, for efficient demand response (DR) program implementation. This work uses a framework known as CityLearn [2]. It is based on a Machine Learning branch known as Reinforcement Learning (RL), and it is used to test a variety of intelligent agents for optimizing building load consumption and load shape. The RL agent is used for controlling hot water and chilled water storages, as well as the battery system. When compared to the regular building usage, the results demonstrate that utilizing an RL agent for storage system control can be helpful, as the electricity consumption is greatly reduced when it’s compared to the normal building consumption

High Performance Multiview Video Coding

Following the standardization of the latest video coding standard High Efficiency Video Coding in 2013, in 2014, multiview extension of HEVC (MV-HEVC) was published and brought significantly better compression performance of around 50% for multiview and 3D videos compared to multiple independent single-view HEVC coding. However, the extremely high computational complexity of MV-HEVC demands significant optimization of the encoder. To tackle this problem, this work investigates the possibilities of using modern parallel computing platforms and tools such as single-instruction-multiple-data (SIMD) instructions, multi-core CPU, massively parallel GPU, and computer cluster to significantly enhance the MVC encoder performance. The aforementioned computing tools have very different computing characteristics and misuse of the tools may result in poor performance improvement and sometimes even reduction. To achieve the best possible encoding performance from modern computing tools, different levels of parallelism inside a typical MVC encoder are identified and analyzed. Novel optimization techniques at various levels of abstraction are proposed, non-aggregation massively parallel motion estimation (ME) and disparity estimation (DE) in prediction unit (PU), fractional and bi-directional ME/DE acceleration through SIMD, quantization parameter (QP)-based early termination for coding tree unit (CTU), optimized resource-scheduled wave-front parallel processing for CTU, and workload balanced, cluster-based multiple-view parallel are proposed. The result shows proposed parallel optimization techniques, with insignificant loss to coding efficiency, significantly improves the execution time performance. This , in turn, proves modern parallel computing platforms, with appropriate platform-specific algorithm design, are valuable tools for improving the performance of computationally intensive applications

Methods and design issues for next generation network-aware applications

Networks are becoming an essential component of modern cyberinfrastructure and this work describes methods of designing distributed applications for high-speed networks to improve application scalability, performance and capabilities. As the amount of data generated by scientific applications continues to grow, to be able to handle and process it, applications should be designed to use parallel, distributed resources and high-speed networks. For scalable application design developers should move away from the current component-based approach and implement instead an integrated, non-layered architecture where applications can use specialized low-level interfaces. The main focus of this research is on interactive, collaborative visualization of large datasets. This work describes how a visualization application can be improved through using distributed resources and high-speed network links to interactively visualize tens of gigabytes of data and handle terabyte datasets while maintaining high quality. The application supports interactive frame rates, high resolution, collaborative visualization and sustains remote I/O bandwidths of several Gbps (up to 30 times faster than local I/O). Motivated by the distributed visualization application, this work also researches remote data access systems. Because wide-area networks may have a high latency, the remote I/O system uses an architecture that effectively hides latency. Five remote data access architectures are analyzed and the results show that an architecture that combines bulk and pipeline processing is the best solution for high-throughput remote data access. The resulting system, also supporting high-speed transport protocols and configurable remote operations, is up to 400 times faster than a comparable existing remote data access system. Transport protocols are compared to understand which protocol can best utilize high-speed network connections, concluding that a rate-based protocol is the best solution, being 8 times faster than standard TCP. An HD-based remote teaching application experiment is conducted, illustrating the potential of network-aware applications in a production environment. Future research areas are presented, with emphasis on network-aware optimization, execution and deployment scenarios

Älykäs sähköverkko Suomessa - Muuntamoautomaatio

Smart grid is an umbrella term, which describes an electricity grid, where next generation technologies connect all stakeholders with each other, in order to operate the system as efficiently and reliably as possible. For medium voltage network, smart grid means more distribution automation. Feeder automation, which is a part of distribution automation, refers to the control and monitoring of secondary substations and disconnector stations. Biggest benefit of feeder automation is related to fault management. The Electricity Mar-ket Act (2013) and the new regulation model (2016 – 2023) are both driving forward the feasibility of feeder automation. This thesis studied the current state of the Finnish smart medium voltage network by interviewing six large distribution companies. The interviews also investigated the companies’ opinions regarding the future of smart gird technologies, and opinions towards regulations driving smart grid technologies. The biggest, but still relatively minor, concern the Finnish distribution companies had with the current regulation and legislation, was the lack of flexibility in the ‘component value list’ in the regulation model. This lack of flexibility does not encourage large-scale investments towards new technologies, if the particular component is not on the ‘list’. The most common expectations of future smart grid technologies were related to better fault detection. The increase of PV production was not seen as a major issue in the coming years. This thesis also studied the feasibility of feeder automation. The feasibility study was conducted by a case study related to the optimum automation level for a predetermined net-work topology. All the parameters for this case study, such as length of the feeders, power demand and outage restoration time, are based on technical figures published by the Energy Authority. The price of the technology is based on the new regulation model’s ‘component value list’, published by the Energy Authority. The results were calculated for different fault frequency values. The optimum automation level for 1 fault/year was 22 %.Älykäs sähköverkko on sateenvarjotermi, joka kuvaa sähköverkkoa, jossa uuden sukupolven teknologiat yhdistävät sähkömarkkinoiden kaikki sidosryhmät keskenään, mahdollistaen tehokkaamman ja luotettavamman sähköjärjestelmän. Keskijännitejakeluverkolle älykkyys tarkoittaa automaatiota. Muuntamoautomaatio, joka on osa jakeluverkon automaatiota, viittaa muuntamoiden ja erotinasemien hallintaan ja monitorointiin. Suurin hyöty muuntamoautomaatiossa syntyy vian hallinnan kautta. Sähkömarkkinalaki (2013) ja uusi Valvontamenetelmä (2016 – 2023) molemmat ajavat eteenpäin muuntamoautomaation kannattavuutta. Tämä tutkielma tutki Suomen nykyistä älykästä keskijänniteverkkoa haastattelemalla kuutta suurta jakeluverkkoyhtiötä. Haastatteluilla tutkittiin myös jakeluverkkoyhtiöiden näkemyksiä ja mielipiteitä tulevaisuuden teknologioita ja nykyistä regulaatiota kohtaan. Yleisin, vaikkakin vähäinen, verkkoyhtiöiden kehitysehdotus liittyi valvontamenetelmän verkkokomponenttilistan jäykkyyteen. Jos uutta teknologiaa ei löydy kyseiseltä listalta, ei se kannusta kyseisen teknologian massa-asennukseen. Suurimmat odotukset uusiin älykkäisiin teknologioihin liittyi vian havaitsemiseen. Paikallisen aurinkosähkön tuotannon ei nähty aiheuttavan merkittäviä haasteita jakeluverkolle lähitulevaisuudessa. Tämä tutkielman tutki myös muuntamoautomaation kannattavuutta. Kannattavuusanalyysi tehtiin tapaustutkimuksen avulla, laskemalla optimaalisen automaatiotason eri vikatiheysarvoille. Tapaustutkimuksena käytettiin ennalta määrättyä verkkotopologiaa. Kaikki jakeluverkkoon liittyvät parametrit, kuten johtolähtöjen tehot, pituudet ja vian korjausaika, ovat laskennallisia keskiarvolukuja Energiaviraston julkaisemista teknillisistä tunnusluvuista. Muuntamoautomaation hintana on käytetty Valvontamenetelmän (2016 – 2023) verkkokomponenttilistan määrittämiä hintoja. Tapaustutkimuksessa vikatiheydelle 1 vika/vuosi laskettiin optimaaliseksi automaatiotasoksi 22 %

Bioactive conformational ensemble server and database. A public framework to speed up in silico drug discovery.

Modern high-throughput structure-based drug discovery algorithms consider ligand flexibility, but typically with low accuracy, which results in a loss of performance in the derived models. Here we present the Bioactive Conformational Ensemble (BCE) server and its associated database. The server creates conformational ensembles of drug-like ligands and stores them in the BCE database, where a variety of analyses are offered to the user. The workflow implemented in the BCE server combines enhanced sampling molecular dynamics with self-consistent reaction field quantum mechanics (SCRF/QM) calculations. The server automatizes all the steps to transform 1D or 2D representation of drugs into three dimensional molecules, which are then titrated, parametrized, hydrated and optimized before being subjected to Hamiltonian replica-exchange (HREX) molecular dynamics simulations. Ensembles are collected and subjected to a clustering procedure to derive representative conformers, which are then analyzed at the SCRF/QM level of theory. All structural data is organized in a noSQL database accessible through a graphical interface and in a programmatic manner through a REST API. The server allows the user to define a private workspace and offers a deposition protocol as well as input files for "in house" calculations in those cases where confidentiality is a must. The database and the associated server are available at https://mmb.irbbarcelona.org/BC

New approaches to data access in large-scale distributed system

Mención Internacional en el título de doctorA great number of scientific projects need supercomputing resources, such as, for example, those carried out in physics, astrophysics, chemistry, pharmacology, etc. Most of them generate, as well, a great amount of data; for example, a some minutes long experiment in a particle accelerator generates several terabytes of data. In the last years, high-performance computing environments have evolved towards large-scale distributed systems such as Grids, Clouds, and Volunteer Computing environments. Managing a great volume of data in these environments means an added huge problem since the data have to travel from one site to another through the internet. In this work a novel generic I/O architecture for large-scale distributed systems used for high-performance and high-throughput computing will be proposed. This solution is based on applying parallel I/O techniques to remote data access. Novel replication and data search schemes will also be proposed; schemes that, combined with the above techniques, will allow to improve the performance of those applications that execute in these environments. In addition, it will be proposed to develop simulation tools that allow to test these and other ideas without needing to use real platforms due to their technical and logistic limitations. An initial prototype of this solution has been evaluated and the results show a noteworthy improvement regarding to data access compared to existing solutions.Un gran número de proyectos científicos necesitan recursos de supercomputación como, por ejemplo, los llevados a cabo en física, astrofísica, química, farmacología, etc. Muchos de ellos generan, además, una gran cantidad de datos; por ejemplo, un experimento de unos minutos de duración en un acelerador de partículas genera varios terabytes de datos. Los entornos de computación de altas prestaciones han evolucionado en los últimos años hacia sistemas distribuidos a gran escala tales como Grids, Clouds y entornos de computación voluntaria. En estos entornos gestionar un gran volumen de datos supone un problema añadido de importantes dimensiones ya que los datos tienen que viajar de un sitio a otro a través de internet. En este trabajo se propondrá una nueva arquitectura de E/S genérica para sistemas distribuidos a gran escala usados para cómputo de altas prestaciones y de alta productividad. Esta solución se basa en la aplicación de técnicas de E/S paralela al acceso remoto a los datos. Así mismo, se estudiarán y propondrán nuevos esquemas de replicación y búsqueda de datos que, en combinación con las técnicas anteriores, permitan mejorar las prestaciones de aquellas aplicaciones que ejecuten en este tipo de entornos. También se propone desarrollar herramientas de simulación que permitan probar estas y otras ideas sin necesidad de recurrir a una plataforma real debido a las limitaciones técnicas y logísticas que ello supone. Se ha evaluado un prototipo inicial de esta solución y los resultados muestran una mejora significativa en el acceso a los datos sobre las soluciones existentes.Programa Oficial de Doctorado en Ciencia y Tecnología InformáticaPresidente: David Expósito Singh.- Secretario: María de los Santos Pérez Hernández.- Vocal: Juan Manuel Tirado Mart

Monitoring and Optimization of ATLAS Tier 2 Center GoeGrid

The demand on computational and storage resources is growing along with the amount of infor- mation that needs to be processed and preserved. In order to ease the provisioning of the digital services to the growing number of consumers, more and more distributed computing systems and platforms are actively developed and employed. The building block of the distributed computing infrastructure are single computing centers, similar to the Worldwide LHC Computing Grid, Tier 2 centre GoeGrid. The main motivation of this thesis was the optimization of GoeGrid perfor- mance by efficient monitoring. The goal has been achieved by means of the GoeGrid monitoring information analysis. The data analysis approach was based on the adaptive-network-based fuzzy inference system (ANFIS) and machine learning algorithm such as Linear Support Vector Machine (SVM). The main object of the research was the digital service, since availability, reliability and ser- viceability of the computing platform can be measured according to the constant and stable provisioning of the services. Due to the widely used concept of the service oriented architecture (SOA) for large computing facilities, in advance knowing of the service state as well as the quick and accurate detection of its disability allows to perform the proactive management of the com- puting facility. The proactive management is considered as a core component of the computing facility management automation concept, such as Autonomic Computing. Thus in time as well as in advance and accurate identification of the provided service status can be considered as a contribution to the computing facility management automation, which is directly related to the provisioning of the stable and reliable computing resources. Based on the case studies, performed using the GoeGrid monitoring data, consideration of the approaches as generalized methods for the accurate and fast identification and prediction of the service status is reasonable. Simplicity and low consumption of the computing resources allow to consider the methods in the scope of the Autonomic Computing component