Model-driven Scheduling for Distributed Stream Processing Systems

Distributed Stream Processing frameworks are being commonly used with the evolution of Internet of Things(IoT). These frameworks are designed to adapt to the dynamic input message rate by scaling in/out.Apache Storm, originally developed by Twitter is a widely used stream processing engine while others includes Flink, Spark streaming. For running the streaming applications successfully there is need to know the optimal resource requirement, as over-estimation of resources adds extra cost.So we need some strategy to come up with the optimal resource requirement for a given streaming application. In this article, we propose a model-driven approach for scheduling streaming applications that effectively utilizes a priori knowledge of the applications to provide predictable scheduling behavior. Specifically, we use application performance models to offer reliable estimates of the resource allocation required. Further, this intuition also drives resource mapping, and helps narrow the estimated and actual dataflow performance and resource utilization. Together, this model-driven scheduling approach gives a predictable application performance and resource utilization behavior for executing a given DSPS application at a target input stream rate on distributed resources.Comment: 54 page

TOWARDS AN EFFICIENT MULTI-CLOUD OBSERVABILITY FRAMEWORK OF CONTAINERIZED MICROSERVICES IN KUBERNETES PLATFORM

A recent trend in software development adopts the paradigm of distributed microservices architecture (MA). Kubernetes, a container-based virtualization platform, has become a de facto environment in which to run MA applications. Organizations may choose to run microservices at several cloud providers to optimize cost and satisfy security concerns. This leads to increased complexity, due to the need to observe the performance characteristics of distributed MA systems. Following a decision guidance models (DGM) approach, this research proposes a decentralized and scalable framework to monitor containerized microservices that run on same or distributed Kubernetes clusters. The framework introduces efficient techniques to gather, distribute, and analyze the observed runtime telemetry data. It offers extensible and cloud-agnostic modules that can exchange data by using a multiplexing, reactive, and non-blocking data streaming approach. An experiment to observe samples of microservices deployed across different cloud platforms was used as a method to evaluate the efficacy and usefulness of the framework. The proposed framework suggests an innovative approach to the development and operations (DevOps) practitioners to observe services across different Kubernetes platforms. It could also serve as a reference architecture for researchers to guide further design options and analysis techniques

A novel methodology for the assessment or wave energy opions at early stages

276 p.El aumento de la proporción de generación de electricidad a partir de fuentes renovables es clave para garantizar un sistema energético totalmente descarbonizado y luchar contra el cambio climático. La energía undimotriz es un recurso abundante pero, al mismo tiempo, es la menos desarrollada de todas las tecnologías renovables. El marco de evaluación común desarrollado en la tesis se basa en principios sólidos de ingeniería de sistemas y abarca el contexto externo, los requisitos del sistema y los criterios de evaluación. Se puede aplicar a diferentes niveles de madurez tecnológica y capta los aspectos cualitativos relacionados con las expectativas de las partes interesadas. El enfoque novedoso guía las decisiones de diseño a lo largo del proceso de desarrollo para la gestión adecuada del riesgo y la incertidumbre, y facilita la selección y evaluación comparativa de la tecnología undimotriz a diferentes niveles de madurez de manera controlada. Los métodos propuestos en esta investigación brindan información valiosa para enfocar los esfuerzos de innovación en aquellas áreas que tienen la mayor influencia en el desempeño de la tecnología. La incorporación de estrategias de innovación eficaces en el desarrollo de la energía undimotriz ayuda a gestionar la complejidad del sistema y canalizar la innovación hacia mejoras útiles.Tecnali

Quantitative Detection of Syntrophic Fatty Acid-degrading Bacterial Communities in Methanogenic Environments

In methanogenic habitats, volatile fatty acids (VFA), such as propionate and butyrate, are major intermediates in organic matter degradation. VFA are further metabolized to H2, acetate and CO2 by syntrophic fatty acid-degrading bacteria (SFAB) in association with methanogenic archaea. Despite their indispensable role in VFA degradation, little is known about SFAB abundance and their environmental distribution. To facilitate ecological studies, we developed four novel genus-specific quantitative PCR (qPCR) assays, with primer sets targeting known SFAB: Syntrophobacter, Smithella, Pelotomaculum and Syntrophomonas. Primer set specificity was confirmed using in silico and experimental (target controls, clone libraries and melt-curve analysis) approaches. These qPCR assays were applied to quantify SFAB in a variety of mesophilic methanogenic habitats, including a laboratory propionate enrichment culture, pilotand full-scale anaerobic reactors, cow rumen, horse faeces, an experimental rice paddy soil, a bog stream and swamp sediments. The highest SFAB 16S rRNA gene copy numbers were found in the propionate enrichment culture and anaerobic reactors, followed by the bog stream and swamp sediment samples. In addition, it was observed that SFAB and methanogen abundance varied with reactor configuration and substrate identity. To our knowledge, this research represents the first comprehensive study to quantify SFAB in methanogenic habitats using qPCR-based methods. These molecular tools will help investigators better understand syntrophic microbial communities in engineered and natural environments

Efficient openMP over sequentially consistent distributed shared memory systems

Nowadays clusters are one of the most used platforms in High Performance Computing and most programmers use the Message Passing Interface (MPI) library to program their applications in these distributed platforms getting their maximum performance, although it is a complex task. On the other side, OpenMP has been established as the de facto standard to program applications on shared memory platforms because it is easy to use and obtains good performance without too much effort. So, could it be possible to join both worlds? Could programmers use the easiness of OpenMP in distributed platforms? A lot of researchers think so. And one of the developed ideas is the distributed shared memory (DSM), a software layer on top of a distributed platform giving an abstract shared memory view to the applications. Even though it seems a good solution it also has some inconveniences. The memory coherence between the nodes in the platform is difficult to maintain (complex management, scalability issues, high overhead and others) and the latency of the remote-memory accesses which can be orders of magnitude greater than on a shared bus due to the interconnection network. Therefore this research improves the performance of OpenMP applications being executed on distributed memory platforms using a DSM with sequential consistency evaluating thoroughly the results from the NAS parallel benchmarks. The vast majority of designed DSMs use a relaxed consistency model because it avoids some major problems in the area. In contrast, we use a sequential consistency model because we think that showing these potential problems that otherwise are hidden may allow the finding of some solutions and, therefore, apply them to both models. The main idea behind this work is that both runtimes, the OpenMP and the DSM layer, should cooperate to achieve good performance, otherwise they interfere one each other trashing the final performance of applications. We develop three different contributions to improve the performance of these applications: (a) a technique to avoid false sharing at runtime, (b) a technique to mimic the MPI behaviour, where produced data is forwarded to their consumers and, finally, (c) a mechanism to avoid the network congestion due to the DSM coherence messages. The NAS Parallel Benchmarks are used to test the contributions. The results of this work shows that the false-sharing problem is a relative problem depending on each application. Another result is the importance to move the data flow outside of the critical path and to use techniques that forwards data as early as possible, similar to MPI, benefits the final application performance. Additionally, this data movement is usually concentrated at single points and affects the application performance due to the limited bandwidth of the network. Therefore it is necessary to provide mechanisms that allows the distribution of this data through the computation time using an otherwise idle network. Finally, results shows that the proposed contributions improve the performance of OpenMP applications on this kind of environments

Densification of spatially-sparse legacy soil data at a national scale: a digital mapping approach

Digital soil mapping (DSM) is a viable approach to providing spatial soil information but its adoption at the national scale, especially in sub-Saharan Africa, is limited by low spread of data. Therefore, the focus of this thesis is on optimizing DSM techniques for densification of sparse legacy soil data using Nigeria as a case study. First, the robustness of Random Forest model (RFM) was tested in predicting soil particle-size fractions as a compositional data using additive log-ratio technique. Results indicated good prediction accuracy with RFM while soils are largely coarse-textured especially in the northern region. Second, soil organic carbon (SOC) and bulk density (BD) were predicted from which SOC density and stock were calculated. These were overlaid with land use/land cover (LULC), agro-ecological zone (AEZ) and soil maps to quantify the carbon sequestration of soils and their variation across different AEZs. Results showed that 6.5 Pg C with an average of 71.60 Mg C ha–1 abound in the top 1 m soil depth. Furthermore, to improve the performance of BD and effective cation exchange capacity (ECEC) pedotransfer functions (PTFs), the inclusion of environmental data was explored using multiple linear regression (MLR) and RFM. Results showed an increase in performance of PTFs with the use of soil and environmental data. Finally, the application of Choquet fuzzy integral (CI) technique in irrigation suitability assessment was assessed. This was achieved through multi-criteria analysis of soil, climatic, landscape and socio-economic indices. Results showed that CI is a better aggregation operator compared to weighted mean technique. A total of 3.34 x 106 ha is suitable for surface irrigation in Nigeria while major limitations are due to topographic and soil attributes. Research findings will provide quantitative basis for framing appropriate policies on sustainable food production and environmental management, especially in resource-poor countries of the world

A looming revolution: Implications of self-generation for the risk exposure of retailers. ESRI WP597, September 2018

Managing the risk associated with uncertain load has always been a challenge for retailers in electricity markets. Yet the load variability has been largely predictable in the past, especially when aggregating a large number of consumers. In contrast, the increasing penetration of unpredictable, small-scale electricity generation by consumers, i.e. self-generation, constitutes a new and yet greater volume risk. Using value-at-risk metrics and Monte Carlo simulations based on German historical loads and prices, the contribution of decentralized solar PV self-generation to retailers’ load and revenue risks is assessed. This analysis has implications for the consumers’ welfare and the overall efficiency of electricity markets