Elastic scaling for data stream processing

Cataloged from PDF version of article.This article addresses the profitability problem associated with auto-parallelization of general-purpose distributed data stream processing applications. Auto-parallelization involves locating regions in the application's data flow graph that can be replicated at run-time to apply data partitioning, in order to achieve scale. In order to make auto-parallelization effective in practice, the profitability question needs to be answered: How many parallel channels provide the best throughput? The answer to this question changes depending on the workload dynamics and resource availability at run-time. In this article, we propose an elastic auto-parallelization solution that can dynamically adjust the number of channels used to achieve high throughput without unnecessarily wasting resources. Most importantly, our solution can handle partitioned stateful operators via run-time state migration, which is fully transparent to the application developers. We provide an implementation and evaluation of the system on an industrial-strength data stream processing platform to validate our solution

Ephemeral Surface Chlorophyll Enhancement at the New England Shelf Break Driven by Ekman Restratification

The Mid-Atlantic Bight (MAB) hosts a large and productive marine ecosystem supported by high phytoplankton concentrations. Enhanced surface chlorophyll concentrations at the MAB shelf-break front have been detected in synoptic measurements, yet this feature is not present in seasonal means. To understand why, we assess the conditions associated with enhanced surface chlorophyll at the shelf break. We employ in-situ and remote sensing data, and a 2-dimensional model to show that Ekman restratification driven by upfront winds drives ephemerally enhanced chlorophyll concentrations at the shelf-break front in spring. Using 8-day composite satellite-measured surface chlorophyll concentration data from 2003–2020, we constructed a daily running mean (DRM) climatology of the cross-shelf chlorophyll distribution for the northern MAB region. While the frontal enhancement of chlorophyll is apparent in the DRM climatology, it is not captured in the seasonal climatology due to its short duration of less than a week. In-situ measurements of the frontal chlorophyll enhancement reveal that chlorophyll is highest in spring when the shelf-break front slumps offshore from its steep wintertime position causing restratification in the upper part of the water column. Several restratification mechanisms are possible, but the first day of enhanced chlorophyll at the shelf break corresponds to increasing upfront winds, suggesting that the frontal restratification is driven by offshore Ekman transport of the shelf water over the denser slope water. The 2-dimensional model shows that upfront winds can indeed drive Ekman restratification and alleviate light limitation of phytoplankton growth at the shelf-break front

Engaging and Empowering Student Representatives as Agents of Change at The University of Nottingham Ningbo China

The Learning Community Forum (LCF) at The University of Nottingham Ningbo China is a student-led forum where views regarding the provision of teaching and learning in general are collected and voiced. While the LCF may potentially identify issues that are otherwise overlooked by university-sanctioned surveys, it is confronted with various challenges. This paper investigates into the challenges and problems as encountered by the LCF student representatives by looking at two case studies: the LCFs of the Language Centre (LC) and of the Department of Mechanical, Materials and Manufacturing Engineering (MMME) at UNNC. The study aimed at identifying areas that worked well and those that needed improvement through interviews with student representatives to provide input for the design of a Nottingham Advantage Award (NAA) module. The NAA, as an instrument to address those identified challenges and problems, provides the following: 1) formal recognition of student contribution to teaching and learning, 2) training and relevant skills to empower student representatives as agents of change, and 3) increased future employability through encouraging active reflection on their experience

Safe data parallelism for general streaming

Streaming applications process possibly infinite streams of data and often have both high throughput and low latency requirements. They are comprised of operator graphs that produce and consume data tuples. General streaming applications use stateful, selective, and user-defined operators. The stream programming model naturally exposes task and pipeline parallelism, enabling it to exploit parallel systems of all kinds, including large clusters. However, data parallelism must either be manually introduced by programmers, or extracted as an optimization by compilers. Previous data parallel optimizations did not apply to selective, stateful and user-defined operators. This article presents a compiler and runtime system that automatically extracts data parallelism for general stream processing. Data-parallelization is safe if the transformed program has the same semantics as the original sequential version. The compiler forms parallel regions while considering operator selectivity, state, partitioning, and graph dependencies. The distributed runtime system ensures that tuples always exit parallel regions in the same order they would without data parallelism, using the most efficient strategy as identified by the compiler. Our experiments using 100 cores across 14 machines show linear scalability for parallel regions that are computation-bound, and near linear scalability when tuples are shuffled across parallel regions. © 2015 IEEE

Comparative analysis of diversity and environmental niches of soil bacterial, archaeal, fungal and protist communities reveal niche divergences along environmental gradients in the Alps

Although widely used in ecology, comparative analyses of diversity and niche properties are still lacking for microorganisms, especially focusing on niche variations. Quantifying the niches of microbial taxa is necessary to then forecast how taxa and the communities they compose might respond to environmental changes. In this study, we first identified important topoclimatic, edaphic, spatial and biotic drivers of the alpha and beta di-versity of bacterial, archaeal, fungal and protist communities. Then, we calculated the niche breadth and position of each taxon along the important environmental gradients to determine how these vary within and among the taxonomic groups. We found that edaphic properties were the most important drivers of both, community di-versity and composition, for all microbial groups. Protists and bacteria presented the largest niche breadths on average, followed by archaea, with fungi displaying the smallest. Niche breadth generally decreased towards environmental extremes, especially along edaphic gradients, suggesting increased specialization of microbial taxa in highly selective environments. Overall, we showed that microorganisms have well defined niches, as do macro-organisms, likely driving part of the observed spatial patterns of community variations. Assessing niche variation more widely in microbial ecology should open new perspectives, especially to tackle global change effects on microbes.Peer reviewe

Toward sharing brain images: Differentially private TOF-MRA images with segmentation labels using generative adversarial networks

Sharing labeled data is crucial to acquire large datasets for various Deep Learning applications. In medical imaging, this is often not feasible due to privacy regulations. Whereas anonymization would be a solution, standard techniques have been shown to be partially reversible. Here, synthetic data using a Generative Adversarial Network (GAN) with differential privacy guarantees could be a solution to ensure the patient's privacy while maintaining the predictive properties of the data. In this study, we implemented a Wasserstein GAN (WGAN) with and without differential privacy guarantees to generate privacy-preserving labeled Time-of-Flight Magnetic Resonance Angiography (TOF-MRA) image patches for brain vessel segmentation. The synthesized image-label pairs were used to train a U-net which was evaluated in terms of the segmentation performance on real patient images from two different datasets. Additionally, the Fréchet Inception Distance (FID) was calculated between the generated images and the real images to assess their similarity. During the evaluation using the U-Net and the FID, we explored the effect of different levels of privacy which was represented by the parameter ϵ. With stricter privacy guarantees, the segmentation performance and the similarity to the real patient images in terms of FID decreased. Our best segmentation model, trained on synthetic and private data, achieved a Dice Similarity Coefficient (DSC) of 0.75 for ϵ = 7.4 compared to 0.84 for ϵ = ∞ in a brain vessel segmentation paradigm (DSC of 0.69 and 0.88 on the second test set, respectively). We identified a threshold of ϵ <5 for which the performance (DSC <0.61) became unstable and not usable. Our synthesized labeled TOF-MRA images with strict privacy guarantees retained predictive properties necessary for segmenting the brain vessels. Although further research is warranted regarding generalizability to other imaging modalities and performance improvement, our results mark an encouraging first step for privacy-preserving data sharing in medical imaging

State of Emergency Medicine in Switzerland: a national profile of emergency departments in 2006

BACKGROUND: Emergency departments (EDs) are an essential component of any developed health care system. There is, however, no national description of EDs in Switzerland. Our objective was to establish the number and location of EDs, patient visits and flow, medical staff and organization, and capabilities in 2006, as a benchmark before emergency medicine became a subspecialty in Switzerland. METHODS: In 2007, we started to create an inventory of all hospital-based EDs with a preliminary list from the Swiss Society of Emergency and Rescue Medicine that was improved with input from ED physicians nationwide. EDs were eligible if they offered acute care 24 h per day, 7 days per week. Our goal was to have 2006 data from at least 80% of all EDs. The survey was initiated in 2007 and the 80% threshold reached in 2012. RESULTS: In 2006, Switzerland had a total of 138 hospital-based EDs. The number of ED visits was 1.475 million visits or 20 visits per 100 inhabitants. The median number of visits was 8,806 per year; 25% of EDs admitted 5,000 patients or less, 31% 5,001-10,000 patients, 26% 10,001-20,000 patients, and 17% &gt;20,000 patients per year. Crowding was reported by 84% of EDs with &gt;20,000 visits/year. Residents with limited experience provided care for 77% of visits. Imaging was not immediately available for all patients: standard X-ray within 15 min (70%), non-contrast head CT scan within 15 min (38%), and focused sonography for trauma (70%); 67% of EDs had an intensive care unit within the hospital, and 87% had an operating room always available. CONCLUSIONS: Swiss EDs were significant providers of health care in 2006. Crowding, physicians with limited experience, and the heterogeneity of emergency care capabilities were likely threats to the ubiquitous and consistent delivery of quality emergency care, particularly for time-sensitive conditions. Our survey establishes a benchmark to better understand future improvements in Swiss emergency care

Predicting spatial patterns of soil bacteria under current and future environmental conditions

Soil bacteria are largely missing from future biodiversity assessments hindering comprehensive forecasts of ecosystem changes. Soil bacterial communities are expected to be more strongly driven by pH and less by other edaphic and climatic factors. Thus, alkalinisation or acidification along with climate change may influence soil bacteria, with subsequent influences for example on nutrient cycling and vegetation. Future forecasts of soil bacteria are therefore needed. We applied species distribution modelling (SDM) to quantify the roles of environmental factors in governing spatial abundance distribution of soil bacterial OTUs and to predict how future changes in these factors may change bacterial communities in a temperate mountain area. Models indicated that factors related to soil (especially pH), climate and/or topography explain and predict part of the abundance distribution of most OTUs. This supports the expectations that microorganisms have specific environmental requirements (i.e., niches/envelopes) and that they should accordingly respond to environmental changes. Our predictions indicate a stronger role of pH over other predictors (e.g. climate) in governing distributions of bacteria, yet the predicted future changes in bacteria communities are smaller than their current variation across space. The extent of bacterial community change predictions varies as a function of elevation, but in general, deviations from neutral soil pH are expected to decrease abundances and diversity of bacteria. Our findings highlight the need to account for edaphic changes, along with climate changes, in future forecasts of soil bacteria.Peer reviewe