Improving Performance of M-to-N Processing and Data Redistribution in In Transit Analysis and Visualization

In an in transit setting, a parallel data producer, such as a numerical simulation, runs on one set of ranks M, while a data consumer, such as a parallel visualization application, runs on a different set of ranks N. One of the central challenges in this in transit setting is to determine the mapping of data from the set of M producer ranks to the set of N consumer ranks. This is a challenging problem for several reasons, such as the producer and consumer codes potentially having different scaling characteristics and different data models. The resulting mapping from M to N ranks can have a significant impact on aggregate application performance. In this work, we present an approach for performing this M-to-N mapping in a way that has broad applicability across a diversity of data producer and consumer applications. We evaluate its design and performance with a study that runs at high concurrency on a modern HPC platform. By leveraging design characteristics, which facilitate an “intelligent” mapping from M-to-N, we observe significant performance gains are possible in terms of several different metrics, including time-to-solution and amount of data moved

Visualization, analysis, and design of COMBO-FISH probes in the grid-based GLOBE 3D genome platform

The genome architecture in cell nuclei plays an important role in modern microscopy for the monitoring of medical diagnosis and therapy since changes of function and dynamics of genes are interlinked with changing geometrical parameters. The planning of corresponding diagnostic experiments and their imaging is a complex and often interactive IT intensive challenge and thus makes high-performance grids a necessity. To detect genetic changes we recently developed a new form of fluorescence in situ hybridization (FISH) - COMBinatorial Oligonucleotide FISH (COMBO-FISH) - which labels small nucleotide sequences clustering at a desired genomic location. To achieve a unique hybridization spot other side clusters have to be excluded. Therefore, we have designed an interactive pipeline using the grid-based GLOBE 3D Genome Viewer and Platform to design and display different labelling variants of candidate probe sets. Thus, we have created a grid-based virtual "paper" tool for easy interactive calculation, analysis, management, and representation for COMBO-FISH probe design with many an advantage: Since all the calculations and analysis run in a grid, one can instantly and with great visual ease locate duplications of gene subsequences to guide the elimination of side clustering sequences during the probe design process, as well as get at least an impression of the 3D architectural embedding of the respecti

In situ visualization of large-scale turbulence simulations in Nek5000 with ParaView Catalyst

In situ visualization on high-performance computing systems allows us to analyze simulation results that would otherwise be impossible, given the size of the simulation data sets and offline post-processing execution time. We develop an in situ adaptor for Paraview Catalyst and Nek5000, a massively parallel Fortran and C code for computational fluid dynamics. We perform a strong scalability test up to 2048 cores on KTH’s Beskow Cray XC40 supercomputer and assess in situ visualization’s impact on the Nek5000 performance. In our study case, a high-fidelity simulation of turbulent flow, we observe that in situ operations significantly limit the strong scalability of the code, reducing the relative parallel efficiency to only ≈ 21 % on 2048 cores (the relative efficiency of Nek5000 without in situ operations is ≈ 99 %). Through profiling with Arm MAP, we identified a bottleneck in the image composition step (that uses the Radix-kr algorithm) where a majority of the time is spent on MPI communication. We also identified an imbalance of in situ processing time between rank 0 and all other ranks. In our case, better scaling and load-balancing in the parallel image composition would considerably improve the performance of Nek5000 with in situ capabilities. In general, the result of this study highlights the technical challenges posed by the integration of high-performance simulation codes and data-analysis libraries and their practical use in complex cases, even when efficient algorithms already exist for a certain application scenario

Socio-Economic Benefits from the Use of Earth Observations

This report summarizes the outcomes of the discussion of the workshop on Socioeconomic Benefit from the use of Earth Observation workshop held at JRC from 11 to 13 July 2011.JRC.H.6-Spatial data infrastructure