LOOM: Interweaving tightly coupled visualization and numeric simulation framework

Traditional post-hoc high-fidelity scientific visualization (HSV) of numerical simulations requires multiple I/O check-pointing to inspect the simulation progress. The costs of these I/O operations are high and can grow exponentially with increasing problem sizes. In situ HSV dispenses with costly check-pointing I/O operations, but requires additional computing resources to generate the visualization, increasing power and energy consumption. In this paper we present LOOM, a new interweaving approach supported by a task scheduling framework to allow tightly coupled in situ visualization without significantly adding to the overall simulation runtime. The approach exploits the idle times of the numerical simulation threads, due to workload imbalances, to perform the visualization steps. Overall execution time (simulation plus visualization) is minimized. Power requirements are also minimized by sharing the same computational resources among numerical simulation and visualization tasks. We demonstrate that LOOM reduces time to visualization by 3 × compared to a traditional non-interwoven pipeline. Our results here demonstrate good potential for additional gains for large distributed-memory use cases with larger interleaving opportunities.This work was supported in part by National Funds through the Portuguese funding agency, FCT - Fundação para a Ciência e a Tecnologia, within project UIDB/50014/2020. This research was also supported in part by the Frontera computing project at the Texas Advanced Computing Center. Frontera is made possible by National Science Foundation award OAC-1818253

Heterogeneous hierarchical workflow composition

Workflow systems promise scientists an automated end-to-end path from hypothesis to discovery. However, expecting any single workflow system to deliver such a wide range of capabilities is impractical. A more practical solution is to compose the end-to-end workflow from more than one system. With this goal in mind, the integration of task-based and in situ workflows is explored, where the result is a hierarchical heterogeneous workflow composed of subworkflows, with different levels of the hierarchy using different programming, execution, and data models. Materials science use cases demonstrate the advantages of such heterogeneous hierarchical workflow composition.This work is a collaboration between Argonne National Laboratory and the Barcelona Supercomputing Center within the Joint Laboratory for Extreme-Scale Computing. This research is supported by the U.S. Department of Energy, Office of Science, Office of Advanced Scientific Computing Research, under contract number DE-AC02- 06CH11357, program manager Laura Biven, and by the Spanish Government (SEV2015-0493), by the Spanish Ministry of Science and Innovation (contract TIN2015-65316-P), by Generalitat de Catalunya (contract 2014-SGR-1051).Peer ReviewedPostprint (author's final draft

ATMSeer: Increasing Transparency and Controllability in Automated Machine Learning

To relieve the pain of manually selecting machine learning algorithms and tuning hyperparameters, automated machine learning (AutoML) methods have been developed to automatically search for good models. Due to the huge model search space, it is impossible to try all models. Users tend to distrust automatic results and increase the search budget as much as they can, thereby undermining the efficiency of AutoML. To address these issues, we design and implement ATMSeer, an interactive visualization tool that supports users in refining the search space of AutoML and analyzing the results. To guide the design of ATMSeer, we derive a workflow of using AutoML based on interviews with machine learning experts. A multi-granularity visualization is proposed to enable users to monitor the AutoML process, analyze the searched models, and refine the search space in real time. We demonstrate the utility and usability of ATMSeer through two case studies, expert interviews, and a user study with 13 end users.Comment: Published in the ACM Conference on Human Factors in Computing Systems (CHI), 2019, Glasgow, Scotland U

Enabling adaptive scientific workflows via trigger detection

Next generation architectures necessitate a shift away from traditional workflows in which the simulation state is saved at prescribed frequencies for post-processing analysis. While the need to shift to in~situ workflows has been acknowledged for some time, much of the current research is focused on static workflows, where the analysis that would have been done as a post-process is performed concurrently with the simulation at user-prescribed frequencies. Recently, research efforts are striving to enable adaptive workflows, in which the frequency, composition, and execution of computational and data manipulation steps dynamically depend on the state of the simulation. Adapting the workflow to the state of simulation in such a data-driven fashion puts extremely strict efficiency requirements on the analysis capabilities that are used to identify the transitions in the workflow. In this paper we build upon earlier work on trigger detection using sublinear techniques to drive adaptive workflows. Here we propose a methodology to detect the time when sudden heat release occurs in simulations of turbulent combustion. Our proposed method provides an alternative metric that can be used along with our former metric to increase the robustness of trigger detection. We show the effectiveness of our metric empirically for predicting heat release for two use cases.Comment: arXiv admin note: substantial text overlap with arXiv:1506.0825

PyFrac: A planar 3D hydraulic fracture simulator

Fluid driven fractures propagate in the upper earth crust either naturally or in response to engineered fluid injections. The quantitative prediction of their evolution is critical in order to better understand their dynamics as well as to optimize their creation. We present a Python implementation of an open-source hydraulic fracture propagation simulator based on the implicit level set algorithm originally developed by Peirce & Detournay (2008) -- "An implicit level set method for modeling hydraulically driven fractures". Comp. Meth. Appl. Mech. Engng, (33-40):2858--2885. This algorithm couples a finite discretization of the fracture with the use of the near tip asymptotic solutions of a steadily propagating semi-infinite hydraulic fracture. This allows to resolve the multi-scale processes governing hydraulic fracture growth accurately, even with relatively coarse meshes. We present an overview of the mathematical formulation, the numerical scheme and the details of our implementation. A series of problems including a radial hydraulic fracture verification benchmark, the propagation of a height contained hydraulic fracture, the lateral spreading of a magmatic dyke and the handling of fracture closure are presented to demonstrate the capabilities, accuracy and robustness of the implemented algorithm

Recovering complete and draft population genomes from metagenome datasets.

Assembly of metagenomic sequence data into microbial genomes is of fundamental value to improving our understanding of microbial ecology and metabolism by elucidating the functional potential of hard-to-culture microorganisms. Here, we provide a synthesis of available methods to bin metagenomic contigs into species-level groups and highlight how genetic diversity, sequencing depth, and coverage influence binning success. Despite the computational cost on application to deeply sequenced complex metagenomes (e.g., soil), covarying patterns of contig coverage across multiple datasets significantly improves the binning process. We also discuss and compare current genome validation methods and reveal how these methods tackle the problem of chimeric genome bins i.e., sequences from multiple species. Finally, we explore how population genome assembly can be used to uncover biogeographic trends and to characterize the effect of in situ functional constraints on the genome-wide evolution

Visualization of lithium-ion transport and phase evolution within and between manganese oxide nanorods.

Multiple lithium-ion transport pathways and local phase changes upon lithiation in silver hollandite are revealed via in situ microscopy including electron diffraction, imaging and spectroscopy, coupled with density functional theory and phase field calculations. We report unexpected inter-nanorod lithium-ion transport, where the reaction fronts and kinetics are maintained within the neighbouring nanorod. Notably, this is the first time-resolved visualization of lithium-ion transport within and between individual nanorods, where the impact of oxygen deficiencies is delineated. Initially, fast lithium-ion transport is observed along the long axis with small net volume change, resulting in two lithiated silver hollandite phases distinguishable by orthorhombic distortion. Subsequently, a slower reaction front is observed, with formation of polyphase lithiated silver hollandite and face-centred-cubic silver metal with substantial volume expansion. These results indicate lithium-ion transport is not confined within a single nanorod and may provide a paradigm shift for one-dimensional tunnelled materials, particularly towards achieving high-rate capability