2 research outputs found
Recommended from our members
Sequential data consistent inversion algorithms for parameter estimation in coastal storm surge models using high performance computing applications
Computational methods have gained significant prominence in addressing complex challenges across science and industry. These methods are grounded in physical principles and expressed in mathematical models, usually as a set of partial differential equations (PDEs), which are then discretized and solved, often times requiring High-Performance Computing (HPC) resources. Nevertheless, the efficacy of these computational methods is inherently tied to their ability to align predictions with observations and quantify uncertainties. This dissertation explores the critical intersection of three domains: Uncertainty Quantification (UQ), Storm-Surge Modeling, and High-Performance Computing (HPC) Applications for Ensemble Simulations (ES). The primary contribution is a set of novel algorithms for generating sequential parameter estimates and quantifying epistemic uncertainty in dynamical systems within a data-consistent (DC) framework.
In particular, data-constructed Quantity of Interest (QoI) maps are introduced using observed and simulated data to reduce uncertainty in model parameters and learn the optimal space to perform parameter inversion. A Maximal Updated Density (MUD) parameter estimate, similar to MAP estimate in Bayesian frameworks, is constructed using the DC update in a sequential context, with previous iterations informing subsequent ones, and with computational diagnostics within the DC framework providing critical information to both evaluate the quality of the DC update and MUD estimate as well as helping detect potential parameter value drifts.
The results presented showcase the potential of DC methods and MUD estimation in operational settings for analyzing and quantifying uncertainties in real- or near-real-time as packets of noisy observational data are obtained at discrete times from a network of sensors. In order to apply the sequential parameter estimation techniques presented, Ensemble Simulations (ES) of the high-fidelity forward models must be coordinated and executed in a reliable and reproducible manner.
Thus this dissertation also delves into the issue of HPC Applications for Ensemble Simulations, recognizing the emergence of complex workflows as the dominant form of research applications.
Developing these workflows presents its own set of challenges, including data staging, HPC system configuration, and job management. To address these challenges, the TACC Job Manager (taccjm) and related tools are introduced, enabling efficient interaction with HPC resources and the development of complex HPC workflows at the Texas Advanced Computing Center (TACC). Additionally, two versatile ES applications, PySLURM Task Queue (pyslurmtq) and tapis-pylauncher, are introduced as general purpose parametric job launchers for running ES. The culmination of this work is the application of the sequential DC algorithms to estimate wind drag parameters for a simulated extreme weather event using the high-fidelity storm-surge model ADCIRC. The results demonstrate how sequential DC algorithms cannot only effectively estimate parameters that match observations, but also inform decision makers on the quality of these parameter estimates and their sensitivity towards the dynamics of the system. Furthermore the application to reproduce the simulations on HPC systems is published via the DesignSafe cyber-infrastructure platform - a cloud based platform providing access to HPC compute resources and data storage - making the application openly available and reproducible.Computational Science, Engineering, and Mathematic
Contribution à la convergence d'infrastructure entre le calcul haute performance et le traitement de données à large échelle
The amount of produced data, either in the scientific community or the commercialworld, is constantly growing. The field of Big Data has emerged to handle largeamounts of data on distributed computing infrastructures. High-Performance Computing (HPC) infrastructures are traditionally used for the execution of computeintensive workloads. However, the HPC community is also facing an increasingneed to process large amounts of data derived from high definition sensors andlarge physics apparati. The convergence of the two fields -HPC and Big Data- iscurrently taking place. In fact, the HPC community already uses Big Data tools,which are not always integrated correctly, especially at the level of the file systemand the Resource and Job Management System (RJMS).In order to understand how we can leverage HPC clusters for Big Data usage, andwhat are the challenges for the HPC infrastructures, we have studied multipleaspects of the convergence: We initially provide a survey on the software provisioning methods, with a focus on data-intensive applications. We contribute a newRJMS collaboration technique called BeBiDa which is based on 50 lines of codewhereas similar solutions use at least 1000 times more. We evaluate this mechanism on real conditions and in simulated environment with our simulator Batsim.Furthermore, we provide extensions to Batsim to support I/O, and showcase thedevelopments of a generic file system model along with a Big Data applicationmodel. This allows us to complement BeBiDa real conditions experiments withsimulations while enabling us to study file system dimensioning and trade-offs.All the experiments and analysis of this work have been done with reproducibilityin mind. Based on this experience, we propose to integrate the developmentworkflow and data analysis in the reproducibility mindset, and give feedback onour experiences with a list of best practices.RésuméLa quantité de données produites, que ce soit dans la communauté scientifiqueou commerciale, est en croissance constante. Le domaine du Big Data a émergéface au traitement de grandes quantités de données sur les infrastructures informatiques distribuées. Les infrastructures de calcul haute performance (HPC) sont traditionnellement utilisées pour l’exécution de charges de travail intensives en calcul. Cependant, la communauté HPC fait également face à un nombre croissant debesoin de traitement de grandes quantités de données dérivées de capteurs hautedéfinition et de grands appareils physique. La convergence des deux domaines-HPC et Big Data- est en cours. En fait, la communauté HPC utilise déjà des outilsBig Data, qui ne sont pas toujours correctement intégrés, en particulier au niveaudu système de fichiers ainsi que du système de gestion des ressources (RJMS).Afin de comprendre comment nous pouvons tirer parti des clusters HPC pourl’utilisation du Big Data, et quels sont les défis pour les infrastructures HPC, nousavons étudié plusieurs aspects de la convergence: nous avons d’abord proposé uneétude sur les méthodes de provisionnement logiciel, en mettant l’accent sur lesapplications utilisant beaucoup de données. Nous contribuons a l’état de l’art avecune nouvelle technique de collaboration entre RJMS appelée BeBiDa basée sur 50lignes de code alors que des solutions similaires en utilisent au moins 1000 fois plus.Nous évaluons ce mécanisme en conditions réelles et en environnement simuléavec notre simulateur Batsim. En outre, nous fournissons des extensions à Batsimpour prendre en charge les entrées/sorties et présentons le développements d’unmodèle de système de fichiers générique accompagné d’un modèle d’applicationBig Data. Cela nous permet de compléter les expériences en conditions réellesde BeBiDa en simulation tout en étudiant le dimensionnement et les différentscompromis autours des systèmes de fichiers.Toutes les expériences et analyses de ce travail ont été effectuées avec la reproductibilité à l’esprit. Sur la base de cette expérience, nous proposons d’intégrerle flux de travail du développement et de l’analyse des données dans l’esprit dela reproductibilité, et de donner un retour sur nos expériences avec une liste debonnes pratiques