168,127 research outputs found
Post-processing and visualisation of large-scale DEM simulation data with the open-source VELaSSCo platform
Regardless of its origin, in the near future the challenge will not be how to generate data, but rather how to manage big and highly distributed
data to make it more easily handled and more accessible by users on their personal devices. VELaSSCo (Visualization for Extremely Large-Scale
Scientific Computing) is a platform developed to provide new visual analysis methods for large-scale simulations serving the petabyte era. The
platform adopts Big Data tools/architectures to enable in-situ processing for analytics of engineering and scientific data and
hardware-accelerated interactive visualization. In large-scale simulations, the domain is partitioned across several thousand nodes, and the data
(mesh and results) are stored on those nodes in a distributed manner. The VELaSSCo platform accesses this distributed information, processes
the raw data, and returns the results to the users for local visualization by their specific visualization clients and tools. The global goal of
VELaSSCo is to provide Big Data tools for the engineering and scientific community, in order to better manipulate simulations with billions of
distributed records. The ability to easily handle large amounts of data will also enable larger, higher resolution simulations, which will allow the
scientific and engineering communities to garner new knowledge from simulations previously considered too large to handle. This paper shows,
by means of selected Discrete Element Method (DEM) simulation use cases, that the VELaSSCo platform facilitates distributed post-processing
and visualization of large engineering datasets
New technologies to bridge the gap between High Performance Computing (HPC) and Big Data
The unification of HPC and Big Data has received increasing attention in the
last years. It is a common belief that exascale computing and Big Data are closely associated since HPC requires
processing large-scale data from scientific instruments and simulations. But, at the same time, it was observed that
tools and cultures of HPC and Big Data communities differ significantly. One of the most important issues in the
path to the convergence is caused by the differences in their software stacks. This thesis will address the research
challenge of bridging the gap between Big Data and HPC worlds. With this goal in mind, a set of tools and
technologies will be developed and integrated into a new unified Big Data-HPC framework that will allow the
execution of scientific multi-language applications on both environments using containers
Building a scientific workflow framework to enable realâtime machine learning and visualization
Nowadays, we have entered the era of big data. In the area of high performance computing, largeâscale simulations can generate huge amounts of data with potentially critical information. However, these data are usually saved in intermediate files and are not instantly visible until advanced data analytics techniques are applied after reading all simulation data from persistent storages (eg, local disks or a parallel file system). This approach puts users in a situation where they spend long time on waiting for running simulations while not knowing the status of the running job. In this paper, we build a new computational framework to couple scientific simulations with multiâstep machine learning processes and inâsitu data visualizations. We also design a new scalable simulationâtime clustering algorithm to automatically detect fluid flow anomalies. This computational framework is built upon different software components and provides plugâin data analysis and visualization functions over complex scientific workflows. With this advanced framework, users can monitor and get realâtime notifications of special patterns or anomalies from ongoing extremeâscale turbulent flow simulations
Big Data Analytics for Earth Sciences: the EarthServer approach
Big Data Analytics is an emerging field since massive storage and computing capabilities have been made available by advanced e-infrastructures. Earth and Environmental sciences are likely to benefit from Big Data Analytics techniques supporting the processing of the large number of Earth Observation datasets currently acquired and generated through observations and simulations. However, Earth Science data and applications present specificities in terms of relevance of the geospatial information, wide heterogeneity of data models and formats, and complexity of processing. Therefore, Big Earth Data Analytics requires specifically tailored techniques and tools. The EarthServer Big Earth Data Analytics engine offers a solution for coverage-type datasets, built around a high performance array database technology, and the adoption and enhancement of standards for service interaction (OGC WCS and WCPS). The EarthServer solution, led by the collection of requirements from scientific communities and international initiatives, provides a holistic approach that ranges from query languages and scalability up to mobile access and visualization. The result is demonstrated and validated through the development of lighthouse applications in the Marine, Geology, Atmospheric, Planetary and Cryospheric science domains
Large-scale grid-enabled lattice-Boltzmann simulations of complex fluid flow in porous media and under shear
Well designed lattice-Boltzmann codes exploit the essentially embarrassingly
parallel features of the algorithm and so can be run with considerable
efficiency on modern supercomputers. Such scalable codes permit us to simulate
the behaviour of increasingly large quantities of complex condensed matter
systems. In the present paper, we present some preliminary results on the large
scale three-dimensional lattice-Boltzmann simulation of binary immiscible fluid
flows through a porous medium derived from digitised x-ray microtomographic
data of Bentheimer sandstone, and from the study of the same fluids under
shear. Simulations on such scales can benefit considerably from the use of
computational steering and we describe our implementation of steering within
the lattice-Boltzmann code, called LB3D, making use of the RealityGrid steering
library. Our large scale simulations benefit from the new concept of capability
computing, designed to prioritise the execution of big jobs on major
supercomputing resources. The advent of persistent computational grids promises
to provide an optimal environment in which to deploy these mesoscale simulation
methods, which can exploit the distributed nature of compute, visualisation and
storage resources to reach scientific results rapidly; we discuss our work on
the grid-enablement of lattice-Boltzmann methods in this context.Comment: 17 pages, 6 figures, accepted for publication in
Phil.Trans.R.Soc.Lond.
Complex systems simulations to develop agency and citizenship skills through science education
In the era of big data, the progressively more widespread use of computational and data-intensive approaches is leading to changes in the ways of doing science and conducting research. The new methodologies and techniques are routine for researchers and professionals who make everyday use of big data analytics or simulations tools but are mainly unknown to ordinary people. Nevertheless, the impact of computational and data-intensive approaches has gone far beyond the scientific community, reaching the entire society. Indeed, the applications of machine learning and big data analytics, as well as the results and methods of computational simulations have reached peopleâs life and behaviour and, even more importantly, are at the methodological core of studies on urgent issues like the climate change or the pandemic, on which policymakers and citizens have to make decisions. Hence, the educational community cannot ignore the ongoing transformation of all peopleâs lives, behaviors, and culture. Within the research field of education to data science and computation, this dissertation addresses the issue of introducing in teaching-learning activities one of the methods of the on-going data science revolution: the computational simulations. Addressing the conceptual, methodological, and epistemological novelty of these objects, we will show how they embed, in a very specific, disciplinary-grounded way, the paradigm shift and cultural revolution of the data science age. We do that using lenses that come from the science of complexity, with its key-ideas that, originated from the physical modelling, can be applied to the analysis of a range of different phenomena. In the dissertation, we will guide the readers to recognize how dealing with simulations not only requires technical competences of coding, but a change of mindset and ways to think about the problems and the scientific method to address them
Research and Education in Computational Science and Engineering
Over the past two decades the field of computational science and engineering
(CSE) has penetrated both basic and applied research in academia, industry, and
laboratories to advance discovery, optimize systems, support decision-makers,
and educate the scientific and engineering workforce. Informed by centuries of
theory and experiment, CSE performs computational experiments to answer
questions that neither theory nor experiment alone is equipped to answer. CSE
provides scientists and engineers of all persuasions with algorithmic
inventions and software systems that transcend disciplines and scales. Carried
on a wave of digital technology, CSE brings the power of parallelism to bear on
troves of data. Mathematics-based advanced computing has become a prevalent
means of discovery and innovation in essentially all areas of science,
engineering, technology, and society; and the CSE community is at the core of
this transformation. However, a combination of disruptive
developments---including the architectural complexity of extreme-scale
computing, the data revolution that engulfs the planet, and the specialization
required to follow the applications to new frontiers---is redefining the scope
and reach of the CSE endeavor. This report describes the rapid expansion of CSE
and the challenges to sustaining its bold advances. The report also presents
strategies and directions for CSE research and education for the next decade.Comment: Major revision, to appear in SIAM Revie
- âŠ