2 research outputs found
Building a scientific workflow framework to enable realātime machine learning and visualization
Nowadays, we have entered the era of big data. In the area of high performance computing, largeāscale simulations can generate huge amounts of data with potentially critical information. However, these data are usually saved in intermediate files and are not instantly visible until advanced data analytics techniques are applied after reading all simulation data from persistent storages (eg, local disks or a parallel file system). This approach puts users in a situation where they spend long time on waiting for running simulations while not knowing the status of the running job. In this paper, we build a new computational framework to couple scientific simulations with multiāstep machine learning processes and ināsitu data visualizations. We also design a new scalable simulationātime clustering algorithm to automatically detect fluid flow anomalies. This computational framework is built upon different software components and provides plugāin data analysis and visualization functions over complex scientific workflows. With this advanced framework, users can monitor and get realātime notifications of special patterns or anomalies from ongoing extremeāscale turbulent flow simulations