Besseron, Xavier

Open Repository and Bibliography - Luxembourg

14th International SuperComputing Camphttps://sc-camp.orgInstalling Scientific Software on HPCwith EasyBuildSC-Camp 2023Cartagena, ColombiaMay 14-20, 2023Xavier BesseronUniversity of LuxembourgLuXDEM Research Teamhttps://luxdem.uni.lu2X. Besseron              Installing Scientific Software with EasyBuild SC-Camp 2023Outline● Part 1: Short Introduction to EasyBuild● Part 2: EasyBuild at workThis tutorial is based on EasyBuild tutorials available at http://tutorial.easybuild.io/.3X. Besseron              Installing Scientific Software with EasyBuild SC-Camp 2023Short Introduction to EasyBuild4X. Besseron              Installing Scientific Software with EasyBuild SC-Camp 2023What is EasyBuild?● Software build and installation framework ● Targeted to scientific software● Tailored towards High Performance Computing systems● Focus points: Performance, Reproducibility, CollaborationKey features● Fully autonomous installation● No admin privileges are required● Highly configurableObjective → Empower scientists to self-manage their software stack on HPCAlternative to EasyBuild → Spack https://spack.io/5X. Besseron              Installing Scientific Software with EasyBuild SC-Camp 2023Supported softwareEasyBuild supports 2995 different software packages (incl. toolchains, bundles):ABAQUS   AlphaFold   Amber   ANSYS   Blender   Boost   Caffe   Cantera   CGAL   COMSOL   Eigen   FastQC   GROMACS   gmsh   HDF5   Horovod   Hypre   Julia   JupyterLap   LAMMPS   MATLAB   matplotlib  METIS   MOOSE   MUMPS   NAMD   Nektar++   netCFD  OpenBLAS   OpenFOAM   OSPRay   pandas   PAPI   ParaView   PETSc   preCICE   PyTorch   R   RSEM   Rstudio-Server   SAGE   SAMTools   Scalasca   scikit-learn   scipy   Score-P   SQLite  STAR-CCM+ TensorFlow   Valgrind   Vampire   VASP   WRF   Yade   …Full list at https://docs.easybuild.io/version-specific/supported-software6X. Besseron              Installing Scientific Software with EasyBuild SC-Camp 2023Toolchains● Set of compilers and libraries used to build a software● Typically– compilers for C, C++ and Fortran, and possibly CUDA– libraries for MPI, linear algebra, fast fourier transform, etc.● Combined hierarchically to form bigger toolchains● Defined/released twice a year → 2019a, 2019b, 2020a, 2020b, 2021a, ...Common toolchains● System toolchain: OS compilers and libraries, used to bootstrap other toolchains● FOSS toolchain consists of open source components→ GCC, Open MPI, FlexiBLAS with OpenBLAS, ScaLAPACK, and FFTW● Intel toolchain based on Intel software→ Intel C, C++ and Fortran compilers, Intel MPI, and Intel MKL libraries7X. Besseron              Installing Scientific Software with EasyBuild SC-Camp 2023Example of toolchain: FOSS 2022bOpenMPI 4.1.4GCCcore 12.2.0binutils 2.39FlexiBLAS 3.2.1FFTW 3.3.10FFTW.MPI 3.3.10ScaLAPACK 2.2.0ToolchainsGCC-12.2.0gompi-2022bgfbf-2022bfoss-2022bSoftware8X. Besseron              Installing Scientific Software with EasyBuild SC-Camp 2023EasyConfig file → recipe to build a softwareNaming convention:     <name>-<version>-<toolchain><versionsuffix>.eb● <name> represents the software name;● <version> represents the software version;● <toolchain> represents the toolchain: <toolchain name>-<toolchain version>→ omitted for the system toolchain● <versionsuffix> is an optional name to represent the variants→ To keep in mind to search for easyconfig filesExamples    GCC-12.2.0.eb                     OpenMPI-4.1.4-GCC-12.2.0.ebHDF5-1.12.2-gompi-2022a.eb        PyTorch-1.12.1-foss-2022a.ebHDF5-1.12.2-GCC-11.3.0-serial.eb  PyTorch-1.12.1-foss-2022a-CUDA-11.7.0.eb9X. Besseron              Installing Scientific Software with EasyBuild SC-Camp 2023Software Installation and Environment ModulesEasyBuild will● install each software in its own sub-directory→ under <install prefix>/software/● create a module file to use the software→ under <install prefix>/modules/➔ The installation  prefix can be customized using the environment variable EASYBUILD_INSTALLPATH➔ Default install prefix in ~/.local/easybuild/➔ See https://docs.easybuild.io/configuration/10X. Besseron              Installing Scientific Software with EasyBuild SC-Camp 2023Question?11X. Besseron              Installing Scientific Software with EasyBuild SC-Camp 2023Practical session!EasyBuild at workHave a look at EasyBuild documentation https://docs.easybuild.io/ or eb --help to understand the meaning of command line parameters.12X. Besseron              Installing Scientific Software with EasyBuild SC-Camp 2023Getting started$ ssh USERNAME@toctoc.sc3.uis.edu.co$ ssh guane● Connect to Guane$ srun -N 1 -n 2 -c 1 --time 2:0:0 --pty bash● Get a computing node13X. Besseron              Installing Scientific Software with EasyBuild SC-Camp 2023Installing EasyBuild$ pip3 install --user easybuild● Install your own version of EasyBuild● More details on https://docs.easybuild.io/installation/ $ which eb$ eb --version$ eb --help● Check the installation14X. Besseron              Installing Scientific Software with EasyBuild SC-Camp 2023Configuring EasyBuild$ export EASYBUILD_INSTALLPATH=~/sccamp2023/easybuild● Installation path$ eb --show-config● Check the configuration➔ Software will be installed under ~/sccamp2023/easybuild/software➔ Modules will be installed under ~/sccamp2023/easybuild/modules➔ Many configuration variables → see eb –-help➔ Can be specified via a configuration file, environment variables or command line parameters➔ See https://docs.easybuild.io/configuration/15X. Besseron              Installing Scientific Software with EasyBuild SC-Camp 2023Searching for available software$ eb -S PyTorch$ eb -S ^PyTorch.*.eb$ eb -S ^PyTorch.*foss-2022.*.eb● Search for PyTorch$ eb --search scikit-learn$ eb --search scikit-learn.*2022b● Search for Scikit-Learn$ eb -S OSU$ eb -S OSU-Micro-Benchmarks.*2022b● Search for OSU Micro Benchmark16X. Besseron              Installing Scientific Software with EasyBuild SC-Camp 2023Checking dependencies and installations$ eb PyTorch-1.12.1-foss-2022a.eb -D$ eb PyTorch-1.12.1-foss-2022a.eb -x● What is required to install PyTorch?$ eb OSU-Micro-Benchmarks-6.2-gompi-2022b.eb -D● What is required to install OSU Micro Benchmarks?$ eb scikit-learn-1.2.1-gfbf-2022b.eb -D● What is required to install Scikit-Learn?17X. Besseron              Installing Scientific Software with EasyBuild SC-Camp 2023Building on top of other installations$ ls -l /home/xbesseron/sccamp-2023/easybuild/$ ls -l /home/xbesseron/sccamp-2023/easybuild/modules/$ ls -l /home/xbesseron/sccamp-2023/easybuild/software/$ module use /home/xbesseron/sccamp-2023/easybuild/modules/all/● Rely on software installed at another location$ eb OSU-Micro-Benchmarks-6.2-gompi-2022b.eb -D● Check again what is required to install OSU Micro Benchmarks$ eb scikit-learn-1.2.1-gfbf-2022b.eb -D● Check again what is required to install Scikit-Learn18X. Besseron              Installing Scientific Software with EasyBuild SC-Camp 2023Installing a software$ eb OSU-Micro-Benchmarks-6.2-gompi-2022b.eb● Install OSU Micro Benchmarks$ eb scikit-learn-1.2.1-gfbf-2022b.eb$ eb scikit-learn-1.2.1-gfbf-2022b.eb --robot● Install Scikit-Learn19X. Besseron              Installing Scientific Software with EasyBuild SC-Camp 2023Using an installed software$ module use ~/sccamp2023/easybuild/modules/all$ module avail● Update the MODULEPATH$ module avail OSU$ module show OSU-Micro-Benchmarks/6.2-gompi-2022b$ module load OSU-Micro-Benchmarks/6.2-gompi-2022b$ echo $EBROOTOSUMINMICROMINBENCHMARKS$ which osu_latency● Load the module for OSU Micro Benchmarks $ mpirun -n 2 osu_latency$ mpirun -n 2 osu_bibw● Load the module for OSU Micro Benchmarks 

Installing Scientific Software on HPC with EasyBuild

https://orbilu.uni.lu/bitstream/10993/55306/1/Installing_Scientific_Software_with_EasyBuild.pdf

Installing Scientific Software on HPC with EasyBuild

Abstract

Similar works

Full text

Available Versions

Open Repository and Bibliography - Luxembourg