10 research outputs found

    DAOS as HPC Storage: Exploring Interfaces

    Full text link
    This work in progress paper outlines research looking at the performance impact of using different storage interfaces to access the high performance object store DAOS. We demonstrate that using DAOS through a FUSE based filesystem interface can provide high performance, but there are impacts when choosing what I/O library or interface to utilises, with HDF5 exhibiting the highest impact. However, this varied depending on what type of I/O operations were undertaken

    StartR: a tool for large multi-dimensional data processing

    Get PDF
    Nowadays, the huge amount of data produced in various scientific domains has made data analysis challenging. In the climate science domain, with the constant increase of resolution in all possible dimensions of model output and the growing need for using computationally demanding analytical methodologies (e.g. bootstrapping), basic operations like extracting data from storage and performing statistical analysis on them must fulfill scientific and operational needs taking into account the growing volume and variety of data. A tool that facilitates data processing and leverages computational resources can largely save researchers’ time and effort. In this work, we introduce startR, an R package that allows to retrieve, arrange, and process large multi-dimensional datasets automatically with a concise workflow, smoothing the data-processing difficulty mentioned above

    Climate forecast analysis tools framework

    Get PDF
    The climate forecast analysis tools provide functions implementing the steps required for the analysis of sub-seasonal, seasonal and decadal forecast and operational climate services, allowing researchers to manipulate climate data and apply state-of-the-art methods taking advantage of the high-performance computational resources. Researchers can share their methods while reducing development and maintenance cost. An ecosystem of R packages covering these needs is under continuous development

    The CSTools (v4.0) toolbox : from climate forecasts to climate forecast information

    No full text
    Despite the wealth of existing climate forecast data, only a small part is effectively exploited for sectoral applications. A major cause of this is the lack of integrated tools that allow the translation of data into useful and skilful climate information. This barrier is addressed through the development of an R package. CSTools is an easy-to-use toolbox designed and built to assess and improve the quality of climate forecasts for seasonal to multi–annual scales. The package contains process-based state-of-the-art methods for forecast calibration, bias correction, statistical and stochastic downscaling, optimal forecast combination and multivariate verification, as well as basic and advanced tools to obtain tailored products. Due to the design of the toolbox in individual functions, the users can develop their own post-processing chain of functions as shown in the use cases presented in this manuscript: the analysis of an extreme wind speed event, the generation of seasonal forecasts of snow depth based on the SNOWPACK model and the post-processing of data to be used as input for the SCHEME hydrological model

    An R package for climate forecast verification

    No full text
    Forecast verification is necessary to determine the skill and quality of a forecasting system, and whether it shows improvement with pre- or post-processing. s2dverification v2.8.0 is an open-source R package for the quality assessment of climate forecasts using state-of-the-art verification scores. The package provides tools for each step of the forecast verification process: data retrieval, processing, calculation of verification measures and visualisation of the results. Examples are provided and explained for each of these stages using climate model output

    Climate Services Toolbox (CSTools) v4.0: from climate forecasts to climate forecast information

    Get PDF
    International audienceAbstract. Despite the wealth of existing climate forecast data, only a small part is effectively exploited for sectoral applications. A major cause of this is the lack of integrated tools that allow the translation of data into useful and skillful climate information. This barrier is addressed through the development of an R package. Climate Services Toolbox (CSTools) is an easy-to-use toolbox designed and built to assess and improve the quality of climate forecasts for seasonal to multi-annual scales. The package contains process-based, state-of-the-art methods for forecast calibration, bias correction, statistical and stochastic downscaling, optimal forecast combination, and multivariate verification, as well as basic and advanced tools to obtain tailored products. Due to the modular design of the toolbox in individual functions, the users can develop their own post-processing chain of functions, as shown in the use cases presented in this paper, including the analysis of an extreme wind speed event, the generation of seasonal forecasts of snow depth based on the SNOWPACK model, and the post-processing of temperature and precipitation data to be used as input in impact models
    corecore