Hydro-meteorological data quality assurance and improvement

Abstract

Advances in measurement equipment and data transfer enabled easy and economic automatic monitoring of various hydro-meteorological variables. The main characteristic of such automatic monitoring systems is that they do not rely on human activities, but only on electronic devices. Even if those electronic devices are of highest quality and accuracy, and properly tuned to specific problem, the reliability of measured values relyieson many other factors and unexpected or undesired occurrences, like modification of measurement micro location, power supply shortages or surges, etc. The sampled and acquired data values have to be additionally checked, validated and sometimes improved or cleared before further use. This paper presents an innovative approach to data validation and improvement through the framework generally applicable to all hydrological data acquisition systems. The proposed framework can incorporate any number of validation methods and can be easily customized according to the characteristics of every single measured variable. The framework allows for the self-adjustment and feedback to support self-learning of used validation methods, same as expert-controlled learning and supervision. After data validation, for low-scored data, its value quality can be improved if redundant data exist, so framework has the data reconstruction module. By applying different interpolation techniques or using redundant data value the new data is created same as accompanying metadata with the reconstruction history. After data reconstruction, the framework supports the data adjustment, the post-processing phase where the data is adjusted for the specific needs of each user. Every validated and sometimes improved data value is accompanied with a meta-data that holds its validation grade as a quality indicator for further use.