The present paper briefly describes a new modelling support tool (MoST) aimed at facilitating better quality assurance of the modelling process. MoST comprises a Knowledge Base with guidelines on good modelling practise for seven scientific domains. It supports multi-domain modelling and working in teams of different user types (water managers, modellers, auditors/reviewers, stakeholders and members of the public). The key functionality of MoST is to: (a) Guide to ensure that a model has been properly applied; (b) Monitor to record decisions, methods and data used in the modelling work and in this way enable transparency and reproducibility of the modelling process; (c) Report to provide suitable reports on what has been done by the various actors. MoST has been developed under the HarmoniQuA project (www.HarmoniQuA.org

Henriksen, H.J.

Højberg, A.L.

Kassahun, A.

Old, G.H.

Packman, J.C.

Refsgaard, J.C.

Scholten, H.

Name not available

Quality Assurance of the modelling process

Wageningen Yield 

QUALITY ASSURANCE OF THE MODELLING PROCESS
Jens Christian Refsgaard1, Anker Lajer Højberg1, Hans Jørgen Henriksen1, Huub
Scholten2, Ayalew Kassahun2, John Packman3 and Gareth Old3
1Geological Survey of Denmark and Greenland, jcr@geus.dk, Copenhagen, Denmark
2Wageningen University, Wageningen, The Netherlands
3Centre for Ecology and Hydrology, Wallingford, UK
ABSTRACT
The present paper briefly describes a new modelling support tool (MoST)
aimed at facilitating better quality assurance of the modelling process.
MoST comprises a Knowledge Base with guidelines on good modelling
practise for seven scientific domains. It supports multi-domain modelling
and working in teams of different user types (water managers, modellers,
auditors/reviewers, stakeholders and members of the public). The key
functionality of MoST is to: (a) Guide to ensure that a model has been
properly applied; (b) Monitor to record decisions, methods and data used in
the modelling work and in this way enable transparency and reproducibility
of the modelling process; (c) Report to provide suitable reports on what has
been done by the various actors. MoST has been developed under the
HarmoniQuA project (www.HarmoniQuA.org).
INTRODUCTION
During the last decade many problems have emerged in river basin modelling
projects, including poor quality of modelling, unrealistic expectations, and lack of
credibility of modelling results.  Some of the reasons for this lack of quality can be
evaluated (Refsgaard et al., 2005; Scholten et al., 2006) as the effect of:
• Ambiguous terminology and a lack of understanding between key-players
(modellers, clients, auditors, stakeholders and concerned members of the public)
• Bad practice (careless handling of input data, inadequate model set-up,
insufficient calibration/validation and model use outside of its scope)
• Lack of data or poor quality of available data
• Insufficient knowledge on the processes
• Poor communication between modellers and end-users on the possibilities and
limitations of the modelling project and overselling of model capabilities
• Confusion on how to use model results in decision making
• Lack of documentation and clarity on the modelling process, leading to results
that are difficult to audit or reproduce
• Insufficient consideration of economic, institutional and political issues and a
lack of integrated modelling.
In the water resources management community many different guidelines on good
modelling practice have been developed, see Refsgaard et al. (2005) for a review. One
of, if not the most, comprehensive example of a modelling guideline has been
developed in The Netherlands (Van Waveren et al., 1999) as a result of a process
involving all the main players in the Dutch water management field. The background
for this was a perceived need to improve the quality of modelling (Scholten et al.,
2000). Similarly, modelling guidelines for the Murray-Darling Basin in Australia were
developed due to the perception among end-users that model capabilities may have
been ‘over-sold’, and that there was a lack of consistency in approaches,
communication and understanding among and between the modellers and the water
managers, which often resulted in considerable uncertainty for decision making
(Middlemis, 2000).
The present paper briefly describes a new modelling support tool (MoST) aimed at
facilitating better quality assurance of the modelling process.
TERMINOLOGY AND SCIENTIFIC PHILOSOPHICAL BASIS
A key problem in relation to establishment of a theoretical modelling framework is
confusion on terminology (Refsgaard and Henriksen, 2004). For example the terms
validation and verifications are used with different, and some times interchangeable,
meaning by different authors. The confusion arises from both semantic and
philosophical considerations (Rykiel, 1996). Another important problem is the lack of
consensus related to the so far non-conclusive debate on the fundamental question
concerning whether a water resources model can be validated or verified, and whether
it as such can be claimed to be suitable or valid for particular applications (Konikow
and Bredehoeft, 1992; De Marsily et al., 1992; Oreskes et al., 1994). Therefore,
HarmoniQuA has developed an internally consistent terminology and a glossary
comprising more than 1000 terms.
The terminology and methodology are based on Refsgaard and Henriksen (2004).
The key elements in the terminology are illustrated in Fig. 1 and the most important
definitions are:
• A model code is a generic software program, which can be used for different
study areas without modifying the source code. 
• A model is a site application of a code to a particular study area, including input
data and parameter values.
• A model code can be verified. A code verification involves comparison of the
numerical solution generated by the code with one or more analytical solutions
or with other numerical solutions. Verification ensures that the computer
programme accurately solves the equations that constitute the mathematical
model.
• Model validation is here defined as the process of demonstrating that a given
site-specific model is capable of making accurate predictions for periods outside
a calibration period. A model is said to be validated if its accuracy and
predictive capability in the validation period have been proven to lie within
acceptable limits or errors. 
These terms are commonly used, although with differences in meaning between
authors. Our views on these terms and the ongoing discussion on validation-
falsification-confirmation as well as between the terms perceptual model, conceptual
model and site-specific model are given in Refsgaard and Henriksen (2004). Here we
just note that, from a Quality Assurance guideline point of view, it is fundamental for
us to make a clear distinction between the terms conceptual model, model code and
(site-specific) model. Furthermore, we never use the terms verification and validation
in a universal sense, but always restricted to clearly defined domains of applicability.
According to Popper (1959) we apply validation and verification in a ‘numerical
universal’ sense. We could call this conditional validation and conditional verification.
Fig. 1 Key elements of modelling terminology (Refsgaard and Henriksen, 2004)
THE KNOWLEDGE BASE 
Quality Assurance (QA) is defined as “the procedural and operational framework
used by an organisation managing the modelling study to build consensus among the
organisations concerned in its implementation, to assure technically and scientifically
adequate execution of all tasks included in the study, and to assure that all modelling-
based analysis is reproducible and justifiable”. This modification of the older NRC
(1990) definition includes the organisational, technical and scientific aspects, but in
addition the need to build consensus among the organisations concerned.
Guidelines for good modelling practise are included in the Knowledge Base (KB) of
MoST. The modelling process has been decomposed into five steps, see the flowchart
in Fig. 2. Each step includes several tasks. Each task has an internal structure i.e.
name, definition, explanation, interrelations with other tasks, activities, activity related
methods, references, sensitivity/pitfalls, task inputs and outputs.
The KB contains knowledge specific to seven domains (groundwater, precipitation-
runoff, river hydrodynamics, flood forecasting, water quality, ecology, and socio-
economics), and forms the heart of the tool. A computer based journal is produced
within MoST where the water manager and modelling team record the progress and
decisions made during a model study according to the tasks in the flowchart. This
record can be used when auditing the model study to judge its quality.
The most important QA principles incorporated in the KB are:
• The five modelling steps conclude with a formal dialogue between the modeller
and manager, where activities and results from the present step are reported, and
details of plans for the next step (a revised work plan) are discussed.
• External reviews are prescribed as the key mechanism of ensuring that the
knowledge and experience of other independent modellers are used.
• The KB provides public interactive guidelines to facilitate dialogue between
modellers and the water manager, with options to include auditors (reviewers),
stakeholders and the public. The options and approaches to use MoST as part of
a public participation process are described in Henriksen et al. (submitted).
• There are many feed back loops, some technical involving only the modeller,
and others that may require a decision before doing costly additional work.
• The KB allows performance and accuracy criteria to be updated during the
modelling process. In the first step the water manager’s objectives and
requirements are translated into performance criteria that may include
qualitative and quantitative measures. These criteria may be modified during the
formal reviews of subsequent steps.
• Emphasis is put on validation schemes, i.e. tests of model performance against
data that have not been used for model calibration.
• Uncertainties must be explicitly recognised and assessed (qualitatively and/or
quantitatively) throughout the modelling process.
MoST contains descriptions of 90 different Methods that can provide support to the
user on how to perform an activity. Examples illustrating the variety of Methods are
‘Monte Carlo Simulation’ comprising advise on how to perform Monte Carlo analyses
and ‘BMW Toolbox, which simply is a hyperlink (http://www.rbm-
toolbox.net/bmw/index.php) to guidance on selection of an appropriate model code
developed under another EU research project.
MOST – THE SUPPORTING SOFTWARE TOOL
MoST supports multi-domain studies and working in teams of different user types
(water managers, modellers, auditors, stakeholders and members of the public). The
associated glossary is accessible via hyperlinked text. The key functionality of MoST
is to
• Guide, to ensure a model has been properly applied. This is based on the
Knowledge Base.
• Monitor, to record decisions, methods and data used in the modelling work and
in this way enable transparency and reproducibility of the modelling process.
• Report, to provide suitable reports of what has been done for managers/clients,
modellers, auditors, stakeholders and the general public.
Fig. 2 The five steps and 48 tasks of the modelling process in the HarmoniQuA Knowledge Base
Specify or Update 
Calibration and 
Validation Targets
and Criteria
Test Runs
Completed
Model Set-up
Construct Model
OK
Dire
Dire
Not 
OK
OK
Not 
OKReview Model
Set-up and Calibration 
and Validation Plan
Report and Revisit Model 
Study Plan 
(Model Set-up)
Model Study Plan
Describe Problem
and Context
Proposal and 
Tendering
Prepare Terms of 
Reference
Determine 
Requirements
Identify Data 
Availability
Define
Objectives
Yes
No Agree on
Model Study Plan
and Budget
Describe System and
Data Availability
Collect and Process 
Raw Data
Data and 
Conceptualisation
Sufficient
Data?
Code Selection
Assess
Soundness of 
Conceptualis-
ation
Process Model
Structure Data
Model Structure and 
Processes
Need
for Alternative
Conceptual
Models?
Model Parameters
Summarise
Conceptual Model and 
Assumptions
No
Yes
Dire
Yes
No
Dire
OK
OK
Not 
OK
Not 
OK
Review  Data and 
Conceptualisation and 
Model Set-up Plan
Report and Revisit 
Model Study Plan 
(Data and 
Conceptualisation)
Validation
Uncertainty Analysis 
of Calibration and 
Validation
Scope of Applicability
Calibration
and Validation
Assess
Soundness of 
Validation
Assess
Soundness of 
Calibration
Specify Stages in 
Calibration Strategy
Select Calibration 
Method
Define Stop Criteria
Select Calibration 
Parameters
Parameter 
Estimation
All
Calibration Stages
Completed?
OK
Not 
OK
Yes
No
Dire
OK
Not 
OK
OK
Not 
OK
Not 
OK
OK
Review 
Calibration and Validation 
and Simulation Plan
Report and Revisit 
Model Study Plan 
(Calibration and 
Validation)
Dire
Analyse and Interpret 
Results
Uncertainty Analysis 
of Simulation
Simulations
Simulation
and Evaluation
Reporting of 
Simulation and 
Evaluation
Check
Simulations
Assess
Soundness of 
Simulattion
Need for Post Audit
Not 
OK
OK
Not 
OKDire
Not 
OK
OK
Review of
Simulation and 
Evaluation
OK
Model Study
Closure
Set-up Scenario
All Scenarios 
Completed?
No
Yes
Fig. 3 Screen dump of the MoST tool
The screenshot shown in Fig. 3 illustrates the main work window of MoST. It
shows the typical three panel layout under the Project tab for guiding and recording
work on a specific Task within the modelling flowchart. The left-hand panel shows the
sequence of Tasks completed or skipped, and highlights the current Task 2.4: Model
Structure and Processes (which forms part of Step 2: Data and Conceptualisation).
Note that Task 1.6: Proposal and Tendering has been skipped as the work is being
done ‘in-house”. The upper right-hand panel shows the (currently blank) model journal
for an Activity: Spatial resolution currently open under the Task. The user can enter
details of the actions and outcomes relating to this Activity, or can attach files or enter
references relevant to the Activity. If suggested methods are available they will be
listed to the right. The lower-right panel shows part of the guidance text on what the
Activity should address, with hyperlinks to glossary terms. Each panel has a scroll bar
and each can be resized.
ORGANISATIONAL REQUIREMENTS FOR QA GUIDELINES TO BE
EFFECTIVE
Modelling studies involve several parties with different responsibilities. The key
players are modellers and water managers, but often reviewers, stakeholders and the
general public are also involved. To a large extent the quality of the modelling study is
determined by the expertise, attitudes and motivation of the teams involved in the
modelling and QA process.
QA will only be successful if all parties actively support its use. The attitude of the
modellers is important. NRC (1990) characterises this as follows: “most modellers
enjoy the modelling process but find less satisfaction in the process of documentation
and quality assurance”. Scholten and Groot (2002) describe the main problem with the
Dutch Handbook on Good Modelling Practice as “they all like it, but only a few use
it”. The water manager, however, has a particular responsibility, because he/she has
the power to request and pay for adequate QA in modelling studies. Therefore, QA
guidelines can only be expected to be used in practice if the water manager prescribes
their use. It is therefore very important that the water manager has the technical
capacity to organise the QA process. Often, water managers do not have individuals
available with the appropriate training to understand and use models. An external
modelling expert should then be sought to help with the QA process. However, this
requires that the manager is aware of the problem and the need.
CONCLUSIONS
A software tool, MoST, with its associated Knowledge Base (KB) has been
developed by the HarmoniQuA project (Refsgaard and Henriksen, 2004; Refsgaard et
al., 2005, Scholten et al., 2006; Henriksen et al., submitted) to provide QA in the
modelling process through guidance, monitoring and reporting. MoST is unique as
compared to other existing modelling guidelines by providing support to a broad range
of disciplines required for river basin modelling and by facilitating a dialogue between
the various actors in the modelling process. Furthermore, it is to our knowledge the
only QA methodology that is supported by a comprehensive software tool.
ACKNOWLEDGEMENT
The present work was carried out within the Project ‘Harmonising Quality
Assurance in model based catchments and river basin management (HarmoniQuA)’,
which was partly funded by the EC Energy, Environment and Sustainable
Development programme (Contract EVK1-CT2001-00097).
MoST can be downloaded from www.harmoniqua.org, where also more
information about the HarmoniQuA project can be found.
REFERENCES
De Marsily, G Combes P, Goblet P (1992) Comments on 'Ground-water models cannot be
validated', by Konikow LF, Bredehoeft, JD, Advances in Water Resources, 15, 367-369.
Henriksen HJ, Refsgaard JC, Højberg AL, Ferrand N, Gijsbers P, Scholten H (submitted)
Public participation in relation to quality assurance of water resources modelling
(HarmoniQuA)
Konikow LF, Bredehoeft JD (1992) Ground-water models cannot be validated. Advances in
Water Resources, 15, 75-83.
Middlemis H (2000) Murray-Darling Basin Commission. Groundwater flow modelling
guideline. Aquaterra Consulting Pty Ltd., South Perth. Western Australia. Project no. 125.
NRC (1990) Ground Water Models: Scientific and Regulatory Applications. National
Research Council, National Academy Press, Washington, D.C.
Oreskes N, Shrader-Frechette K, Belitz K (1994) Verification, validation and confirmation of
numerical models in the earth sciences. Science, 264, 641-646.
Popper KR (1959) The logic of scientific discovery. Hutchingson & Co, London.
Refsgaard JC, Henriksen HJ (2004) Modelling guidelines – terminology and guiding
principles. Advances in Water Resources, 27, 71-82.
Refsgaard JC, Henriksen HJ, Harrar WG, Scholten H, Ayalew K (2005) Quality assurance in
model based water management – review of existing practice and outline of new
approaches. Environmental Modelling & Software, 20, 1201-1215.
Rykiel ER (1996) Testing ecological models: the meaning of validation. Ecological
Modelling, 90, 229-244.
Scholten H, Groot S (2002) Dutch guidelines. In: Refsgaard, JC (Ed) State-of-the-Art Report
on Quality Assurance in modelling related to river basin management. Chapter 12,
Geological Survey of Denmark and Greenland, Copenhagen. www.harmoniqua.org.
Scholten H, Van Waveren RH, Groot S, Van Geer FC, Wösten JHM, Koeze RD, Noort JJ
(2000) Good Modelling Practice in water management. Paper presented on
Hydroinformatics 2000, Cedar Rapids, IA, USA.
Scholten H, Kassahun A, Refsgaard JC, Kargas T, Gavardinas C, Beulens AJM (2006) A
methodology to support multidisciplinary model-based water management. Environmental
Modelling & Software accepted.
Van Waveren RH, Groot S, Scholten H, Van Geer FC, Wösten JHM, Koeze RD, Noort JJ
(1999) Good Modelling Practice Handbook, STOWA Report 99-05, Utrecht, RWS-RIZA,
Lelystad, The Netherlands, http://informatics.wur.nl/research%20projects/pub-pdf/gmp.pdf
(UK version)


NARCIS 

Quality Assurance of the modelling process

Abstract

Similar works

Full text

Available Versions

Name not available

Wageningen Yield

NARCIS