This presentation gives an overview of Observing System Simulation Experiments (OSSEs). The components of an OSSE are described, along with discussion of the process for validating, calibrating, and performing experiments. a

Prive, Nikki

NASA Technical Reports Server

Observing System Simulation 
Experiments
Nikki Privé
30 July 2015
https://ntrs.nasa.gov/search.jsp?R=20150020824 2019-08-31T05:41:13+00:00Z
What is an OSSE?
●A long free model run is used as the “truth” - the Nature Run
●The Nature Run fields are used to back out “synthetic 
observations” from all current and new observing systems.
●Suitable errors are added to the synthetic observations
●The synthetic observations are assimilated into a different 
operational model
●Forecasts are made with the second model and compared 
with the Nature Run to quantify improvements due to the new 
observing system
An OSSE is a modeling experiment used to evaluate the impact 
of new observing systems on operational forecasts when 
actual observational data is not available.
Reality Data Analysis Forecasts
DAS Forecast  model
background
verification
observation
Nature 
Run
Synthetic data
Analysis Forecasts
DAS Forecast model
background
verification
interpolation
Real world forecasts
OSSE forecasts
OSSEs vs. the Real World
Why do an OSSE?
1.You want to find out if a new observing system will 
add value to NWP analyses and forecasts
2.You want to make design decisions for a new 
observing system
3.You want to investigate the behavior of data 
assimilation systems in an environment where the 
truth is known  
When not to run an OSSE
●When you can't model the phenomena you are 
interested in
●When you can't simulate your new observations
●When you can't assimilate your new 
observations
Nature Runs
● Nature Runs act as the 'truth' in the OSSE, 
replacing the real atmosphere.
● Usually, a long free (non-cycling) forecast from the 
best available model is used as the NR
● Model forecast has continuity of fields in time
● Sometimes an analysis or reanalysis sequence is 
used, but the sequence of states of truth can never 
be replicated by a model
● Always a push for bigger, higher resolution NR
Nature Run Requirements
●Must be able to realistically model phenomena 
of interest
–Dynamics and physics should be realistic
–Must produce fields needed for “observations”
–Should be verified against real world
●Ideally is ‘better’ than the operational model to 
be used for experiments
●Preferably a different model base is used for the 
NR and the experimental forecast model to 
reduce incestuousness
G5 Nature Run
2 year, 7 km/72L, 30 minute resolution
15 aerosols, ozone, CO,  CO2
Common Problems with Nature Runs
●Nonexistence
●Identical or fraternal twins
●Outdated by the time you get to use them
●Gigantic output files and huge computational 
resource requirements
–Output saved at full spatial resolution but 30 min 
+ intervals
Comparison of temporal and spatial
Interpolation errors compared to 1.5 
km run for Typhoon Guchol (2012).
Spatial interpolation to 27 kmFull resolution (1.5 km, 10 min)
Temporal interpolation (3 hrs)
Nature Run Validation
●Evaluate if NR is sufficiently realistic to yield 
meaningful results
●In addition to the phenomena of interest, the NR 
needs to realistically replicate fields needed to 
generate synthetic observations
●Can't validate everything; corollary – don't 
expect a NR to come pre-validated for your 
needs

Example of  AIRS observations channel 295 at 18 UTC 12 July
Real
Simulated
Synthetic Observations
Observation Errors
●Synthetic observations contain some intrinsic 
interpolation/operator errors, but less than real 
observations (usually)
●Synthetic errors are created and added to the 
synthetic observations to compensate
●Error is complex and poorly understood
●Error magnitude
●Biases
●Correlated errors
Calibration
●Adjust synthetic observations and their errors to 
increase realism of the OSSE in a statistical 
sense
●Compare OSSE statistics to statistics using real 
data in the same DAS/forecast system
●Need to decide what statistical metrics to use for 
the calibration, depending on your needs
●Calibrating new observation types?
●Find an analogous data type if possible
Observation count is easy to calibrate
O-F is fairly easy to calibrate because you can 
manipulate O directly.
Some observation correlations are relatively 
easy to calibrate
A-B (analysis increment) is a little harder to 
calibrate, as A and B are not directly controlled
Real OSSE
Latitude Latitude
Forecast errors are harder to calibrate, 
especially for longer forecasts. Matching of 
this statistic by manipulation of observations is 
difficult to impossible beyond ~24 hour 
forecasts.
Model error 
determines 
forecast skill in the 
longer term 
forecast, so 
calibration is not 
possible (unless 
you want to mess 
with your model).
Red: OSSE
Blue: Real
500 hPa anomaly correlations of
geopotential height
Why believe OSSE results?
New observations can be put into context relative to 
existing observation impacts
Criticisms of OSSEs
●Results only apply within the OSSE system – no 
concrete connection to the real world
●Even the best OSSEs are far from perfect: 
incestuousness, difficulty in generating 
observations and errors, deficiencies of the 
Nature Run
●By the time the new instrument is deployed, both 
the global observing network and the forecast 
models/DAS will be different
●Examples of sloppy or unsuccessful OSSEs
Common Pitfalls
●Very reduced baseline of assimilated 
observational data (ex. no radiance data)
●Other artificial degradation of analysis state
●No validation or calibration of OSSE framework
●Obtaining robust results from case studies is 
very challenging
–Use ensemble forecasts if you can!
Choosing Metrics
●Long cycling periods necessary to get 
statistically significant results for most new 
observations
●Anomaly correlation is a difficult metric to show 
appreciable impacts
●What fields do you expect the instrument to 
improve?
●Largest impacts found at analysis time or short-
term forecasts
Idealized Studies
●Identical twin experiments
●Idealized observations
●Manipulation of observation errors
●Experiments with B, R
●Make use of available “Truth”
Takeaway
●OSSEs can provide useful information about 
new observational types and the workings of data 
assimilation systems
●Careful consideration of research goals should 
guide each step of the OSSE process
●OSSEs are hard, good OSSEs are harder


Observing System Simulation Experiments

Abstract

Similar works

Full text

Available Versions

NASA Technical Reports Server