Some General and Fundamental Requirements for Designing Observing System Simulation Experiments (OSSEs)

There is an increasing demand to provide OSSE support when seeking funding for new atmospheric observing instruments. Various individuals and groups are running or developing OSSEs with little experience in OSSEs in particular or DA in general. In this presentation we will describe some key issues that are often neglected and some of the poor practices to be avoided. These include issues regarding NR and OSSE validation, consideration of instrument, observation operator, and forecast model error, relationships between observations and synoptic conditions, and conflicts of interest

Observing System Simulation Experiments for Fun and Profit

Observing System Simulation Experiments can be powerful tools for evaluating and exploring both the behavior of data assimilation systems and the potential impacts of future observing systems. With great power comes great responsibility - given a pure modeling framework, how can we be sure our results are meaningful? The challenges and pitfalls of OSSE calibration and validation will be addressed, as well as issues of incestuousness, selection of appropriate metrics, and experiment design. The use of idealized observational networks to investigate theoretical ideas in a fully complex modeling framework will also be discusse

The Role of Model and Initial Condition Error in Numerical Weather Forecasting Investigated with an Observing System Simulation Experiment

A series of experiments that explore the roles of model and initial condition error in numerical weather prediction are performed using an observing system simulation experiment (OSSE) framework developed at the National Aeronautics and Space Administration Global Modeling and Assimilation Office (NASA/GMAO). The use of an OSSE allows the analysis and forecast errors to be explicitly calculated, and different hypothetical observing networks can be tested with ease. In these experiments, both a full global OSSE framework and an 'identical twin' OSSE setup are utilized to compare the behavior of the data assimilation system and evolution of forecast skill with and without model error. The initial condition error is manipulated by varying the distribution and quality of the observing network and the magnitude of observation errors. The results show that model error has a strong impact on both the quality of the analysis field and the evolution of forecast skill, including both systematic and unsystematic model error components. With a realistic observing network, the analysis state retains a significant quantity of error due to systematic model error. If errors of the analysis state are minimized, model error acts to rapidly degrade forecast skill during the first 24-48 hours of forward integration. In the presence of model error, the impact of observation errors on forecast skill is small, but in the absence of model error, observation errors cause a substantial degradation of the skill of medium range forecasts

Introduction to Observing System Simulation Experiments (OSSEs)

This presentation gives a brief overview of Observing System Simulation Experiments (OSSEs), including what OSSEs are, and how and why they are performed. The intent is to educate the audience in light of the OSSE-related sections of the Forecast Improvement Act (H.R. 2413)

Status of the NASA GMAO Observing System Simulation Experiment

An Observing System Simulation Experiment (OSSE) is a pure modeling study used when actual observations are too expensive or difficult to obtain. OSSEs are valuable tools for determining the potential impact of new observing systems on numerical weather forecasts and for evaluation of data assimilation systems (DAS). An OSSE has been developed at the NASA Global Modeling and Assimilation Office (GMAO, Errico et al 2013). The GMAO OSSE uses a 13-month integration of the European Centre for Medium- Range Weather Forecasts 2005 operational model at T511/L91 resolution for the Nature Run (NR). Synthetic observations have been updated so that they are based on real observations during the summer of 2013. The emulated observation types include AMSU-A, MHS, IASI, AIRS, and HIRS4 radiance data, GPS-RO, and conventional types including aircraft, rawinsonde, profiler, surface, and satellite winds. The synthetic satellite wind observations are colocated with the NR cloud fields, and the rawinsondes are advected during ascent using the NR wind fields. Data counts for the synthetic observations are matched as closely as possible to real data counts, as shown in Figure 2. Errors are added to the synthetic observations to emulate representativeness and instrument errors. The synthetic errors are calibrated so that the statistics of observation innovation and analysis increments in the OSSE are similar to the same statistics for assimilation of real observations, in an iterative method described by Errico et al (2013). The standard deviations of observation minus forecast (xo-H(xb)) are compared for the OSSE and real data in Figure 3. The synthetic errors include both random, uncorrelated errors, and an additional correlated error component for some observational types. Vertically correlated errors are included for conventional sounding data and GPS-RO, and channel correlated errors are introduced to AIRS and IASI (Figure 4). HIRS, AMSU-A, and MHS have a component of horizontally correlated error. The forecast model used by the GMAO OSSE is the Goddard Earth Observing System Model, Version 5 (GEOS-5) with Gridpoint Statistical Interpolation (GSI) DAS. The model version has been updated to v. 5.13.3, corresponding to the current operational model. Forecasts are run on a cube-sphere grid with 180 points along each edge of the cube (approximately 0.5 degree horizontal resolution) with 72 vertical levels. The DAS is cycled at 6-hour intervals, with 240 hour forecasts launched daily at 0000 UTC. Evaluation of the forecasting skill for July and August is currently underway. Prior versions of the GMAO OSSE have been found to have greater forecasting skill than real world forecasts. It is anticipated that similar forecast skill will be found in the updated OSSE

Some General and Fundamental Requirements for Designing Observing System Simulation Experiments (OSSEs)

The intent of this white paper is to inform WMO projects and working groups, together with the broader weather research and general meteorology and oceanography communities, regarding the use of Observing System Simulation Experiments (OSSEs). This paper is not intended to be either a critical or cursory review of past OSSE efforts. Instead, it describes some fundamental, but often neglected, aspects of OSSEs and prescribes important caveats regarding their design, validation, and application. Well designed, properly validated, and carefully conducted OSSEs can be invaluable for examining, understanding, and estimating impacts of proposed observing systems and new data assimilation techniques. Although significant imperfections and limitations should be expected, OSSEs either profoundly complement or uniquely provide both qualitative and quantitative characterizations of potential analysis of components of the earth system

Use of an OSSE to Evaluate Background Error Covariances Estimated by the 'NMC Method'

The NMC method has proven utility for prescribing approximate background-error covariances required by variational data assimilation systems. Here, untunedNMCmethod estimates are compared with explicitly determined error covariances produced within an OSSE context by exploiting availability of the true simulated states. Such a comparison provides insights into what kind of rescaling is required to render the NMC method estimates usable. It is shown that rescaling of variances and directional correlation lengths depends greatly on both pressure and latitude. In particular, some scaling coefficients appropriate in the Tropics are the reciprocal of those in the Extratropics. Also, the degree of dynamic balance is grossly overestimated by the NMC method. These results agree with previous examinations of the NMC method which used ensembles as an alternative for estimating background-error statistics

Observing System Simulation Experiments: An Overview

An overview of Observing System Simulation Experiments (OSSEs) will be given, with focus on calibration and validation of OSSE frameworks. Pitfalls and practice will be discussed, including observation error characteristics, incestuousness, and experimental design. The potential use of OSSEs for investigation of the behaviour of data assimilation systems will be explored, including some results from experiments using the NASAGMAO OSSE

Observing System Simulation Experiments as Tools for Investigating the Behavior of Data Assimilation Systems

Data assimilation systems (DAS) are difficult to evaluate in part because there is limited independent data to use for verification of performance. In an Observing System Simulation Experiment (OSSE), the full true state is known exactly, in the form of the Nature Run. The availability of this truth allows the investigation of DAS characteristics in the OSSE framework that are not quantifiable in the real world. The synthetic observations can also be manipulated to test configurations that range from idealized to highly realistic. A sampling of OSSE investigations into the behavior of 3DVar and 4DEnVar DAS and adjoint observation impact estimation tools will be illustrated using the National Aeronautics and Space Administration Global Modeling and Assimilation Office (NASA/GMAO) OSSE

Development, Validation, and Application of OSSEs at NASA-GMAO

The GMAO OSSE framework has two general classes of applications. One is to estimate the potential improvements to weather forecasting and analysis by using new proposed instruments that are not as yet built or deployed. This exploits the simulated nature of the OSSE. The other is to assess various aspects of the GMAO data assimilation system. This exploits the availability of truth provided by the OSSE. Two examples of the first class of application will be offered. One concerns deployment of constellations of passive MW sounders on small CUBESATs placed in very low orbits. The other concern is increasing the frequency of radiosonde observations to 4-times daily at all current stations. Several examples of the second class will also be presented. One is an estimation of analysis error characteristics. Another is a comparison between covariances directly determined from explicitly known background errors and those estimated by computing differences between lagged forecasts using the NMC-method prior to tuning. A third is a comparison of effects of model and observation errors on analysis and forecast skill. The last is an examination of spectra of forecast errors in a study of predictability. This latter is an ongoing study