Exploring practical estimates of the ensemble size necessary for particle filters

Author Posting. © American Meteorological Society, 2015. This article is posted here by permission of American Meteorological Society for personal use, not for redistribution. The definitive version was published in Monthly Weather Review 144 (2016): 861-875, doi:10.1175/MWR-D-14-00303.1.Particle filtering methods for data assimilation may suffer from the “curse of dimensionality,” where the required ensemble size grows rapidly as the dimension increases. It would, therefore, be useful to know a priori whether a particle filter is feasible to implement in a given system. Previous work provides an asymptotic relation between the necessary ensemble size and an exponential function of , a statistic that depends on observation-space quantities and that is related to the system dimension when the number of observations is large; for linear, Gaussian systems, the statistic can be computed from eigenvalues of an appropriately normalized covariance matrix. Tests with a low-dimensional system show that these asymptotic results remain useful when the system is nonlinear, with either the standard or optimal proposal implementation of the particle filter. This study explores approximations to the covariance matrices that facilitate computation in high-dimensional systems, as well as different methods to estimate the accumulated system noise covariance for the optimal proposal. Since may be approximated using an ensemble from a simpler data assimilation scheme, such as the ensemble Kalman filter, the asymptotic relations thus allow an estimate of the ensemble size required for a particle filter before its implementation. Finally, the improved performance of particle filters with the optimal proposal, relative to those using the standard proposal, in the same low-dimensional system is demonstrated.Slivinski was supported by the NSF through Grants DMS-0907904 and DMS-1148284, by ONR through DOD (MURI) Grant N000141110087, and by NCAR’s Advanced Study Program during a collaborative visit to NCAR.2016-05-1

An evaluation of the performance of the 20th Century Reanalysis

The performance of a new historical reanalysis, the NOAA-CIRES-DOE 20th Century Reanalysis Version 3 (20CRv3), is evaluated via comparisons with other reanalyses and independent observations.&nbsp; This dataset provides global, 3-hourly estimates of the atmosphere from 1806 to 2015 by assimilating only surface pressure observations and prescribing sea surface temperature, sea ice concentration, and radiative forcings. Comparisons with independent observations, other reanalyses, and satellite products suggest that 20CRv3 can reliably produce atmospheric estimates on scales ranging from weather events to long-term climatic trends.&nbsp;&nbsp;Not only does 20CRv3 recreate a &ldquo;best estimate&rdquo; of the weather, including extreme events, it also provides an estimate of its confidence through the use of an ensemble. Surface pressure statistics suggest that these confidence estimates are reliable. Comparisons with independent upper-air observations in the Northern Hemisphere demonstrate that 20CRv3 has skill throughout the 20th century. Upper-air fields from 20CRv3 in the late 20th century and early 21st century correlate well with full-input reanalyses, and the correlation is predicted by the confidence fields from 20CRv3.&nbsp; The skill of analyzed 500hPa geopotential heights from 20CRv3 for 1979-2015 is comparable to that of modern operational 3- to 4-day forecasts.&nbsp; Finally, 20CRv3 performs well on climate timescales. Long time series and multidecadal averages of mass, circulation, and precipitation fields agree well with modern reanalyses and station- and satellite-based products. 20CRv3 is also able to capture trends in tropospheric layer temperatures that correlate well with independent products in the 20th century, placing recent trends in a longer historical context.</p

Historical Reanalysis: What, How, and Why?

Historical reanalyses combine observations of past weather with simulations from modern numerical weather prediction models to provide a consistent, global history of the weather. Recently, a new reanalysis was released that allows observations of the ocean to impact the atmosphere and vice versa. “CERA‐20C: A Coupled Reanalysis of the Twentieth Century” by P. Laloyaux et al. (2018, https://doi.org/10.1029/2018MS001273 ) describes the first coupled centennial reanalysis, thereby providing more balanced estimates of the ocean‐atmosphere system and allowing for a broader range of studies of the entire Earth system. Results suggest that similar methods could also be leveraged to improve modern weather forecasts. Historical reanalyses can bridge the gap between climate and weather by providing a century‐long history of the weather The first coupled reanalysis of the twentieth century was recently released with promising result

A hybrid particle–ensemble Kalman filter for Lagrangian data assimilation

Author Posting. © American Meteorological Society, 2015. This article is posted here by permission of American Meteorological Society for personal use, not for redistribution. The definitive version was published in Monthly Weather Review 143 (2015): 195–211, doi:10.1175/MWR-D-14-00051.1.Lagrangian measurements from passive ocean instruments provide a useful source of data for estimating and forecasting the ocean’s state (velocity field, salinity field, etc.). However, trajectories from these instruments are often highly nonlinear, leading to difficulties with widely used data assimilation algorithms such as the ensemble Kalman filter (EnKF). Additionally, the velocity field is often modeled as a high-dimensional variable, which precludes the use of more accurate methods such as the particle filter (PF). Here, a hybrid particle–ensemble Kalman filter is developed that applies the EnKF update to the potentially high-dimensional velocity variables, and the PF update to the relatively low-dimensional, highly nonlinear drifter position variable. This algorithm is tested with twin experiments on the linear shallow water equations. In experiments with infrequent observations, the hybrid filter consistently outperformed the EnKF, both by better capturing the Bayesian posterior and by better tracking the truth.The work of Apte benefited from the support of the AIRBUS Group Corporate Foundation Chair in Mathematics of Complex Systems established in ICTS-TIFR. Spiller would like to acknowledge support by NSF Grant DMS-1228265 and ONR Grant N00014-11-1-0087. Sandstede gratefully acknowledges support by the NSF through Grant DMS-0907904. Slivinski was supported by the NSF through Grants DMS-0907904 and DMS-1148284.2015-07-0

Influence of warming and atmospheric circulation changes on multidecadal European flood variability

International audienceEuropean flood frequency and intensity change on a multidecadal scale. Floods were more frequent in the 19th (central Europe) and early 20th century (western Europe) than during the mid-20th century and again more frequent since the 1970s. The causes of this variability are not well understood and the relation to climate change is unclear. Palaeoclimate studies from the northern Alps suggest that past flood-rich periods coincided with cold periods. In contrast, some studies suggest that more floods might occur in a future, warming world. Here we address the contribution of atmospheric circulation and of warming to multidecadal flood variability. For this, we use long series of annual peak streamflow, daily weather data, reanalyses, and reconstructions. We show that both changes in atmospheric circulation and moisture content affected multidecadal changes of annual peak streamflow in central and western Europe over the past two centuries. We find that during the 19th and early 20th century, atmospheric circulation changes led to high peak values of moisture flux convergence. The circulation was more conducive to strong and long-lasting precipitation events than in the mid-20th century. These changes are also partly reflected in the seasonal mean circulation and reproduced in atmospheric model simulations, pointing to a possible role of oceanic variability. For the period after 1980, increasing moisture content in a warming atmosphere led to extremely high moisture flux convergence. Thus, the main atmospheric driver of flood variability changed from atmospheric circulation variability to water vapour increase.La fréquence et l'intensité des inondations en Europe changent à une échelle multidécennale. Les inondations étaient plus fréquentes au 19ème (Europe centrale) et au début du 20ème siècle (Europe occidentale) qu'au milieu du 20ème siècle et à nouveau plus fréquentes depuis les années 1970. Les causes de cette variabilité ne sont pas bien comprises et la relation avec le changement climatique n'est pas claire. Les études paléoclimatiques menées dans les Alpes du Nord suggèrent que les périodes passées riches en inondations coïncidaient avec des périodes froides. En revanche, certaines études suggèrent que davantage d'inondations pourraient se produire dans un monde futur en réchauffement. Nous abordons ici la contribution de la circulation atmosphérique et du réchauffement à la variabilité multidécennale des inondations. Pour cela, nous utilisons de longues séries de débit maximal annuel, des données météorologiques quotidiennes, des réanalyses et des reconstructions climatiques. Nous montrons que les changements de la circulation atmosphérique et du contenu en humidité ont affecté les changements multidécennaux du débit maximal annuel en Europe centrale et occidentale au cours des deux derniers siècles. Nous constatons qu'au cours du 19ème et du début du 20ème siècle, les changements de la circulation atmosphérique ont conduit à des valeurs de pointe élevées de convergence du flux d'humidité. La circulation était plus propice à des événements de précipitations forts et durables qu'au milieu du 20e siècle. Ces changements se reflètent également en partie dans la circulation moyenne saisonnière et sont reproduits dans les simulations des modèles atmosphériques, ce qui indique un rôle possible de la variabilité océanique. Pour la période après 1980, l'augmentation de la teneur en humidité dans une atmosphère qui se réchauffe a conduit à une convergence extrêmement élevée des flux d'humidité. Ainsi, le principal moteur atmosphérique de la variabilité des crues est passé de la variabilité de la circulation atmosphérique à l'augmentation de la vapeur d'eau

Assimilating Lagrangian data for parameter estimation in a multiple-inlet system

© The Author(s), 2017. This is the author's version of the work. It is posted here under a nonexclusive, irrevocable, paid-up, worldwide license granted to WHOI. It is made available for personal use, not for redistribution. The definitive version was published in Ocean Modelling 113 (2017): 131-144, doi:10.1016/j.ocemod.2017.04.001.Numerical models of ocean circulation often depend on parameters that must be tuned to match either results from laboratory experiments or field observations. This study demonstrates that an initial, suboptimal estimate of a parameter in a model of a small bay can be improved by assimilating observations of trajectories of passive drifters. The parameter of interest is the Manning's n coefficient of friction in a small inlet of the bay, which had been tuned to match velocity observations from 2011. In 2013, the geometry of the inlet had changed, and the friction parameter was no longer optimal. Results from synthetic experiments demonstrate that assimilation of drifter trajectories improves the estimate of n, both when the drifters are located in the same region as the parameter of interest and when the drifters are located in a different region of the bay. Real drifter trajectories from field experiments in 2013 also are assimilated, and results are compared with velocity observations. When the real drifters are located away from the region of interest, the results depend on the time interval (with respect to the full available trajectories) over which assimilation is performed. When the drifters are in the same region as the parameter of interest, the value of n estimated with assimilation yields improved estimates of velocity throughout the bay.This work was supported by: Department of Defense Multidisciplinary University Research Initiative (MURI) [grant N000141110087], administered by the Office of Naval Research; the National Science Foundation (NSF); the National Oceanic and Atmospheric Administration (NOAA); NOAA's Climate Program Office; the Department of Energy's Office for Science (BER); and the Assistant Secretary of Defense (Research & Development)

Specializing pedestrian maps to address the needs of people using wheelchairs: a case study in community-sustainable information systems

Gemstone Team FASTR (Finding Alternative Specialized Travel Routes)This study examined whether a community-sustainable information system could be competitive with a centrally-maintained system. We focused on a pedestrian navigation system designed specifically to address the needs of people using wheelchairs. To ascertain the need for such a system, we interviewed people who use wheelchairs on campus. After establishing the need for a new interactive map, we designed and commissioned the construction of TerpNav, an online navigation system that allows users to find a route that avoids certain obstacles, a feature specifically for people using wheelchairs. After TerpNav’s release, we conducted surveys to determine user satisfaction. We found user maintainability was important to the system’s responsiveness to change, which also affected user satisfaction. We then incorporated new community-sustainable features into a second TerpNav version. TerpNav’s success demonstrates that community-sustainable information systems may be a viable alternative to centrally-maintained systems that are less easily specialized to serve individual community needs

Sensitivities of the NCEP Global Forecast System

An important issue in developing a forecast system is its sensitivity to additional observations for improving initial conditions, to the data assimilation (DA) method used, and to improvements in the forecast model. These sensitivities are investigated here for the Global Forecast System (GFS) of the National Centers for Environmental Prediction (NCEP). Four parallel sets of 7-day ensemble forecasts were generated for 100 forecast cases in mid-January to mid-March 2016. The sets differed in their 1) inclusion or exclusion of additional observations collected over the eastern Pacific during the El Ni&ntilde;o Rapid Response (ENRR) field campaign, 2) use of a hybrid 4D&ndash;EnVar versus a pure EnKF DA method to prepare the initial conditions, and 3) inclusion or exclusion of stochastic parameterizations in the forecast model. The Control forecast set used the ENRR observations, hybrid DA, and stochastic parameterizations. Errors of the ensemble-mean forecasts in this Control set were compared with those in the other sets, with emphasis on the upper-tropospheric geopotential heights and vorticity, midtropospheric vertical velocity, column-integrated precipitable water, near-surface air temperature, and surface precipitation. In general, the forecast errors were found to be only slightly sensitive to the additional ENRR observations, more sensitive to the DA methods, and most sensitive to the inclusion of stochastic parameterizations in the model, which reduced errors globally in all the variables considered except geopotential heights in the tropical upper troposphere. The reduction in precipitation errors, determined with respect to two independent observational datasets, was particularly striking.</p

An evaluation of the performance of the twentieth century reanalysis version 3

The performance of a new historical reanalysis, the NOAA–CIRES–DOE Twentieth Century Reanalysis version 3 (20CRv3), is evaluated via comparisons with other reanalyses and independent observations. This dataset provides global, 3-hourly estimates of the atmosphere from 1806 to 2015 by assimilating only surface pressure observations and prescribing sea surface temperature, sea ice concentration, and radiative forcings. Comparisons with independent observations, other reanalyses, and satellite products suggest that 20CRv3 can reliably produce atmospheric estimates on scales ranging from weather events to long-term climatic trends. Not only does 20CRv3 recreate a ‘‘best estimate’’ of the weather, including extreme events, it also provides an estimate of its confidence through the use of an ensemble. Surface pressure statistics suggest that these confidence estimates are reliable. Comparisons with independent upper-air observations in the Northern Hemisphere demonstrate that 20CRv3 has skill throughout the twentieth century. Upper-air fields from 20CRv3 in the late twentieth century and early twenty-first century correlate well with full-input reanalyses, and the correlation is predicted by the confidence fields from 20CRv3. The skill of analyzed 500-hPa geopotential heights from 20CRv3 for 1979–2015 is comparable to that of modern operational 3–4-day forecasts. Finally, 20CRv3 performs well on climate time scales. Long time series and multidecadal averages of mass, circulation, and precipitation fields agree well with modern reanalyses and station- and satellite-based products. 20CRv3 is also able to capture trends in tropospheric-layer temperatures that correlate well with independent products in the twentieth century, placing recent trends in a longer historical context.The research work of R. Przybylak and P. Wyszynski was supported by the National Science Centre, Poland (Grants DEC-2012/07/B/ST10/04002 and 2015/19/B/ST10/02933)

Meteorological data rescue: citizen science lessons learned from Southern Weather Discovery

Daily weather reconstructions (called "reanalyses") can help improve our understanding of meteorology and long-term climate changes. Adding undigitized historical weather observations to the datasets that underpin reanalyses is desirable; however, time requirements to capture those data from a range of archives is usually limited. Southern Weather Discovery is a citizen science data rescue project that recovered tabulated handwritten meteorological observations from ship log books and land-based stations spanning New Zealand, the Southern Ocean, and Antarctica. We describe the Zooniverse-hosted Southern Weather Discovery campaign, highlight promotion tactics, and replicate keying levels needed to obtain 100% complete transcribed datasets with minimal type 1 and type 2 transcription errors. Rescued weather observations can augment optical character recognition (OCR) text recognition libraries. Closer links between citizen science data rescue and OCR-based scientific data capture will accelerate weather reconstruction improvements, which can be harnessed to mitigate impacts on communities and infrastructure from weather extremes