Some results of a nationwide survey of National Weather Service forecasters with regard to probability forecasting in general and precipitation probability forecasting in particular are summarized. Specifically, the questionnaire which was used in the survey, the participants in the survey (i.e., the forecasters), and the nature of the results are briefly described, and some recommendations based upon these results are presented

Murphy, A.H.

Winkler, R.L.

International Institute for Applied Systems Analysis (IIASA)

[Reprinled from BULLETIN OF TRE AMERICAN ~ ~ E T ~ O R O L O G I C A L  SOCI~~ 'Y,  Vol. 55, No. 12, December 1974, pp. 1449-14531 Printed in U. S. A. ~rollil bility lorecasts: a survey Allan H. Murphy National Center for Atmospheric Research 3 ol National Weather Service Boulder, Colorado 80303 and Robert L. Winkler lorecasters ' Graduate School of Business Indiana University R P -75-9 Bloomington, Indiana 4740 1 Reprint Abstract Some results of a nationwide survey of National Wea- ther Service forecasters with regard to probability forecasting in general and precipitation probability fore- casting in particular are summarized. Specifically, the questionnaire which was used in the survey, the partici- pants in the survey (i.e., the forecasters), and the nature of the results are briefly described, and some recom- mendations based upon these results are presented. 1. Introduction This paper summarizes some of the results of a na- tionwide survey of l\Tational Weather Service (NWS) forecasters relative to precipitation probability fore- casting in particular and probability forecasting in general.5 The survey, which was conducted in May and June, 1972, was inspired by an earlier survey of the forecasters of the Travelers Weather Service (TWS) (Murphy and Winkler, 1971a, b) and by research related to probability assessment and evaluation undertaken by the authors of this paper (e.g., Murphy, 1972; Murphy and Winkler, 1970; Peterson, Snapper, and Murphy, 1972; and Winkler and Murphy, 1968a, b). The objective of the survey was to obtain information about probability forecasting in an operational setting from a large sample of forecasters who make such fore- casts on a regular basis. Specifically, we wanted to obtain information about the process by which precipitation probability forecasts are formulated; about the judg- ments of the forecasters relative to the definition, in- terpretation, and evaluation of these forecasts; about the 1 Supported in part by the National Science Foundation (Atmospheric Sciences Section) under Grants GA-31735 and GA-41232. 2On leave and visiting the International Institute for Applied Systems Analysis, Laxenburg, Austria (September 1974-March 1975). 3 The National Center for Atmosplieric Research is spon- sored by the National Science Foundation. 40n leave and visiting the International Institute for Applied Systems Analysis, Laxenburg, Austria (July 1973 to January 1974), and the Department of Statistics, Stanford University, Stanford, Calif. (February-August 1974). 5 This paper is based upon a lengthy report (Murphy and Winkler, 1973) that describes these results in some detail. Copies of the report can be obtained from the authors of this paper. effects of feedback and experience upon such forecasts; and about the forecasters' opinions concerning the need for and use of probability forecasts by the general public and by specific user groups. This information was ex- pected to provide a reasonably accurate description of 1) current practices and beliefs relative to probability forecasting in meteorology and 2) the nature and extent of any problems related to these practices and beliefs. With regard to the latter, the results of the TWS survey (Murphy and Winkler, 1971a,- b) and of other studies (see, for example, Murphy, 1972, and Murphy and Winkler, 1974) have suggested that a number of im- portant theoretical and practical problems do indeed exist. We believe that such problems must be resolved if probability forecasting in meteorology is to realize its full potential. In this paper we briefly describe the questionnaire (Section 2), the participants (i.e., the forecasters) (Section 3), and the nature of the principal results (Section 4) and present some recommendations based upon these results (Section 5). 2. The questionnaire a. Questionnaire topics The questionnaire was designed to elicit information from the NWS forecasters with regard to the following topics related to probability forecasting: 1) the process by which probability of precipitation (Pop) forecasts are formulated, including the infor- mation sources examined, their relative importance, aiid their order of examination; 2) the relationships between a forecaster's judgments (expressed in probabilistic terms) and that fore- caster's oficial POP forecasts, and the factors which lead to differences between judgments and forecasts; 3) the definition of the event "precipitation"; 4) the definition and interpretation of POP forecasts; 5) the availability to forecasters of information in the form of feedback regarding the accuracy, reliability, etc., of their POP forecasts and those of other fore- casters, aiid the effects of such information upon their forecasts; 6) the evaluation of POP lorecasts and the reasons for any improvements in probability forecasting ability; 7) the effects of forecasting experience upon POP fore- casts; Bulletin American Meteorological Society Vol.  55, No. 12, December 1974 8) the present and future need for and usefulness of 2) age; forecasts expressed in probabilistic terms; and 3) education and training (general education and mete- 9') the forecasters' opinions regarding the interpretation orological education and training); and use of POP forecasts by members of the general 4) forecasting experience (total and station experience public and specific user groups, and the effects of in both general forecasting and probability Porecast- these opinions upon their forecasts. ing); 5) nature of participation in POP forecasting program These topics are similar to the topics considered in the (regular or occasional participant) :a and TWS survey. However, the NWS questionnaire con- 6) attendance at the NWS Forecasters' Training Course. tained approximately twice as many questions as the TWS and, as a result, the- NWS question- naire was able to treat these topics in considerably more detail. b. Development and administration of questionnaire A preliminary version of the questionnaire was devel- oped in March 1972. This version of the questionnaire was pretested by a small group of NWS forecasters in early April. The  results of the pretest, as well as the comments and suggestions received from individuals who were asked to review the preliminary version of the questionnaire, led to the preparation of a revised ques- tionnaire during the latter half of April. This version of the questionnaire was mailed to all Weather Service Forecast Offices (WSFOs) and Weather Service Offices (WSOs) involved in the survey (see Section 3) in May. Almost all of the completed questionnaires were received prior to 1 July 1972. The  questionnaire was prepared in two formats, and the only difference between these formats was that the order in which the alternative answers to the ques- tions were presented was reversed. Otherwise, question- naires in the two formats were indistinguishable. T h e  set of questionnaires that was sent to each WSFO and WSO consisted of approximately an equal number of questionnaires in each format. Thus, the probability that a particular forecaster in a specific office would re- ceive a questionnaire of a given format was approxi- mately one-half. 3. The participants The survey was designed to obtain responses from all of the NWS meteorologists who were experienced in making POP forecasts, as well as from a sample of the NWS meteorological technicians who alsb were experi- enced in making such forecasts. The  regional contacts (see Acknowledgments) provided estimates of the number of meteorologists who were involved (at that time) in making precipitation probability forecasts at each WSFO and WSO in their respective regions. I n  addition, they selected a sample of at least 30 meteorological technicians a t  various WSFOs and TVSOs in their respective regions who were making these forecasts. An appropriate number of questionnaires was then sent to each WSFO and WSO. The  forecasters who participated in the survey were asked to provide the following personal information: I )  classification (meteorologist, meteorological techni- cian, etc.); Nationwide and regional tabulations of the forecasters' responses to the personal information questions are pre- sented in Murphy and Winkler (1973). This informa- tion permits the results of the survey to be stratified by the forecasters' classification, age, education, and so forth. 4. The results a. Response rate The  overall response of the forecasters to the survey is presented in Table 1. Nationwide, the response rate was almost 68%, while the regional response rates varied from 89.1% in the Central Region to 58.3% in the Western Region. These differences are largely the result of somewhat different procedures used by the regional contacts in identifying the survey participants, primarily meteorologists, in their respective regions. Moreover, the "real" response rates were actually higher than the "apparent" response rates indicated in Table I, be- cause some WSFOs and WSOs are believed to have received too many questionnaires. In  any case, a re- sponse rate exceeding 50%',, which was obtained in each region, is considered excellent for a survey administered by mail. b. Response bias In  order to investigate the existence of any response bias that might have been produced by the order in TABLE 1. Number of questionnaires sent out and returned and the response rates for each region and for the nation as a whole. Number of Number of questionnaires questionnaires Response rate Region sent out returned (%) Eastern 293 173 59.0 Southern 253 189 74.7 Central 174 155 89.1 Western 295 172 58.3 Nation 1015 689 6'1.9 6These processes were made somewhat more difficult by the fact that, just prior to and during the period in which the survey was conducted, the NWS policy according to which forecasters were concerned with either aviation fore- casting or  public weather forecasting was undergoing a significant change. POP forecasts were, and still are, ex- clusively part of the public weather forecasting progrgm, but present NWS policy is such that the forecasters are each involved in both programs. Vol. 55, No. 12, December 1974 11) Approximately one-half of the forecasters are affected by their impressions of the general public's interpre- tation of their POP forecasts, and these impressions cause some forecasters to modify their forecasts in the direction of categorical forecasts and other fore- casters to modify their forecasts in the direction of the climatological probabilities. 5. Recommendations The conclusions presented in Section 4 lead to the fol- lowing recommendations: 1) A need exists to train forecasters in probability and statistics as these topics relate to probability forecast- ing in meteorology. This training could take several forms: i) self-training by the forecasters themselves in their WSFOs and WSOs through the use of pro- grammed learning material; ii) short courses con- ducted at several locations (e.g., at each regional headquarters for the forecasters in that region); and iii) lectures and written material presented during the NWS Forecasters' Training Course. We believe that the approach involving programmed learning ma- terial offers certain advantages vis-a-vis the other approaches in that this approach would accomplish the objective of training all of the forecasters in a relatively short period of time and at a relatively modest cost. Thus, we recommend that a programmed learning manual be developed for use by the NWS forecasters which would contain detailed explana- tions and illustrations of those concepts in probabil- ity and statistics that are related to the theory and practice of probability forecasting in meteorology. 2) The  fact that the forecasters perceive some confusion on the part of the public with regard to probability forecasts and the fact that the forecasters themselves exhibit some confusion both suggest that some con- fusion undoubtedly exists among members of the general public. Thus, we believe that a need also exists to initiate a program to educate the general public and members of specific user groups with regard to the purposes, meaning, and use of probability fore- casts. Such a program should be designed to reach potential users of these forecasts in a variety of ways (e.g., directly by means of brochures and pamphlets, as well as indirectly through articles in newspapers, maga- zines, and other publications). Furthermore, this program should be conducted on a continuing basis. As a first step in initiating such a program, we recom- mend that a brochure be prepared which can be distributed free of charge to potential users and to others, such as members of the news media, who have direct contact with users. The  brochure prepared by Bennett et al. (1969) would represent an excellent starting point for this effort. 3) With regard to forecasting practices, we believe that the ability of forecasters to converse in the "lan- guage of uncertainty" (i.e., in terms of probabilities) could be significantly enhanced if they began to think and speak in these terms when formulating their forecasts on a day-to-day basis. Thus, we recommend that forecasters be encouraged to formulate all (or at least more) of their forecasts in probabilistic terms,s and then, if necessary, they could translate these fore- casts into categorical terms for dissemination to the public. In  addition, we recommend that WSFOs and WSOs be encouraged to undertake probability fore- casting experiments designed to provide forecasters with actual operational (or pseudo-operational) ex- perience in expressing forecasts of a variety of mete- orological variables in probabilistic terms. Acknowledgments. We would like to express our ap- preciation to all of the NWS forecasters who partici- pated in the survey and to those individuals in the regional and local offices of the NWS who assisted in the administration of the questionnaire. I n  particular, we would like to thank Dr. Duane S. Cooley, Technical Procedures Branch, NWS, and our regional contacts, Messrs. Carlos R. Dunn (Eastern Region), Paul L. Moore (Southern Region), Lawrence A. Hughes (Central Region), and Leonard W. Snellman (Western Region), without whose interest and cooperation this survey could not have been conducted. 8 The fact that "objective7' probability forecasts of a variety of parameters are routinely provided as guidance to forecasters in the WSFOs (see Klein and Glahn, 1974) should facilitate this practice. References Bennett, E. S., J. E. Newman, P. W. Dailey, Jr., C. E. Petz- hold, and L. H. Smith, 1969: Precipitation probabilities in weather forecasts. Lafayette, Ind., Purdue University, Coop- erative Extension Service, Publication AY-174, 15 pp. Brier, G. W., 1950: Verification of forecasts expressed in terms of probability. Mon. Wea. Rev., 78, 1-3. Hughes, L. A., 1965: On the probability forecasting of the occurrence of precipitation. Washington, D .C., Environ- mental Science Services Administration, Weather Bureau, Central Region Technical Note 20-CR-3, 36 pp. -, 1967: Improving precipitation probability forecasts using the Central Region verification printout. Kansas City, Mo., Environmental Science Services Administration, Wea- ther Bureau, Central Region Technical Memorandum 15, 18 PP. Klein, W. H., and H. R. Glahn, 1974: Forecasting local wea- ther by means of model output statistics. Bull. Amer. Meteor. Soc., 55, 1217-1227. AiIurphy, A. H., 1972: Probability forecasting in meteorology: a review of recent developments. Boulder, Colo., National Center for Atmospheric Research, unpublished manu- script, 19 pp. -, and R. L. Winkler, 1970: Scoring rules in probability assessment and evaluation. Acta Psychologica, 34, 273-286. - , and -, 1971a: Forecasters and probability forecasts: the responses to a questionnaire. Bull. Amer. Meteor. Soc., 52, 158-165. -, and -, 1971b: Forecasters and probability forecasts: some current problems. Bull. Amer. Meteor. Soc., 52, 239- 247. Bulletin American Meteorological Society - , and -, 1973: National Weather Service forecasters and probability forecasts: preliminary results of a na- tionwide survey. Boulder, Colo., National Center for Atmo- spheric Research and Bloomington, Ind., Indiana Univer- sity, unpublished report, 146 pp. - , and -, 1974: Subjective probability forecasting ex- periments in meteorology: some preliminary results. Bull. Amer. Meteor. Soc., 55, 1206-1216. Peterson, C. R., K. J. Snapper, and A. H. Murphy, 1972: Credible interval temperature forecasting. Bull. Amer. Meteor. Soc., 53, 966-970. Winkler, R. L., and A. H. Murphy, 1968a: Evaluation of subjective precipitation probability forecasts. Proc. First National Conference on Statistical Meteorology (Hartford, 1968). Boston, Mass., American Meteorological Society, 148-157. - , and -, 1968b: "Good" probability assessors. J .  Appl.  Meteor., 7, 751-758. 

Probability Forecasts: A Survey of National Weather Service Forecasters

Abstract

Similar works

Full text

Available Versions

International Institute for Applied Systems Analysis (IIASA)