Evaluating Forecasting Methods

Ideally, forecasting methods should be evaluated in the situations for which they will be used. Underlying the evaluation procedure is the need to test methods against reasonable alternatives. Evaluation consists of four steps: testing assumptions, testing data and methods, replicating outputs, and assessing outputs. Most principles for testing forecasting methods are based on commonly accepted methodological procedures, such as to prespecify criteria or to obtain a large sample of forecast errors. However, forecasters often violate such principles, even in academic studies. Some principles might be surprising, such as do not use R-square, do not use Mean Square Error, and do not use the within-sample fit of the model to select the most accurate time-series model. A checklist of 32 principles is provided to help in systematically evaluating forecasting methods

Rule-Based Forecasting: Using Judgment in Time-Series Extrapolation

Rule-Based Forecasting (RBF) is an expert system that uses judgment to develop and apply rules for combining extrapolations. The judgment comes from two sources, forecasting expertise and domain knowledge. Forecasting expertise is based on more than a half century of research. Domain knowledge is obtained in a structured way; one example of domain knowledge is managers= expectations about trends, which we call “causal forces.” Time series are described in terms of 28 conditions, which are used to assign weights to extrapolations. Empirical results on multiple sets of time series show that RBF produces more accurate forecasts than those from traditional extrapolation methods or equal-weights combined extrapolations. RBF is most useful when it is based on good domain knowledge, the domain knowledge is important, the series is well behaved (such that patterns can be identified), there is a strong trend in the data, and the forecast horizon is long. Under ideal conditions, the error for RBF’s forecasts were one-third less than those for equal-weights combining. When these conditions are absent, RBF neither improves nor harms forecast accuracy. Some of RBF’s rules can be used with traditional extrapolation procedures. In a series of studies, rules based on causal forces improved the selection of forecasting methods, the structuring of time series, and the assessment of prediction intervals

Comparing videofluoroscopy and endoscopy to assess swallowing in bottle-fed young infants in the neonatal intensive care unit

Contains fulltext : 235605.pdf (Publisher’s version ) (Closed access

Extrapolation for Time-Series and Cross-Sectional Data

Extrapolation methods are reliable, objective, inexpensive, quick, and easily automated. As a result, they are widely used, especially for inventory and production forecasts, for operational planning for up to two years ahead, and for long-term forecasts in some situations, such as population forecasting. This paper provides principles for selecting and preparing data, making seasonal adjustments, extrapolating, assessing uncertainty, and identifying when to use extrapolation. The principles are based on received wisdom (i.e., experts’ commonly held opinions) and on empirical studies. Some of the more important principles are:• In selecting and preparing data, use all relevant data and adjust the data for important events that occurred in the past.• Make seasonal adjustments only when seasonal effects are expected and only if there is good evidence by which to measure them.• In extrapolating, use simple functional forms. Weight the most recent data heavily if there are small measurement errors, stable series, and short forecast horizons. Domain knowledge and forecasting expertise can help to select effective extrapolation procedures. When there is uncertainty, be conservative in forecasting trends. Update extrapolation models as new data are received.• To assess uncertainty, make empirical estimates to establish prediction intervals.• Use pure extrapolation when many forecasts are required, little is known about the situation, the situation is stable, and expert forecasts might be biased

Scratch lottery tickets are a poor incentive to respond to mailed questionnaires

BACKGROUND: It has been demonstrated that the enclosure of money with a mailed questionnaire increases the response rate significantly. We evaluated scratch lottery tickets as an alternative to cash. METHODS: 1500 randomly selected Norwegians between the ages of 40 and 65 years were sent a short questionnaire. 250 received one lottery scratch ticket worth 20 Norwegian kroner (approximately 3 US$) together with the questionnaire, 250 received two scratch tickets, and 250 were promised two scratch tickets if they replied within one week. A fourth group of 250 persons received a 50 kroner banknote with the questionnaire. The remaining 500 letters served as controls. RESULTS: The overall response rate after 6 weeks was 77%. Logistic regression analysis showed that only the 50 kroner group had a response rate that was statistically significantly higher than the controls (p < 0.0001). It was also significantly higher than that in any of the other incentive groups (p < 0.0001, p < 0.004 and p < 0.0001 respectively). Female sex (p < 0.001) and age (p < 0.002) increased the response rate significantly. CONCLUSION: It is possible that the recipients scratched their cards before completing the questionnaire, and that it was a disincentive for the majority that they did not win anything. Lottery scratch tickets are no substitute for cash as an incentive to respond to a questionnaire

Differential effects of pre and post-payment on neurologists' response rates to a postal survey

<p>Abstract</p> <p>Background</p> <p>Monetary incentives are an effective way of increasing response rates to surveys, though they are generally less effective in physicians, and are more effective when the incentive is paid up-front rather than when made conditional on completion.</p> <p>Methods</p> <p>In this study we examine the effectiveness of pre- and post-completion incentives on the response rates of all the neurologists in the UK to a survey about conversion disorder, using a cluster randomised controlled design. A postal survey was sent to all practicing consultant neurologists, in two rounds, including either a book token, the promise of a book token, or nothing at all.</p> <p>Results</p> <p>Three hundred and fifty-one of 591 eligible neurologists completed the survey, for a response rate of 59%. While the post-completion incentive exerted no discernible influence on response rates, a pre-completion incentive did, with an odds-ratio of 2.1 (95% confidence interval 1.5 - 3.0).</p> <p>Conclusions</p> <p>We conclude that neurologists, in the UK at least, may be influenced to respond to a postal survey by a pre-payment incentive but are unaffected by a promised reward.</p

Lung Cancer Risk after Exposure to Polycyclic Aromatic Hydrocarbons: A Review and Meta-Analysis

Typical polycyclic aromatic hydrocarbon mixtures are established lung carcinogens, but the quantitative exposure–response relationship is less clear. To clarify this relationship we conducted a review and meta-analysis of published reports of occupational epidemiologic studies. Thirty-nine cohorts were included. The average estimated unit relative risk (URR) at 100 μg/m(3) years benzo[a]pyrene was 1.20 [95% confidence interval (CI), 1.11–1.29] and was not sensitive to particular studies or analytic methods. However, the URR varied by industry. The estimated means in coke ovens, gasworks, and aluminum production works were similar (1.15–1.17). Average URRs in other industries were higher but imprecisely estimated, with those for asphalt (17.5; CI, 4.21–72.78) and chimney sweeps (16.2; CI, 1.64–160.7) significantly higher than the three above. There was no statistically significant variation of URRs within industry or in relation to study design (including whether adjusted for smoking), or source of exposure information. Limited information on total dust exposure did not suggest that dust exposure was an important confounder or modified the effect. These results provide a more secure basis for risk assessment than was previously available