If you want good forecasts for your industry, you should hire the best experts. Right? Well, maybe not.experts, forecast

Armstrong JS

English

Research Papers in Economics

Published in Inc., December 1981, 15-16. 
 
How Expert Are the Experts? 
 
J. Scott Armstrong 
The Wharton School, University of Pennsylvania 
 
If you want good forecasts for your industry, you should hire the best experts. Right? Well, maybe 
not.  
Certainly experts are in demand. Businessmen pay economists generously to tell them how the 
economy will change; brokerage firms give stock analysts large salaries to forecast company earnings; and 
politicians part with substantial fees for expert predictions, too. 
The evidence, however, indicates that this money is poorly spent. Despite all the expertise now 
available to us, there’s still no such thing as a reliable seer. In fact, dozens of carefully constructed studies 
have demonstrated that expertise beyond a minimal level is of little value in forecasting change. This evidence 
is both surprising and useful, and its implications are clear: Avoid hiring expert forecasters. And if you must 
hire an expert forecaster, don’t hire the best, hire the cheapest. This is not to say that experts are useless. They 
can contribute in many ways. Experts can be helpful in analyzing your current situation and suggesting 
measures that might improve it. But in forecasting, great expertise doesn’t seem to improve accuracy. 
Most evidence that expertise doesn’t promote better forecasts comes from studies of the stock 
market, but scholars have also examined forecasts in psychology, economics, medicine, sports, and sociology. 
Expertise in a particular field has been measured by asking people to rate their own levels of expertise or by 
examining their education, experience, reputation, or previous forecasting success. Almost all studies have 
shown that expertise above a minimal level was not related to accuracy (see graph). 
The Relationship Between Expertise and Forecasting Accuracy 
Beyond a minimum level, expertise doesn’t increase accuracy. 
 
High
Low
No
Expertise
Great
Expertise
Level of Expertise
Forecast
Accuracy
Minimal
Level
 
 
The minimal level can be reached easily – by taking an introductory course in a subject, for example. 
It’s possible that accuracy may even drop when expertise exceeds a certain level, as indicated by the dotted 
line on the graph. In one study, for example, psychologists with many years of experience did worse at 
predicting the existence of homosexual tendencies in subjects on the basis of reports about the individuals’ 
personality traits than did students who had just read a paper on characteristics of homosexuals.   2
What about the stock market? Should you rely on your broker’s predictions? The performance of 
experts and novices in forecasting prices of stocks was first examined by Garfield Cox, an economist, in 1930. 
He found no advantage for expertise. A few years later another economist, Alfred Cowles, found that 
following the advice supplied by 16 financial services or acting on the forecasts in 24 financial publications 
would have produced a record slightly worse than the market averages from 1928 to 1932. Many later studies 
have confirmed that expertise does not help in forecasting stock prices. 
A few studies have shown expertise to be useful in some fields, but the differences between experts 
and non-experts have always been small. In a 1971 study, sportswriters were a little more accurate than 
graduate students and faculty members in forecasting football scores. And bookmakers’ forecasts, in turn, 
were slightly better than those of sportswriters. 
The evidence suggests that your bookmaker is one of very few experts whose predictions are really 
worth listening to. 
Why then do people continue to purchase worthless forecasts? Perhaps the clients are mainly 
interested in avoiding responsibility. Who can blame someone for a decision that turned out badly if it was 
based on a forecast by the best of wizards? 
Forecasters fail partly because, whether expert or not, people tend to avoid information that refutes 
their preconceptions. In an experiment, P.C. Wason, a psychologist, presented people with a three number 
sequence: 2, 4, 6. He told them that this sequence had been generated by a rule that he had in his head. The 
subjects were then asked to figure out the rule by proposing additional three-number sequences (e.g., 8, 10, 
12). After each try, Wason would tell the subjects if the new sequence agreed with the rule, and when the 
subjects felt confident, they were to write the rule down. 
The rule Wason had in mind was “three numbers in increasing order of magnitude.” In other words, 
the rule was simply that the second number be larger than the first, and the third number be larger than the 
second. 
Only 25% of the subjects discovered the correct rule. The majority selected other hypotheses –
guessing, for example, that the rule was, “Add two to each successive number.” Then they looked for 
evidence only to confirm their pet hypotheses. They would not attempt to refute their hypotheses by proposing 
sequences that didn’t conform to them. In other words, most subjects refused to entertain the possibility that 
their hypotheses were wrong! 
The story gets worse. Subjects who had given wrong answers were allowed to try again by proposing 
other sets of numbers. About half continued to seek confirmation for the rule they had been told was wrong. 
Can experts’ forecasts be improved? The prospects are not good; evidence shows that experts are 
often unaware of how they make predictions. For example, a 1964 study showed that decisions in 
employment interviews typically were made in the first 30 seconds and that the interviewers didn’t 
understand the reasons for their own decisions. 
There are three ways to get better forecasts. First, the experts should be asked explicitly to list 
the reasons why their forecasts may be wrong. This helps keep them in touch with reality. Second, they 
should be asked to list alternative possibilities that cover the range of potential outcomes. 
Finally, averaging the predictions of several cheap experts is probably more reliable than 
accepting the forecasts of a single expensive expert. Robin Hogarth, a professor at the University of 
Chicago, found that averaging the predictions of as many as 10 people generally produced better 
forecasts than relying on the judgment of a well-qualified individual. 
But often the best course is not to try to improve experts’ forecasts but to dispense with them 
entirely. Although you may not regard yourself as “good” at forecasting, your level of expertise is almost   3
certainly high enough that your forecasts are as reliable as the best expert’s. In fact, many of your 
employees’ forecasts are probably as reliable as the best expert’s. 
Despite all the evidence that vast expertise doesn’t produce better forecasts, businessmen and 
others continue to pay experts as if they were seers. That realization has led me to produce “The Seer-
Sucker Theory.” It says: No matter how much evidence exists that seers do not exist, suckers will pay for seers. 
I forecast that no one is likely to refute this theory any time soon. 

How Expert Are the Experts?

http://128.118.178.162/eps/get/papers/0502/0502029.pdf

How Expert Are the Experts?

Abstract

Similar works

Full text

Available Versions

Research Papers in Economics