This paper summarizes the key conditions under which the index method is valuable for forecasting and describes the procedures one should use when developing index models. The paper also addresses the specific concern of selecting inferior candidates when using the bio-index as a nomination helper. Political decision-makers should not use the bioindex as a stand-alone method but should combine forecasts from a variety of different methods that draw upon different information

Armstrong, J. Scott

Graefe, Andreas

English

ScholarlyCommons@Penn

University of PennsylvaniaScholarlyCommonsMarketing Papers Wharton School8-2010Conditions Under Which Index Models AreUsefulAndreas GraefeKarlsruhe Institute of TechnologyJ. Scott ArmstrongUniversity of Pennsylvania, armstrong@wharton.upenn.eduFollow this and additional works at: http://repository.upenn.edu/marketing_papersPart of the Business CommonsThis paper is posted at ScholarlyCommons. http://repository.upenn.edu/marketing_papers/163For more information, please contact libraryrepository@pobox.upenn.edu.Recommended CitationGraefe, A., & Armstrong, J. S. (2010). Conditions Under Which Index Models Are Useful. Retrieved fromhttp://repository.upenn.edu/marketing_papers/163Conditions Under Which Index Models Are UsefulAbstractThis paper summarizes the key conditions under which the index method is valuable for forecasting anddescribes the procedures one should use when developing index models. The paper also addresses the specificconcern of selecting inferior candidates when using the bio-index as a nomination helper. Political decision-makers should not use the bioindex as a stand-alone method but should combine forecasts from a variety ofdifferent methods that draw upon different information.DisciplinesBusinessThis other is available at ScholarlyCommons: http://repository.upenn.edu/marketing_papers/163Conditions under which index models are useful: Reply to Bio-index Commentaries   Andreas Graefe Institute for Technology Assessment and Systems Analysis Karlsruhe Institute of Technology, Germany graefe@kit.edu  J. Scott Armstrong The Wharton School University of Pennsylvania, Philadelphia, PA armstrong@wharton.upenn.edu August 2010    Send correspondence to Andreas Graefe, Institute for Technology Assessment and Systems Analysis, Karlsruhe Institute of Technology, Germany (graefe@kit.edu).   Abstract This paper summarizes the key conditions under which the index method is valuable for forecasting and describes the procedures one should use when developing index models. The paper also addresses the specific concern of selecting inferior candidates when using the bio-index as a nomination helper. Political decision-makers should not use the bio-index as a stand-alone method but should combine forecasts from a variety of different methods that draw upon different information.  . The commentators raise important issues that Armstrong and Graefe (2011) do not address. Voss (2011) argues that (1) many of the variables used in the bio-index might have no causal relationship with the election outcome and (2) the bio-index might yield a poor selection of candidates, as it does not consider candidates’ personalities, their stands on political issues, or their likely performance. More generally, Cote (2011) has concerns with the suitability of the index method for other problems such as business decision-making. The commentaries help to explain the conditions under which the bio-index in particular, and the index method in general, is valuable for forecasting. This response provides a more complete summary of the key conditions that favor the index method and expands on the procedures one should use when developing index models.  This response also addresses the concern of selecting inferior candidates when using the bio-index as a nomination helper. Political decision-makers should not use the bio-index as a stand-alone method but should combine forecasts from a variety of different methods. In particular, a new bio-index, based on variables that relate to the question of how a candidate will perform as a leader rather than how he will emerge as a leader, could help voters and political parties to select the best candidate.  Key conditions of the index method The index method is an alternative to multiple regression models in situations with small samples and many variables. The usefulness of multiple regression models for prediction depends on the availability of valid and reliable quantitative data relative to the number of causal variables. Multiple regression analyzes historical data to determine the variable weights that provide the best fit. Given non-experimental data, the literature  4 recommends a ratio of 100 observations per predictor variable for using multiple regression models to draw conclusions and make forecasts about human behavior (Armstrong and Graefe, 2011).  In contrast to regression, the index method is suitable for situations in which a large number of causal variables are important and for which one can, at least subjectively, assess the directional effect of each variable on the outcome.  Rather than estimating relationships, the index method uses the prior knowledge on the problem to develop the forecasting model. The index method is based partly on the idea of unit weighting. That is, the method assumes all variables as equally important until proven otherwise. This avoids the problem of spurious effects that occur with non-experimental data. When providing ex ante forecasts for the same data, unit weights are often more accurate than regression weights. This is especially likely when one estimates weights from non-experimental data.  Using simulated data, Einhorn and Hogarth (1975) find that unit weighting is more accurate than regression when the sample is small and the number of, and inter-correlation among, predictor variables is high. Empirical studies support this finding. In analyzing published data in psychology, Schmidt (1971) finds that unit weighting is more accurate than regression weights. A review of the literature (Armstrong 1985, pp.230) finds unit weights to be slightly less accurate in three studies (for academic performance, personnel selection, and medicine) but more accurate in five (three on academic performance, and one each on personnel selection and psychology).   5 Czerlinski et al. (1999) compare unit weighting and multiple regression for 20 selection problems (including psychological, economic, environmental, biological, and health problems), for which the number of predictor variables varies between 3 and 19 (average: 7.7). Most of these examples were taken from statistical textbooks where they were used to demonstrate the application of multiple regression analysis. Not surprisingly, when calculating in-sample forecasts, multiple regression model forecasts were more accurate at 77% correct predictions than forecasts from a unit weight model (73% correct). However, when making out-of-sample predictions, the unit-weight model forecasts were more accurate (69% correct) than multiple regression model forecasts (68% correct).   The index method differs from the models in the unit weights literature in that index models are not limited to using only the set of variables and data that are available for regression analysis. Rather, the method draws upon the cumulative knowledge of a problem, which might come from experts’ domain knowledge or from prior empirical studies. Index models can easily accommodate new knowledge.  Type of problems The conditions favoring the index method are common in selection problems such as choosing political and job candidates, choosing sites for retail outlets, choosing between potential marriage partners, or choosing between contending advertising proposals. As Armstrong and Graefe (2011) show with the bio-index model, one can also make forecasts about the relative performance of alternatives. If one has sufficient historical data on a quantitative dependent variable and can assess the values of the causal variables, one can use simple linear regression against index scores to estimate a model  6 that produces quantitative forecasts such as the percentage vote-share of candidates in an election. Selection of variables When building an index model, use prior knowledge to prepare a list of predictor variables and to assess each variable’s directional influence on the outcome. This prior knowledge can come from empirical evidence or expert domain knowledge. Results from experiments are especially useful. If possible, draw on findings from meta-analyses of experimental studies. Use judgment to assess the variables if no prior knowledge is available. In such cases, use structured approaches such as the Delphi method to combine judgments from several experts. If prior knowledge is ambiguous or contradictory and thus does not allow for estimating a variable’s directional influence on the outcome, do not include the variable in the model.  Index models as decision aids  The primary advantage of index models is that decision-makers can take action upon the forecasts. For example, political parties can use the bio-index to inform their decision about whom to nominate. By comparison, traditional econometric models provide limited or no advice with respect to questions such as what type of candidate a party should nominate or what issues a candidate should stress in the campaign.  Validity of variables Voss raises concerns regarding the validity of some of the variables included in the bio-index. The composite of variables is indeed easy to criticize, as there was often  7 little prior evidence to draw upon. Thus, the selection and coding of many variables relies on weak domain knowledge. There is certainly much more that might be learned about the validity of some of the variables in our model.  The index method is especially valuable in environments with many compensatory variables. In such an environment no single variable is more important than any combination of other variables. Thus, candidates can compensate disadvantages on one variable by scoring favorably on another. For example, the fact that a candidate has not written a book will not, on its own, lose the election for a candidate.  In contrast, the index method is less useful for constructing forecasting models that include non-compensatory variables. In situations in which the importance of one variable is greater than the importance of all other variables put together, one can use the take-the-best heuristic (Gigerenzer & Goldstein 1996). This simple decision rule makes predictions based on a single piece of information. For example, Graefe and Armstrong (2010a) use the take-the-best heuristic to develop a forecasting model that predicts election outcomes based on how voters expect the candidates to handle the single most important issue facing the country. The forecasts from the take-the-best model are almost as accurate as an index model based on all the issues. Combining forecasts One should not use the bio-index as a stand-alone method for forecasting elections since the approach evaluates only one dimension of a candidate. Other factors clearly matter. For example, Graefe and Armstrong (2010a, 2010b) developed two models that provide accurate forecasts of the winner in U.S. presidential elections based  8 on information about how voters expect the candidates to handle the issues facing the country.  One should combine forecasts from different models that draw on different information to get a more complete picture of the future. Such a combined forecast also conforms to Simonton’s inferential framework for judging presidential candidates. Simonton (1993) argues that voters evaluate candidates along three dimensions when deciding whom to vote for: performance, policy, and personality.  Selecting the best candidate Bio-indexes aim at solving the question of who will win, not who should win. Thus, the index contains variables that have an influence on leader emergence. These variables may not necessarily relate to leader performance. For example, while being clean-shaven may improve how voters perceive a candidate, a beard would probably not harm the performance of the candidate once elected.   Voss raises concerns that the candidates themselves might want to take action and improve their biographical score. For example, candidates may want to wear glasses or get plastic surgery. Candidates trying to do well on each variable do not pose a serious problem for the bio-index. First, the bio-index will still differ on many variables. Second, the bio-index model can use relative measures (e.g., who published the most popular book). Third, if the candidates were able to gain equality with one another on certain variables, the impact of these variables decreases – most of which have nothing to do with the competency of a president. Voss notes that the bio-index could yield the selection of an inferior president because the index does not provide evidence on which candidate would be best for the  9 country. One can address this problem by developing an index model based solely on variables that have an impact on performance (e.g., intelligence that exceeds a certain cut-off level). Such a model could be a valuable decision tool for voters, as it could aid in deciding about which candidate to vote for.    10 References Armstrong, J.S. Long-range forecasting: From crystal ball to computer. New York: John Wiley, 1985. Armstrong, Scott J. S., Graefe, Andreas. Predicting elections from biographical information about candidates, Journal of Business Research 2011; XX.  Cote, Joseph A. Predicting elections from biographical information about candidates: A Commentary Essay, Journal of Business Research 2011; XX.  Czerlinski, Jean, Gigerenzer, Gerd, Goldstein, Daniel G. How good are simple heuristics? In: Gigerenzer, Gerd, Todd, Peter M. (Eds.), Simple heuristics that make us smart, New York: Oxford University Press, pp. 97-118, 1999. Einhorn, Hillel J., Hogarth, Robin. M. Unit weighting schemes for decision-making, Organizational Behavior & Human Performance 1975; 13(2): 171-192. Gigerenzer, Gerd, Goldstein, Daniel G. Reasoning the fast and frugal way: Models of bounded rationality, Psychological Review 1996; 103(4): 650-669.  Graefe, Andreas, Armstrong, Scott J. Predicting elections from the most important issue: A test of the take-the-best heuristic, Journal of Behavioral Decision Making 2010a; DOI: 10.1002/bdm.710. Graefe, Andreas, Armstrong, Scott J. Forecasting Elections from Voters’ Perceptions of Candidates’ Ability to Handle Issues, Under review (2010b). Available at: http://dl.dropbox.com/u/3662406/Articles/PollyIssues.pdf  11 Schmidt, Frank L. The relative efficiency of regression and simple unit predictor weights in applied differential psychology, Educational and Psychological Measurement 1971; 31(3): 699-714. Simonton Dean K. Putting the Best Leaders in the White House: Personality, Policy, and Performance. Political Psychology 1993; 14(3): 537-548. Voss, Kevin E. Voss Wins the Presidency! A Commentary Essay on “Predicting Elections from Biographical Information about Candidates: A Test of the Index Method”, Journal of Business Research 2011; XX.  

Conditions Under Which Index Models Are Useful

Kosmopolis

University of Pennsylvania
ScholarlyCommons
Marketing Papers Wharton School
8-2010
Conditions Under Which Index Models Are
Useful
Andreas Graefe
Karlsruhe Institute of Technology
J. Scott Armstrong
University of Pennsylvania, armstrong@wharton.upenn.edu
Follow this and additional works at: http://repository.upenn.edu/marketing_papers
Part of the Business Commons
This paper is posted at ScholarlyCommons. http://repository.upenn.edu/marketing_papers/163
For more information, please contact libraryrepository@pobox.upenn.edu.
Recommended Citation
Graefe, A., & Armstrong, J. S. (2010). Conditions Under Which Index Models Are Useful. Retrieved from
http://repository.upenn.edu/marketing_papers/163
Conditions Under Which Index Models Are Useful
Abstract
This paper summarizes the key conditions under which the index method is valuable for forecasting and
describes the procedures one should use when developing index models. The paper also addresses the specific
concern of selecting inferior candidates when using the bio-index as a nomination helper. Political decision-
makers should not use the bioindex as a stand-alone method but should combine forecasts from a variety of
different methods that draw upon different information.
Disciplines
Business
This other is available at ScholarlyCommons: http://repository.upenn.edu/marketing_papers/163
Conditions under which index models are useful: 
Reply to Bio-index Commentaries 
 
 
Andreas Graefe 
Institute for Technology Assessment and Systems Analysis 
Karlsruhe Institute of Technology, Germany 
graefe@kit.edu 
 
J. Scott Armstrong 
The Wharton School 
University of Pennsylvania, Philadelphia, PA 
armstrong@wharton.upenn.edu 
August 2010 
 
 
 
Send correspondence to Andreas Graefe, Institute for Technology Assessment and 
Systems Analysis, Karlsruhe Institute of Technology, Germany (graefe@kit.edu). 
 
 
Abstract 
This paper summarizes the key conditions under which the index method is valuable for 
forecasting and describes the procedures one should use when developing index models. 
The paper also addresses the specific concern of selecting inferior candidates when using 
the bio-index as a nomination helper. Political decision-makers should not use the bio-
index as a stand-alone method but should combine forecasts from a variety of different 
methods that draw upon different information.  
. 
The commentators raise important issues that Armstrong and Graefe (2011) do 
not address. Voss (2011) argues that (1) many of the variables used in the bio-index 
might have no causal relationship with the election outcome and (2) the bio-index might 
yield a poor selection of candidates, as it does not consider candidates’ personalities, their 
stands on political issues, or their likely performance. More generally, Cote (2011) has 
concerns with the suitability of the index method for other problems such as business 
decision-making. 
The commentaries help to explain the conditions under which the bio-index in 
particular, and the index method in general, is valuable for forecasting. This response 
provides a more complete summary of the key conditions that favor the index method and 
expands on the procedures one should use when developing index models.  
This response also addresses the concern of selecting inferior candidates when 
using the bio-index as a nomination helper. Political decision-makers should not use the 
bio-index as a stand-alone method but should combine forecasts from a variety of 
different methods. In particular, a new bio-index, based on variables that relate to the 
question of how a candidate will perform as a leader rather than how he will emerge as a 
leader, could help voters and political parties to select the best candidate.  
Key conditions of the index method 
The index method is an alternative to multiple regression models in situations 
with small samples and many variables. The usefulness of multiple regression models for 
prediction depends on the availability of valid and reliable quantitative data relative to the 
number of causal variables. Multiple regression analyzes historical data to determine the 
variable weights that provide the best fit. Given non-experimental data, the literature 
 4 
recommends a ratio of 100 observations per predictor variable for using multiple 
regression models to draw conclusions and make forecasts about human behavior 
(Armstrong and Graefe, 2011).  
In contrast to regression, the index method is suitable for situations in which a 
large number of causal variables are important and for which one can, at least 
subjectively, assess the directional effect of each variable on the outcome.  Rather than 
estimating relationships, the index method uses the prior knowledge on the problem to 
develop the forecasting model. 
The index method is based partly on the idea of unit weighting. That is, the 
method assumes all variables as equally important until proven otherwise. This avoids the 
problem of spurious effects that occur with non-experimental data. When providing ex 
ante forecasts for the same data, unit weights are often more accurate than regression 
weights. This is especially likely when one estimates weights from non-experimental 
data.  
Using simulated data, Einhorn and Hogarth (1975) find that unit weighting is 
more accurate than regression when the sample is small and the number of, and inter-
correlation among, predictor variables is high. Empirical studies support this finding. In 
analyzing published data in psychology, Schmidt (1971) finds that unit weighting is more 
accurate than regression weights. A review of the literature (Armstrong 1985, pp.230) 
finds unit weights to be slightly less accurate in three studies (for academic performance, 
personnel selection, and medicine) but more accurate in five (three on academic 
performance, and one each on personnel selection and psychology).  
 5 
Czerlinski et al. (1999) compare unit weighting and multiple regression for 20 
selection problems (including psychological, economic, environmental, biological, and 
health problems), for which the number of predictor variables varies between 3 and 19 
(average: 7.7). Most of these examples were taken from statistical textbooks where they 
were used to demonstrate the application of multiple regression analysis. Not 
surprisingly, when calculating in-sample forecasts, multiple regression model forecasts 
were more accurate at 77% correct predictions than forecasts from a unit weight model 
(73% correct). However, when making out-of-sample predictions, the unit-weight model 
forecasts were more accurate (69% correct) than multiple regression model forecasts 
(68% correct).  
 The index method differs from the models in the unit weights literature in that 
index models are not limited to using only the set of variables and data that are available 
for regression analysis. Rather, the method draws upon the cumulative knowledge of a 
problem, which might come from experts’ domain knowledge or from prior empirical 
studies. Index models can easily accommodate new knowledge.  
Type of problems 
The conditions favoring the index method are common in selection problems such 
as choosing political and job candidates, choosing sites for retail outlets, choosing 
between potential marriage partners, or choosing between contending advertising 
proposals. As Armstrong and Graefe (2011) show with the bio-index model, one can also 
make forecasts about the relative performance of alternatives. If one has sufficient 
historical data on a quantitative dependent variable and can assess the values of the causal 
variables, one can use simple linear regression against index scores to estimate a model 
 6 
that produces quantitative forecasts such as the percentage vote-share of candidates in an 
election. 
Selection of variables 
When building an index model, use prior knowledge to prepare a list of predictor 
variables and to assess each variable’s directional influence on the outcome. This prior 
knowledge can come from empirical evidence or expert domain knowledge. Results from 
experiments are especially useful. If possible, draw on findings from meta-analyses of 
experimental studies. 
Use judgment to assess the variables if no prior knowledge is available. In such 
cases, use structured approaches such as the Delphi method to combine judgments from 
several experts. If prior knowledge is ambiguous or contradictory and thus does not allow 
for estimating a variable’s directional influence on the outcome, do not include the 
variable in the model.  
Index models as decision aids 
 The primary advantage of index models is that decision-makers can take action 
upon the forecasts. For example, political parties can use the bio-index to inform their 
decision about whom to nominate. By comparison, traditional econometric models 
provide limited or no advice with respect to questions such as what type of candidate a 
party should nominate or what issues a candidate should stress in the campaign.  
Validity of variables 
Voss raises concerns regarding the validity of some of the variables included in 
the bio-index. The composite of variables is indeed easy to criticize, as there was often 
 7 
little prior evidence to draw upon. Thus, the selection and coding of many variables relies 
on weak domain knowledge. There is certainly much more that might be learned about 
the validity of some of the variables in our model.  
The index method is especially valuable in environments with many 
compensatory variables. In such an environment no single variable is more important 
than any combination of other variables. Thus, candidates can compensate disadvantages 
on one variable by scoring favorably on another. For example, the fact that a candidate 
has not written a book will not, on its own, lose the election for a candidate.  
In contrast, the index method is less useful for constructing forecasting models 
that include non-compensatory variables. In situations in which the importance of one 
variable is greater than the importance of all other variables put together, one can use the 
take-the-best heuristic (Gigerenzer & Goldstein 1996). This simple decision rule makes 
predictions based on a single piece of information. For example, Graefe and Armstrong 
(2010a) use the take-the-best heuristic to develop a forecasting model that predicts 
election outcomes based on how voters expect the candidates to handle the single most 
important issue facing the country. The forecasts from the take-the-best model are almost 
as accurate as an index model based on all the issues. 
Combining forecasts 
One should not use the bio-index as a stand-alone method for forecasting 
elections since the approach evaluates only one dimension of a candidate. Other factors 
clearly matter. For example, Graefe and Armstrong (2010a, 2010b) developed two 
models that provide accurate forecasts of the winner in U.S. presidential elections based 
 8 
on information about how voters expect the candidates to handle the issues facing the 
country.  
One should combine forecasts from different models that draw on different 
information to get a more complete picture of the future. Such a combined forecast also 
conforms to Simonton’s inferential framework for judging presidential candidates. 
Simonton (1993) argues that voters evaluate candidates along three dimensions when 
deciding whom to vote for: performance, policy, and personality.  
Selecting the best candidate 
Bio-indexes aim at solving the question of who will win, not who should win. 
Thus, the index contains variables that have an influence on leader emergence. These 
variables may not necessarily relate to leader performance. For example, while being 
clean-shaven may improve how voters perceive a candidate, a beard would probably not 
harm the performance of the candidate once elected.   
Voss raises concerns that the candidates themselves might want to take action and 
improve their biographical score. For example, candidates may want to wear glasses or 
get plastic surgery. Candidates trying to do well on each variable do not pose a serious 
problem for the bio-index. First, the bio-index will still differ on many variables. Second, 
the bio-index model can use relative measures (e.g., who published the most popular 
book). Third, if the candidates were able to gain equality with one another on certain 
variables, the impact of these variables decreases – most of which have nothing to do 
with the competency of a president. 
Voss notes that the bio-index could yield the selection of an inferior president 
because the index does not provide evidence on which candidate would be best for the 
 9 
country. One can address this problem by developing an index model based solely on 
variables that have an impact on performance (e.g., intelligence that exceeds a certain 
cut-off level). Such a model could be a valuable decision tool for voters, as it could aid in 
deciding about which candidate to vote for.   
 10 
References 
Armstrong, J.S. Long-range forecasting: From crystal ball to computer. New York: John 
Wiley, 1985. 
Armstrong, Scott J. S., Graefe, Andreas. Predicting elections from biographical 
information about candidates, Journal of Business Research 2011; XX.  
Cote, Joseph A. Predicting elections from biographical information about candidates: A 
Commentary Essay, Journal of Business Research 2011; XX.  
Czerlinski, Jean, Gigerenzer, Gerd, Goldstein, Daniel G. How good are simple heuristics? 
In: Gigerenzer, Gerd, Todd, Peter M. (Eds.), Simple heuristics that make us smart, 
New York: Oxford University Press, pp. 97-118, 1999. 
Einhorn, Hillel J., Hogarth, Robin. M. Unit weighting schemes for decision-making, 
Organizational Behavior & Human Performance 1975; 13(2): 171-192. 
Gigerenzer, Gerd, Goldstein, Daniel G. Reasoning the fast and frugal way: Models of 
bounded rationality, Psychological Review 1996; 103(4): 650-669.  
Graefe, Andreas, Armstrong, Scott J. Predicting elections from the most important issue: 
A test of the take-the-best heuristic, Journal of Behavioral Decision Making 
2010a; DOI: 10.1002/bdm.710. 
Graefe, Andreas, Armstrong, Scott J. Forecasting Elections from Voters’ Perceptions of 
Candidates’ Ability to Handle Issues, Under review (2010b). Available at: 
http://dl.dropbox.com/u/3662406/Articles/PollyIssues.pdf 
 11 
Schmidt, Frank L. The relative efficiency of regression and simple unit predictor weights 
in applied differential psychology, Educational and Psychological Measurement 
1971; 31(3): 699-714. 
Simonton Dean K. Putting the Best Leaders in the White House: Personality, Policy, and 
Performance. Political Psychology 1993; 14(3): 537-548. 
Voss, Kevin E. Voss Wins the Presidency! A Commentary Essay on “Predicting 
Elections from Biographical Information about Candidates: A Test of the Index 
Method”, Journal of Business Research 2011; XX.  


https://repository.upenn.edu/cgi/viewcontent.cgi?article=1155&amp;context=marketing_papers

Conditions Under Which Index Models Are Useful

Abstract

Similar works

Full text

Available Versions

ScholarlyCommons@Penn

Kosmopolis