This article shows how admission and enrollment processes affect the interpretation of simple validation studies of academic performance. In a competitive market for students, optimal behavior of admissions committees and applicants drives the simple correlation between test scores and performance toward zero, regardless of the relationship in the population of prospective students. Data from our university’s MBA program support the prediction that applicants exhibit a higher correlation between test scores and undergraduate GPAs than do current students. This suggests that standard validation studies will understate the importance of GMAT scores in predicting performance of potential MBA students

Cushing, Matthew

McGarvey, Mary

English

UNL | Libraries

University of Nebraska - Lincoln 
DigitalCommons@University of Nebraska - Lincoln 
Economics Department Faculty Publications Economics Department 
4-2004 
Sample Selection in Models of Academic Performance 
Matthew Cushing 
University of Nebraska-Lincoln, MCUSHING1@UNL.EDU 
Mary McGarvey 
University of Nebraska-Lincoln, MMCGARVEY1@UNL.EDU 
Follow this and additional works at: https://digitalcommons.unl.edu/econfacpub 
 Part of the Economics Commons 
Cushing, Matthew and McGarvey, Mary, "Sample Selection in Models of Academic Performance" (2004). 
Economics Department Faculty Publications. 42. 
https://digitalcommons.unl.edu/econfacpub/42 
This Article is brought to you for free and open access by the Economics Department at DigitalCommons@University 
of Nebraska - Lincoln. It has been accepted for inclusion in Economics Department Faculty Publications by an 
authorized administrator of DigitalCommons@University of Nebraska - Lincoln. 
I. Introduction
 
Virtually all graduate programs in the United 
States require applicants to submit scores on stan-
dardized tests, such as the Graduate Record Examina-
tion (GRE) or the Graduate Management Admissions 
Test (GMAT). The usefulness of these tests as predic-
tors of graduate student performance, however, has 
never been firmly established. In fact, formal statis-
tical investigations (validation studies) typically find 
that standardized tests have surprisingly little predic-
tive content. Hanson and Harrell (1985) report a neg-
ative correlation between GMAT scores and subse-
quent compensation, Hansen (1971) reports very little 
correlation between GRE scores and graduate grade 
point averages (GPAs) in economics, and Sternberg 
and Williams (1997) find GRE scores of limited use 
in predicting performance in psychology. Even a re-
cent study by the GRE Board of the Educational Test-
ing Service (1998) shows relatively small correlations 
between first-year graduate GPA and general GRE 
scores. Why do admissions committees pay so much 
attention to these scores when formal evidence sug-
gests that they have little or no predictive content?
In this article, we suggest that a form of sample 
selection bias makes interpretation of standard val-
idation studies problematic. As Darlington (1998) 
and Cornell (1998) point out, validation studies use 
data on students who have matriculated at a partic-
ular institution. The admission procedure and accep-
tance decisions make these data a censored sample 
from the population of prospective students. Institu-
tions occasionally admit students with atypically low 
test scores, but only if these students present other, 
countervailing evidence of high ability. Students with 
atypically high test scores occasionally will choose to 
enroll, but only if countervailing evidence of low abil-
ity precluded their admission to higher-ranking insti-
tutions. Given that both high and low scorers exhibit 
some other, countervailing evidence on ability, it is 
not surprising that standardized test scores by them-
selves are not strong predictors of performance.
This article provides a formal model of how the se-
lection process influences the results of standard val-
idation studies. We find that in a competitive market 
for students, optimal behavior on the part of admis-
sions committees and on the part of applicants drives 
the simple correlation between test scores and perfor-
Published in Economic Inquiry 42:2 (April 2004), pp. 319-322; doi 10.1093/ei/cbh063   
Copyright © 2004 Western Economic Association International. Used by permission.
The authors thank an anonymous referee for Economic Inquiry for useful suggestions.
Sample Selection in Models of Academic Performance
Matthew J. Cushing and Mary G. McGarvey
University of Nebraska–Lincoln
M. J. Cushing is Associate Professor, University of Nebraska–Lincoln, Lincoln, NE 68588;  
tel 402 472-2323, fax 402 472-9700, email mcushing1@unl.edu 
M. G. McGarvey is Associate Professor, University of Nebraska–Lincoln, Lincoln, NE 68588; 
 tel 402 472-9415, fax 402 472-9700, email mmcgarveyl@unl.edu
Abstract
This article shows how admission and enrollment processes affect the interpretation of simple validation 
studies of academic performance. In a competitive market for students, optimal behavior of admissions 
committees and applicants drives the simple correlation between test scores and performance toward 
zero, regardless of the relationship in the population of prospective students. Data from our university’s 
MBA program support the prediction that applicants exhibit a higher correlation between test scores and 
undergraduate GPAs than do current students. This suggests that standard validation studies will under-
state the importance of GMAT scores in predicting performance of potential MBA students. 
Abbreviations: GMAT, Graduate Management Admissions Test; GPA, Grade Point Average; GRE, Grad-
uate Record Examination; MBA, Master’s of Business Administration
319
320  Cu s h i n g & MCga r v e y i n Ec o n o m i c in q u i r y 42 (2004)
mance toward zero, regardless of the relationship in 
the population of prospective students. The key fea-
tures of the model are that institutions select students 
(and students self-select) based on more information 
than test scores alone and that a competitive market 
for students results in institutions enrolling students 
of relatively homogenous predicted ability. The sam-
ple selection process generates a negative correlation 
between test scores and other observable informa-
tion, which, as the market becomes more competitive, 
drives the simple correlation between test scores and 
performance toward zero.
II. A Simple Analytic Model
Assume that a simple scalar variable, y, measures 
scholastic ability in the population of potential stu-
dents to a particular institution. This ability variable 
will be revealed perfectly (as performance) if the stu-
dent matriculates but cannot be observed before ma-
triculation. The admissions committee at this and 
other institutions must make acceptance and rejection 
decisions based on observable variables that may be 
correlated with this performance variable. The com-
mittee observes characteristics, xl and x2, which, for 
concreteness, we interpret as the score on a standard-
ized test and an index of undergraduate performance, 
respectively. Assuming joint normality of three vari-
ables, y, xl, and x2, in the population of prospective 
students, expected ability (and performance) will be 
linear in the two observable variables,
E(y│1, x1, x2) = μ + x1β1 + x2β2                   (1)
The observed characteristics are measured so that β1 
and β2 are positive.
Consider first, the (infeasible) validation study us-
ing the population of potential students at a particu-
lar institution. The validation study would reveal pre-
dicted performance y, conditional on x1 (standardized 
test scores), with expectations taken over the popula-
tion of potential students,
      E(y│1, x1) = μ + x1β1 + E(x2│1, x1)β2          (2)
             = μ′ + x1(β1 + b
p
21
β2)                    (3)
where bp
21
 = cov(x1, x2)/var(x1). The coefficient on test 
scores, Bp
1
 = β1 + b
p
21
β2, reflects not only the true mar-
ginal effect of test scores on performance but also the 
marginal effect of x2 and the relationship between test 
scores and x2 in the population. Because we presume 
that test scores measure ability, we would expect a 
positive correlation between test scores and under-
graduate performance in the population, (bp
21
 > 0). 
Therefore, Bp
1
 would be an upwardly biased measure 
of the marginal effect of test scores, β1. The low corre-
lations between performance and test scores in typi-
cal validation studies would appear to indicate that 
the marginal effect of test scores on performance is 
small and that institutions should give little weight to 
test scores in the admissions decision.
Now consider actual validation studies, conducted 
only on matriculating students. In the population 
of matriculating students, the relationship between 
mean performance and x1 is
ET (y│1, x1) = μ + x1β1 + ET (x2│1, x1)β2          (4)
with expectations taken over the truncated subpop-
ulation of matriculating students at institution i. Be-
cause the truncated distribution of x1 and x2 is not 
normal, the conditional expectations in (4) are not lin-
ear in x1. Letting b
s
21
 = ∂ET [x2│1, x1]/∂x1│x1= ‾x1, we can 
approximate (4) with
ET (y│1, x1) ≈ μ″ + x1(β1 + b
s
21
β2)                             (5)
Comparing (5) and (3) tells us that the results of the 
actual validation study will differ from the hypothet-
ical study to the extent that the relationship between 
x1 and x2 in the truncated subpopulation differs from 
that in the population of potential students.
To properly interpret equation (5), we provide a 
simple model of the selection process. Matriculating 
students are a subpopulation of the group of poten-
tial students that have passed through an admissions 
filter and have then chosen to attend the institution. 
Let institution i’s acceptance criterion be expressed as 
the following rule:
                 If γ1x1 + γ2x2 > Mi,  accept student.
Otherwise, reject student.                       (6)
sa M p l e se l e C t i o n i n Mo d e l s o f  aC a d e M i C pe r f o r M a n C e   321
The variables in the admissions criterion are mea-
sured so that the γ’s are positive. The next higher-
ranking institution has the same decision rule, but 
with a higher admissions standard, Mi+1 > Mi. Assum-
ing students will always choose to attend the highest-
ranking institution for which they qualify, the stu-
dent’s acceptance criterion for the i-th institution is
                   If γ1x1 + γ2x2 > Mi + 1,  reject offer.
Otherwise, accept offer.                        (7)
The acceptance and rejection decisions on the part of 
the i-th institution and the applicants make the sub-
population of matriculating students a truncated 
multivariate normal with truncation rule,
Mi < γ1x1 + γ2x2 < Mi + 1                  (8)
The relationship between x1 and x2 induced by the 
sample selection rules follows from the results of Kotz 
et al. (2000). The conditional expectation of x2 doubly 
truncated by the rule, M1 < γ1x1 + γ2x2 < M2 can be ex-
pressed as
              ET (x2│1, x1) = μ′′′ + b
p
21
x1 + σ(φ[1] 
– φ[2])/(Φ[2] – Φ[1])        (9)
where φ(∙) and Φ(∙) are the PDF and CDF, respec-
tively, of a standard normal, with 1 = σ
–1([M1 – 
x1γ1]/γ2 – x1b
p
21
), 2 = σ
–1([M2 – x1γ1 ]/γ2 – x1b
p
21
) and 
σ is the conditional standard deviation of x2, absent 
truncation.
Recall that we defined bs
21
  as the derivative of the 
conditional expectation in (9), evaluated at the mean 
of x1. This derivative can be shown to be
   bs
21
 = ∂ET [x2│1, x1]/∂x1│x1= ‾x1
= [1 – δ(‾1, ‾2)] b
p
21 
+ δ(‾1, ‾2)(–γ1/γ2)       (10)
where ‾1 = σ–1([M1 – ‾x1γ1]/γ2 – ‾x1b
p
21
),  ‾2 = σ–1([M2 – 
‾x1γ1 ]/γ2 – ‾x1b
p
21) and δ(‾1, ‾2) = 1 – varT (x2│1, x1)│x1= ‾x1 
/var(x2│1, x1). In other words, b
s
21
 is a convex com-
bination of the parameter bp
21
 that describes the rela-
tionship between x1 and x2 in the population of po-
tential students and the weights, –γ1/γ2, assigned to 
x1 and x2 in the admissions rule. The weights in this 
convex combination depend on the relative variances 
of x2 in the population of potential students and the 
population of matriculating students.
With these results, we can compare standard vali-
dation studies with the ideal study conducted on the 
population of potential students. The traditional vali-
dation study gives a coefficient on test scores of
        Bs1 = β1 + ([1 – δ(1, 2)] b
p
21
 
+ δ(1, 2)(–γ1/γ2)) β2             (11)
= Bp1 – (δ[1, 2][b
p
21
 + γ1/γ2]) β2               (12)
The second term in (12) reflects the effects of 
truncation.
At one extreme, consider institutions with rela-
tively open admissions policies that do not face com-
petition from more highly ranked institutions. For 
these institutions, the population of students is a ran-
dom sample from the population of potential stu-
dents. The conditional variance of x2 in the matric-
ulating population is close to that of the applicant 
population, so δ(1, 2) → 0, b
s
21
 → bp
21
 and Bs1 → B
p
1 
. For the open-admissions institution faced with lit-
tle competition, a validation study on the matriculat-
ing population will generally provide an upwardly 
biased estimate of β1, the true marginal effect of test 
scores, but provide an unbiased estimate of Bp1 the 
predictive content of test scores alone.
At the other extreme, consider a selective insti-
tution in a highly competitive market for students. 
Here, the gap between the admissions standard at 
competing institutions may be very small, Mi → Mi + 1. 
As this gap narrows, 1 → 2, δ(1, 2) → 1, and b
2
21
 → 
–γ1/γ2. The coefficient on x1 from the validation study 
will be a downwardly biased estimate of the true 
marginal effect of test scores, Bs1 → β1 – (γ1/γ2)β2, and 
a downwardly biased estimate of the predictive con-
tent of test scores in the population, Bs1 → B
p
1 – (b
p
21
 + 
γ1/γ2)β2.
For selective institutions in a competitive market 
for students, the interpretation of Bs1 depends on the 
relative weights attached to x1 and x2 in the admis-
sions process, γ1/γ2. The optimal weights would pre-
sumably be proportional to the relative marginal con-
tributions to ability. If institutions place relatively too 
much weight on test scores (γ1/γ2 > β1/β2), B
s
1 will 
be negative, whereas if they place relatively too little 
weight on test scores, Bs1 will be positive. If institu-
tions weight test scores optimally (γ1/γ2 = β1/β2), B
s
1 
322  Cu s h i n g & MCga r v e y i n Ec o n o m i c in q u i r y 42 (2004)
should be close to zero. In other words, a small cor-
relation between test scores and performance need 
not indicate that test scores are useless as predic-
tors of performance. Instead a small correlation may 
simply indicate that institutions are using test scores 
optimally.
III. Discussion and Conclusion
This article provided a formal model of how the 
admission and enrollment process affects the inter-
pretation of simple validation studies. Under plau-
sible assumptions, we found that the admission and 
acceptance process systematically reduces the corre-
lation between test scores and performance relative 
to the relationship in the population of potential stu-
dents. Traditional validation studies may mislead in-
stitutions into ignoring useful information about the 
potential productivity of prospective students.
The model assumed that institutions seek to pre-
dict performance using undergraduate performance 
in addition to test scores. In practice, admissions 
committees often examine a host of qualitative infor-
mation (e.g., quality of recommendation letters, un-
dergraduate coursework), whereas validation stud-
ies typically use a small number of easily quantifiable 
variables. Of course, admissions decisions are some-
times based on considerations other than expected 
ability, and admissions based on these other consid-
erations would tend to mitigate the bias emphasized 
in this study.
The model also assumed that each institution op-
erates in a competitive environment where appli-
cants base enrollment decisions solely on the ranking 
of institutions. All but the very top institutions face 
competition from more highly ranked programs. Of 
course, to the extent that applicants weigh other fac-
tors (e.g., location, tuition) in their enrollment deci-
sions, the hypothesized bias due to the selection pro-
cess will be mitigated.
One implication of our model is potentially test-
able. The correlation between test scores and other 
productivity characteristics should be systematically 
different in the sample of matriculating students and 
the population of potential students. We tested this 
proposition on the sample of students and applicants 
to the master’s of business administration (MBA) pro-
gram at the University of Nebraska–Lincoln. We cal-
culated the coefficient, bs
21
 from a regression of un-
dergraduate GPA on GMAT scores from currently 
enrolled students and compared it to the regression 
coefficient, bp
21
 using the pool of applicants as the 
population of potential students.
For male applicants, we obtained bˆs
21
 = 0.0038 with 
standard error 0.00097; for males currently in the pro-
gram, bˆs
21
 = 0.0023 with standard error 0.00425. For fe-
males, as is often the case, we found the correlations 
between test scores and performance to be lower for 
both the matriculating and applicant sample. The es-
timate of bp
21
 for female applicants is 0.0014 with stan-
dard error 0.00095, and the estimate of bs
21 for female 
students in the program is 0.0007 with standard error 
0.00099.
The selection process at the MBA program was not 
sufficient to render the correlation between test scores 
and undergraduate GPAs negative. The University of 
Nebraska-Lincoln MBA program draws largely from a 
regional population and does not face direct competi-
tion from higher-ranking institutions. Further, knowl-
edge of the selection rule at UNL and competing in-
stitutions may make the pool of applicants a truncated 
sample of the pool of potential students. Neverthe-
less, the correlation between undergraduate GPAs and 
GMAT scores is almost twice as high in the applicant 
pool as in the pool of matriculating students. Standard 
validation studies will therefore understate the impor-
tance of GMAT scores in predicting performance of 
potential students to the MBA program.
References
Cornell, D. G. “Residual Validity Is Not Meaningful.” Ameri-
can Psychologist, 53, 1998, 575-76.
Darlington, R. B. “Range Restriction and the Graduate Record 
Examination.” American Psychologist, 53, 1998, 572-73.
Graduate Record Examinations Board of the Educational Test-
ing Service. “Guide to the Use of Scores.” Princeton, NJ, 
1998.
Hansen, W. L. “Prediction of Graduate Performance in Eco-
nomics.” Journal of Economic Education, Fall 1971, 49-53.
Hanson, B. A,, and T. W. Harrell. “Predictors of Business Suc-
cess over Two Decades: An MBA Longitudinal Study.” 
Stanford Graduate School of Business Research Paper No. 
788, 1985.
Kotz, S., N. Balakrishnan, and N. Johnson. Continuous Multi-
variate Distributions, Volume 1, Models and Applications. New 
York: Wiley, 2000.
Sternberg, R. J., and W. M. Williams. “Does the Graduate Re-
cord Examination Predict Meaningful Success in the Grad-
uate Training of Psychologists?” American Psychologist, 
52(6), 1997, 630-41.


Sample Selection in Models of Academic Performance

DigitalCommons@University of Nebraska

University of Nebraska - Lincoln DigitalCommons@University of Nebraska - Lincoln Economics Department Faculty Publications Economics Department 4-2004 Sample Selection in Models of Academic Performance Matthew Cushing University of Nebraska-Lincoln, MCUSHING1@UNL.EDU Mary McGarvey University of Nebraska-Lincoln, MMCGARVEY1@UNL.EDU Follow this and additional works at: https://digitalcommons.unl.edu/econfacpub  Part of the Economics Commons Cushing, Matthew and McGarvey, Mary, "Sample Selection in Models of Academic Performance" (2004). Economics Department Faculty Publications. 42. https://digitalcommons.unl.edu/econfacpub/42 This Article is brought to you for free and open access by the Economics Department at DigitalCommons@University of Nebraska - Lincoln. It has been accepted for inclusion in Economics Department Faculty Publications by an authorized administrator of DigitalCommons@University of Nebraska - Lincoln. I. Introduction Virtually all graduate programs in the United States require applicants to submit scores on stan-dardized tests, such as the Graduate Record Examina-tion (GRE) or the Graduate Management Admissions Test (GMAT). The usefulness of these tests as predic-tors of graduate student performance, however, has never been firmly established. In fact, formal statis-tical investigations (validation studies) typically find that standardized tests have surprisingly little predic-tive content. Hanson and Harrell (1985) report a neg-ative correlation between GMAT scores and subse-quent compensation, Hansen (1971) reports very little correlation between GRE scores and graduate grade point averages (GPAs) in economics, and Sternberg and Williams (1997) find GRE scores of limited use in predicting performance in psychology. Even a re-cent study by the GRE Board of the Educational Test-ing Service (1998) shows relatively small correlations between first-year graduate GPA and general GRE scores. Why do admissions committees pay so much attention to these scores when formal evidence sug-gests that they have little or no predictive content?In this article, we suggest that a form of sample selection bias makes interpretation of standard val-idation studies problematic. As Darlington (1998) and Cornell (1998) point out, validation studies use data on students who have matriculated at a partic-ular institution. The admission procedure and accep-tance decisions make these data a censored sample from the population of prospective students. Institu-tions occasionally admit students with atypically low test scores, but only if these students present other, countervailing evidence of high ability. Students with atypically high test scores occasionally will choose to enroll, but only if countervailing evidence of low abil-ity precluded their admission to higher-ranking insti-tutions. Given that both high and low scorers exhibit some other, countervailing evidence on ability, it is not surprising that standardized test scores by them-selves are not strong predictors of performance.This article provides a formal model of how the se-lection process influences the results of standard val-idation studies. We find that in a competitive market for students, optimal behavior on the part of admis-sions committees and on the part of applicants drives the simple correlation between test scores and perfor-Published in Economic Inquiry 42:2 (April 2004), pp. 319-322; doi 10.1093/ei/cbh063   Copyright © 2004 Western Economic Association International. Used by permission.The authors thank an anonymous referee for Economic Inquiry for useful suggestions.Sample Selection in Models of Academic PerformanceMatthew J. Cushing and Mary G. McGarveyUniversity of Nebraska–LincolnM. J. Cushing is Associate Professor, University of Nebraska–Lincoln, Lincoln, NE 68588;  tel 402 472-2323, fax 402 472-9700, email mcushing1@unl.edu M. G. McGarvey is Associate Professor, University of Nebraska–Lincoln, Lincoln, NE 68588;  tel 402 472-9415, fax 402 472-9700, email mmcgarveyl@unl.eduAbstractThis article shows how admission and enrollment processes affect the interpretation of simple validation studies of academic performance. In a competitive market for students, optimal behavior of admissions committees and applicants drives the simple correlation between test scores and performance toward zero, regardless of the relationship in the population of prospective students. Data from our university’s MBA program support the prediction that applicants exhibit a higher correlation between test scores and undergraduate GPAs than do current students. This suggests that standard validation studies will under-state the importance of GMAT scores in predicting performance of potential MBA students. Abbreviations: GMAT, Graduate Management Admissions Test; GPA, Grade Point Average; GRE, Grad-uate Record Examination; MBA, Master’s of Business Administration319320  Cu s h i n g & MCga r v e y i n Ec o n o m i c in q u i r y 42 (2004)mance toward zero, regardless of the relationship in the population of prospective students. The key fea-tures of the model are that institutions select students (and students self-select) based on more information than test scores alone and that a competitive market for students results in institutions enrolling students of relatively homogenous predicted ability. The sam-ple selection process generates a negative correlation between test scores and other observable informa-tion, which, as the market becomes more competitive, drives the simple correlation between test scores and performance toward zero.II. A Simple Analytic ModelAssume that a simple scalar variable, y, measures scholastic ability in the population of potential stu-dents to a particular institution. This ability variable will be revealed perfectly (as performance) if the stu-dent matriculates but cannot be observed before ma-triculation. The admissions committee at this and other institutions must make acceptance and rejection decisions based on observable variables that may be correlated with this performance variable. The com-mittee observes characteristics, xl and x2, which, for concreteness, we interpret as the score on a standard-ized test and an index of undergraduate performance, respectively. Assuming joint normality of three vari-ables, y, xl, and x2, in the population of prospective students, expected ability (and performance) will be linear in the two observable variables,E(y│1, x1, x2) = μ + x1β1 + x2β2                   (1)The observed characteristics are measured so that β1 and β2 are positive.Consider first, the (infeasible) validation study us-ing the population of potential students at a particu-lar institution. The validation study would reveal pre-dicted performance y, conditional on x1 (standardized test scores), with expectations taken over the popula-tion of potential students,      E(y│1, x1) = μ + x1β1 + E(x2│1, x1)β2          (2)             = μ′ + x1(β1 + bp21β2)                    (3)where bp21 = cov(x1, x2)/var(x1). The coefficient on test scores, Bp1 = β1 + bp21β2, reflects not only the true mar-ginal effect of test scores on performance but also the marginal effect of x2 and the relationship between test scores and x2 in the population. Because we presume that test scores measure ability, we would expect a positive correlation between test scores and under-graduate performance in the population, (bp21 > 0). Therefore, Bp1 would be an upwardly biased measure of the marginal effect of test scores, β1. The low corre-lations between performance and test scores in typi-cal validation studies would appear to indicate that the marginal effect of test scores on performance is small and that institutions should give little weight to test scores in the admissions decision.Now consider actual validation studies, conducted only on matriculating students. In the population of matriculating students, the relationship between mean performance and x1 isET (y│1, x1) = μ + x1β1 + ET (x2│1, x1)β2          (4)with expectations taken over the truncated subpop-ulation of matriculating students at institution i. Be-cause the truncated distribution of x1 and x2 is not normal, the conditional expectations in (4) are not lin-ear in x1. Letting bs21 = ∂ET [x2│1, x1]/∂x1│x1= ‾x1, we can approximate (4) withET (y│1, x1) ≈ μ″ + x1(β1 + bs21β2)                             (5)Comparing (5) and (3) tells us that the results of the actual validation study will differ from the hypothet-ical study to the extent that the relationship between x1 and x2 in the truncated subpopulation differs from that in the population of potential students.To properly interpret equation (5), we provide a simple model of the selection process. Matriculating students are a subpopulation of the group of poten-tial students that have passed through an admissions filter and have then chosen to attend the institution. Let institution i’s acceptance criterion be expressed as the following rule:                 If γ1x1 + γ2x2 > Mi,  accept student.Otherwise, reject student.                       (6)sa M p l e se l e C t i o n i n Mo d e l s o f  aC a d e M i C pe r f o r M a n C e   321The variables in the admissions criterion are mea-sured so that the γ’s are positive. The next higher-ranking institution has the same decision rule, but with a higher admissions standard, Mi+1 > Mi. Assum-ing students will always choose to attend the highest-ranking institution for which they qualify, the stu-dent’s acceptance criterion for the i-th institution is                   If γ1x1 + γ2x2 > Mi + 1,  reject offer.Otherwise, accept offer.                        (7)The acceptance and rejection decisions on the part of the i-th institution and the applicants make the sub-population of matriculating students a truncated multivariate normal with truncation rule,Mi < γ1x1 + γ2x2 < Mi + 1                  (8)The relationship between x1 and x2 induced by the sample selection rules follows from the results of Kotz et al. (2000). The conditional expectation of x2 doubly truncated by the rule, M1 < γ1x1 + γ2x2 < M2 can be ex-pressed as              ET (x2│1, x1) = μ′′′ + bp21x1 + σ(φ[1] – φ[2])/(Φ[2] – Φ[1])        (9)where φ(∙) and Φ(∙) are the PDF and CDF, respec-tively, of a standard normal, with 1 = σ–1([M1 – x1γ1]/γ2 – x1bp21), 2 = σ–1([M2 – x1γ1 ]/γ2 – x1bp21) and σ is the conditional standard deviation of x2, absent truncation.Recall that we defined bs21  as the derivative of the conditional expectation in (9), evaluated at the mean of x1. This derivative can be shown to be   bs21 = ∂ET [x2│1, x1]/∂x1│x1= ‾x1= [1 – δ(‾1, ‾2)] bp21 + δ(‾1, ‾2)(–γ1/γ2)       (10)where ‾1 = σ–1([M1 – ‾x1γ1]/γ2 – ‾x1bp21),  ‾2 = σ–1([M2 – ‾x1γ1 ]/γ2 – ‾x1bp21) and δ(‾1, ‾2) = 1 – varT (x2│1, x1)│x1= ‾x1 /var(x2│1, x1). In other words, bs21 is a convex com-bination of the parameter bp21 that describes the rela-tionship between x1 and x2 in the population of po-tential students and the weights, –γ1/γ2, assigned to x1 and x2 in the admissions rule. The weights in this convex combination depend on the relative variances of x2 in the population of potential students and the population of matriculating students.With these results, we can compare standard vali-dation studies with the ideal study conducted on the population of potential students. The traditional vali-dation study gives a coefficient on test scores of        Bs1 = β1 + ([1 – δ(1, 2)] bp21 + δ(1, 2)(–γ1/γ2)) β2             (11)= Bp1 – (δ[1, 2][bp21 + γ1/γ2]) β2               (12)The second term in (12) reflects the effects of truncation.At one extreme, consider institutions with rela-tively open admissions policies that do not face com-petition from more highly ranked institutions. For these institutions, the population of students is a ran-dom sample from the population of potential stu-dents. The conditional variance of x2 in the matric-ulating population is close to that of the applicant population, so δ(1, 2) → 0, bs21 → bp21 and Bs1 → Bp1 . For the open-admissions institution faced with lit-tle competition, a validation study on the matriculat-ing population will generally provide an upwardly biased estimate of β1, the true marginal effect of test scores, but provide an unbiased estimate of Bp1 the predictive content of test scores alone.At the other extreme, consider a selective insti-tution in a highly competitive market for students. Here, the gap between the admissions standard at competing institutions may be very small, Mi → Mi + 1. As this gap narrows, 1 → 2, δ(1, 2) → 1, and b221 → –γ1/γ2. The coefficient on x1 from the validation study will be a downwardly biased estimate of the true marginal effect of test scores, Bs1 → β1 – (γ1/γ2)β2, and a downwardly biased estimate of the predictive con-tent of test scores in the population, Bs1 → Bp1 – (bp21 + γ1/γ2)β2.For selective institutions in a competitive market for students, the interpretation of Bs1 depends on the relative weights attached to x1 and x2 in the admis-sions process, γ1/γ2. The optimal weights would pre-sumably be proportional to the relative marginal con-tributions to ability. If institutions place relatively too much weight on test scores (γ1/γ2 > β1/β2), Bs1 will be negative, whereas if they place relatively too little weight on test scores, Bs1 will be positive. If institu-tions weight test scores optimally (γ1/γ2 = β1/β2), Bs1 322  Cu s h i n g & MCga r v e y i n Ec o n o m i c in q u i r y 42 (2004)should be close to zero. In other words, a small cor-relation between test scores and performance need not indicate that test scores are useless as predic-tors of performance. Instead a small correlation may simply indicate that institutions are using test scores optimally.III. Discussion and ConclusionThis article provided a formal model of how the admission and enrollment process affects the inter-pretation of simple validation studies. Under plau-sible assumptions, we found that the admission and acceptance process systematically reduces the corre-lation between test scores and performance relative to the relationship in the population of potential stu-dents. Traditional validation studies may mislead in-stitutions into ignoring useful information about the potential productivity of prospective students.The model assumed that institutions seek to pre-dict performance using undergraduate performance in addition to test scores. In practice, admissions committees often examine a host of qualitative infor-mation (e.g., quality of recommendation letters, un-dergraduate coursework), whereas validation stud-ies typically use a small number of easily quantifiable variables. Of course, admissions decisions are some-times based on considerations other than expected ability, and admissions based on these other consid-erations would tend to mitigate the bias emphasized in this study.The model also assumed that each institution op-erates in a competitive environment where appli-cants base enrollment decisions solely on the ranking of institutions. All but the very top institutions face competition from more highly ranked programs. Of course, to the extent that applicants weigh other fac-tors (e.g., location, tuition) in their enrollment deci-sions, the hypothesized bias due to the selection pro-cess will be mitigated.One implication of our model is potentially test-able. The correlation between test scores and other productivity characteristics should be systematically different in the sample of matriculating students and the population of potential students. We tested this proposition on the sample of students and applicants to the master’s of business administration (MBA) pro-gram at the University of Nebraska–Lincoln. We cal-culated the coefficient, bs21 from a regression of un-dergraduate GPA on GMAT scores from currently enrolled students and compared it to the regression coefficient, bp21 using the pool of applicants as the population of potential students.For male applicants, we obtained bˆs21 = 0.0038 with standard error 0.00097; for males currently in the pro-gram, bˆs21 = 0.0023 with standard error 0.00425. For fe-males, as is often the case, we found the correlations between test scores and performance to be lower for both the matriculating and applicant sample. The es-timate of bp21 for female applicants is 0.0014 with stan-dard error 0.00095, and the estimate of bs21 for female students in the program is 0.0007 with standard error 0.00099.The selection process at the MBA program was not sufficient to render the correlation between test scores and undergraduate GPAs negative. The University of Nebraska-Lincoln MBA program draws largely from a regional population and does not face direct competi-tion from higher-ranking institutions. Further, knowl-edge of the selection rule at UNL and competing in-stitutions may make the pool of applicants a truncated sample of the pool of potential students. Neverthe-less, the correlation between undergraduate GPAs and GMAT scores is almost twice as high in the applicant pool as in the pool of matriculating students. Standard validation studies will therefore understate the impor-tance of GMAT scores in predicting performance of potential students to the MBA program.ReferencesCornell, D. G. “Residual Validity Is Not Meaningful.” Ameri-can Psychologist, 53, 1998, 575-76.Darlington, R. B. “Range Restriction and the Graduate Record Examination.” American Psychologist, 53, 1998, 572-73.Graduate Record Examinations Board of the Educational Test-ing Service. “Guide to the Use of Scores.” Princeton, NJ, 1998.Hansen, W. L. “Prediction of Graduate Performance in Eco-nomics.” Journal of Economic Education, Fall 1971, 49-53.Hanson, B. A,, and T. W. Harrell. “Predictors of Business Suc-cess over Two Decades: An MBA Longitudinal Study.” Stanford Graduate School of Business Research Paper No. 788, 1985.Kotz, S., N. Balakrishnan, and N. Johnson. Continuous Multi-variate Distributions, Volume 1, Models and Applications. New York: Wiley, 2000.Sternberg, R. J., and W. M. Williams. “Does the Graduate Re-cord Examination Predict Meaningful Success in the Grad-uate Training of Psychologists?” American Psychologist, 52(6), 1997, 630-41.

https://digitalcommons.unl.edu/cgi/viewcontent.cgi?article=1041&context=econfacpub

Sample Selection in Models of Academic Performance

Abstract

Similar works

Full text

Available Versions

UNL | Libraries

DigitalCommons@University of Nebraska