Bayer, Amanda

English

Works

Appendix D: The Econometric Analysis Of The Benefits Of School-Based Mentoring

Swarthmore College
Works
Economics Faculty Works Economics
12-1-2004
Appendix D: The Econometric Analysis Of The
Benefits Of School-Based Mentoring
Amanda Bayer
Swarthmore College, abayer1@swarthmore.edu
Let us know how access to these works benefits you
Follow this and additional works at: https://works.swarthmore.edu/fac-economics
Part of the Economics Commons
This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 4.0 License.
This Paper is brought to you for free and open access by the Economics at Works. It has been accepted for inclusion in Economics Faculty Works by an
authorized administrator of Works. For more information, please contact myworks@swarthmore.edu.
Recommended Citation
Amanda Bayer. (2004). "Appendix D: The Econometric Analysis Of The Benefits Of School-Based Mentoring". School-Based
Mentoring: A Closer Look. 69-40.
https://works.swarthmore.edu/fac-economics/366
Appendices 39
Appendix D: 
The Econometric Analysis of the Benefits 
of School-Based Mentoring
By Amanda Bayer
Table 1 summarizes the results of the regression analyses 
for the 24 outcome measures. The first two columns of data 
report the estimated coefficients and statistical significance 
for the two match-length variables (i.e., the six-to-nine-
month and nine-or-more-month groups as compared to the 
zero-to-six-month group), while the third column records 
whether matches of six to nine months experience the same 
effect on outcomes as do longer matches. The last two col-
umns report differences between the three participating 
programs and the Adjusted R2 (pseudo R2 in the case of 
dichotomous outcome variables), or goodness of fit, of each 
regression. The coefficients on the match-length variables 
indicate the additional change in the follow-up value of 
the outcome measure that youth in each of the two longer-
matched groups experience relative to youth who were 
mentored for less than six months. This change was statisti-
cally significant for seven outcome measures, as recorded 
in the table and discussed in Chapter V.
Additional Analyses
In addition to these basic regression analyses, we also 
conducted analyses designed to test our hypotheses more 
rigorously and to compensate for limitations of the data. 
Specifically, we were concerned about two forms of bias:
Selection Bias. Youth with longer match lengths could differ 
from youth with shorter matches in ways that we could not 
account for but that could affect youth’s receipt of benefits. 
For example, if teachers recommend their less motivated 
students earlier in the school year, these youth may have lon-
ger matches than better students. Alternatively, if the more 
motivated youth remain in the program for a longer period 
of time, it might appear that longer program participation 
leads to better outcomes, when in fact only the youth most 
able and motivated to improve over the school year decide 
to stay in the program and thus have longer matches. In 
either case, this type of bias could contribute to spurious 
associations between match length and benefits.
To help account for selection bias, we tried to use an addi-
tional statistical technique—two-stage least squares regres-
sion (Heckman, 1976)—that examines the extent to which 
results are affected by unobserved differences between par-
ticipants. However, our results are inconclusive because the 
goodness of fit of the first-stage regressions was very poor; 
the data set did not have the information necessary to pre-
dict match length well enough for each child.
Basic Regression Analyses
The results presented in Chapter V on the effects of match 
length on youth outcomes are based on regression analyses. 
This statistical technique allows us to isolate the effect of 
match length on individual outcomes by controlling for the 
effects of other variables, such as race and gender. In cases 
where the dependent variable is continuous (e.g., school 
liking, positive classroom behavior), ordinary least squares 
(OLS) regression was used as follows:
Y = a + b1X1 + b2X2 +...bKXK + e
where: Y = value of the dependent variable (i.e., the  
  follow-up value for the outcome measure of  
  interest);
 Xk = value of kth explanatory variable, k=1 to K;
 a, bk = coefficients; and
 e = a stochastic disturbance term with a mean  
  of zero and a constant variance.
In cases where the dependent variable is dichotomous (e.g., 
fighting in the month prior to the survey, principal’s office 
visits, absences, tardies) logistic regression analysis was used, 
using maximum likelihood estimation by specifying a linear 
function for the logit (the logarithm of the odds) of having 
a positive response on the dependent variable as follows:
log (p/[1–p]) = a + b1X1 + b2X2 +...bKXK + e
where:  p = the probability of having a positive response 
  on the dependent variable (i.e., the follow- 
  up value for the outcome measure of interest);
 1–p = the probability of having a negative  
  response on the dependent variable; and
 a, b, X and e are defined as in the OLS equation 
above.
All regressions include explanatory variables for ethnicity, 
gender, program, length of time between administration of 
the two surveys, the baseline level of the outcome measure 
(i.e., the value of the outcome measure at the beginning of 
the test period), and categorical variables indicating total 
duration of match (i.e., one indicating whether or not a 
match is at least nine months in duration, and a second indi-
cating whether or not a match is six to nine months in dura-
tion). Regressions for the academic performance outcomes 
also include grade level.4 Analyses using teacher-reported 
outcomes are based on approximately 80 youth. Analyses 
using youth-reported outcomes are based on approximately 
150 youth.
40 School-Based Mentoring: A Closer Look
We also tested whether the effects of match length found in 
our basic results appear within our match-length subgroups 
(i.e., six to nine months, nine or more months). Our concern 
was that the positive behaviors demonstrated by youth with 
more than nine months of mentoring might not have been 
a result of that mentoring; rather, it could be that the youth 
who chose to remain in the program from one academic year 
to the next were more motivated to improve in some unmea-
sured way. If longer matches do directly lead to better youth 
outcomes, then we would expect our findings to replicate 
within the match-length subgroups. The effects of match 
length were only present for one of the six outcomes (i.e., 
school liking) in which we found effects using the entire 
sample, suggesting that selection bias may be affecting coef-
ficient estimates in some of our basic regression analyses. 
However, these findings may be due, in part, to the small size 
of the subsamples: in four of the five remaining cases, regres-
sion coefficients for either or both of the subsamples were in 
the same direction and of a larger magnitude than statistically 
significant coefficients using the full sample.
We further investigated the possibility of selection bias by 
repeating our basic regression analyses, but eliminating 
those matches that ended before the follow-up survey for 
which we still have follow-up survey data. Youth who ended 
their match early may have traits that cause them to experi-
ence relatively poor gains over the test period, causing a spu-
rious positive association between match length and changes 
in the outcome measures in the original analysis. On the 
contrary, removing these early-ending matches, about 20 
percent of the original sample, actually yielded stronger pos-
itive correlations between overall match length and improve-
ments in behavior, indicating that our initial estimates were 
relatively free of this particular form of selection bias.
Truncation Bias. Another potential limitation of the data 
stems from the numerical scales used to record teacher and 
youth reports of outcomes. The highest possible score for 
many of our outcome measures is “4” or “5.” Thus, a child 
who starts with a fairly high score at the beginning of the 
school year could not show a large increase by the end of 
the year. In this way, one might expect to see smaller posi-
tive changes for children who start with better behavior and 
bigger changes for youth starting out with lower scores. If 
teachers match their most needy students (i.e., youth scor-
ing lower on our measures) earlier in the year, then we 
would see a spurious positive association between match 
length and improvements. If, on the other hand, youth scor-
ing higher at baseline have longer match lengths, then we 
would see a spurious negative association between match 
length and improvements.
To explore the extent to which truncation bias exists in our 
results we took several approaches. First, we ran a series 
of regressions using only those observations for which the 
outcome measures were not at extreme values at baseline. 
Second, we conducted analyses using standard methods 
of working with limited dependent variables, namely tobit 
regression and ordered logit. These approaches reinforced 
our original results: for all of the outcome measures with sta-
tistically significant match-length effects in the basic analysis, 
the size and statistical significance of the effects were at least 
as large in these additional analyses. Moreover, these meth-
ods revealed that improvements in an additional outcome 
measure, emotional disposition in the classroom, were asso-
ciated with longer mentoring matches.
Conclusions
Our assessment of the effectiveness of SBM programs pre-
sented in Chapter V withstands more rigorous testing as 
reported in this Appendix. While selection bias remains 
a concern, we could produce no strong evidence that the 
basic results are biased; this lack of definitive evidence, how-
ever, is largely a result of data limitations. Truncation bias is 
likely muting our results, and the analyses suggest that the 
effects of mentoring are even stronger than portrayed by 
the basic analysis. In sum, we must use extreme caution in 
interpreting the results reported here, and we recommend 
that future projects utilize a random-assignment design to 
determine the impacts of school-based mentoring.
Appendices 41
Appendix D: Table 1
Coefficient Estimates for Match Length Variables for 24 Outcome Measures
Outcome Effect Of Match Length Program  Adjusted 
  Effects R2
 6-9  9+ 6-9 months 
 months months vs. 
   9+ months
Social Skills and Networks     
Peer social network 0.082 0.314**   .48
Social skills 0.284 0.482**   .29
Relationships with Adults     
Adult social support 0.162 -0.100   .14
Relationship with parent 0.179 0.383   .14
Parent tells youth school is important 0.325 1.480   .12
Parent involvement in school 0.171 0.079   .55
Perception of teacher’s attitude toward child -0.226 -0.169  Prog. 1 > Prog. 3** .23
Academic Performance     
Study skills 0.156 0.274  Prog. 3 > Prog. 1* .59
Language -0.145 -0.048  Prog. 3 > Prog. 1*** .45
    Prog. 3 > Prog. 2**  
Social Studies 0.373 0.362   .40
Math 0.598 0.524   .36
Science 0.811 0.510  Prog. 3 > Prog. 2* .26
Percentage of in-class  0.117 0.103   .21
assignments not completed
Percentage of homework  0.111 0.066   .12
assignments not completed
Classroom Behavior and Attitude     
Fought in last 4 weeks 0.692 -0.728 9+<6-9*  .21
Positive classroom behavior 0.235 0.592*** 9+>6-9* Prog. 3 > Prog. 2* .55
Principal’s office visit in last 4 weeks -0.373 -2.318** 9+<6-9** Prog. 3 < Prog. 2* .24
Classroom effort 0.115 0.182  Prog. 3 > Prog. 2* .43
Academic engagement 0.199 0.350*  Prog. 3 > Prog. 2** .49
Classroom emotional disposition 0.042 0.214  Prog. 3 > Prog. 1** .44
School liking 0.350** 0.386**   .39
Attendance     
Absence in last 4 weeks -0.553 -1.024   .10
Tardy in last 4 weeks -0.021 -0.426   .13
Hygiene     
Hygiene/Appearance -0.078 0.225  Prog. 3 > Prog. 2* .42
Notes:
* Estimated coefficient is statistically different from zero at p < .10 significance level.
** Estimated coefficient is statistically different from zero at p < .05 significance level.
*** Estimated coefficient is statistically different from zero at p < .01 significance level.
42 School-Based Mentoring: A Closer Look
1 Youth whose teachers completed surveys at the second time 
point differed from youth without a teacher survey in only three 
ways: they were less likely to have parents who helped them with 
their school work, they had case managers who spoke more 
often with their parent or guardian, and they felt slightly less 
close to their mentor at the second time point.
2 Youth whose mentors completed the survey differed from youth 
without a mentor survey in the following ways: they were more 
likely to be female, had met with their mentor longer during 
the test period and, at the second time point, reported lower 
levels of adult support and perceived that their teachers had a 
less positive attitude toward them. Case managers reported that, 
relative to youth without mentor surveys, these youth had closer 
relationships with their mentors. Their mentors enjoyed spend-
ing time with them more, engaged in more positive behaviors 
toward the youth and participated more often in agency events. 
Case managers also reported that these youth had less direct 
supervision from the agency but benefited from more communi-
cation between the case manager and their parents. These youth 
did not differ in age, grade, ethnicity, single-parent status or any 
other outcome of interest in the study.
3 This is a measure (ranging from 0 to 1.00) of how well a set of 
variables reflects a single unidimensional construct. In this case, 
these alphas (or “reliability coefficients”) reflect how well the 
three items listed intercorrelate to measure “classroom emo-
tional disposition.”
4 In an initial set of regressions, we included grade level in all 
analyses. These analyses revealed significant effects for this vari-
able only when predicting academic performance. Thus, it was 
only retained in this subset of analyses.
Appendices Endnotes


Swarthmore CollegeWorksEconomics Faculty Works Economics12-1-2004Appendix D: The Econometric Analysis Of TheBenefits Of School-Based MentoringAmanda BayerSwarthmore College, abayer1@swarthmore.eduLet us know how access to these works benefits youFollow this and additional works at: https://works.swarthmore.edu/fac-economicsPart of the Economics CommonsThis work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 4.0 License.This Paper is brought to you for free and open access by the Economics at Works. It has been accepted for inclusion in Economics Faculty Works by anauthorized administrator of Works. For more information, please contact myworks@swarthmore.edu.Recommended CitationAmanda Bayer. (2004). "Appendix D: The Econometric Analysis Of The Benefits Of School-Based Mentoring". School-BasedMentoring: A Closer Look. 69-40.https://works.swarthmore.edu/fac-economics/366Appendices 39Appendix D: The Econometric Analysis of the Benefits of School-Based MentoringBy Amanda BayerTable 1 summarizes the results of the regression analyses for the 24 outcome measures. The first two columns of data report the estimated coefficients and statistical significance for the two match-length variables (i.e., the six-to-nine-month and nine-or-more-month groups as compared to the zero-to-six-month group), while the third column records whether matches of six to nine months experience the same effect on outcomes as do longer matches. The last two col-umns report differences between the three participating programs and the Adjusted R2 (pseudo R2 in the case of dichotomous outcome variables), or goodness of fit, of each regression. The coefficients on the match-length variables indicate the additional change in the follow-up value of the outcome measure that youth in each of the two longer-matched groups experience relative to youth who were mentored for less than six months. This change was statisti-cally significant for seven outcome measures, as recorded in the table and discussed in Chapter V.Additional AnalysesIn addition to these basic regression analyses, we also conducted analyses designed to test our hypotheses more rigorously and to compensate for limitations of the data. Specifically, we were concerned about two forms of bias:Selection Bias. Youth with longer match lengths could differ from youth with shorter matches in ways that we could not account for but that could affect youth’s receipt of benefits. For example, if teachers recommend their less motivated students earlier in the school year, these youth may have lon-ger matches than better students. Alternatively, if the more motivated youth remain in the program for a longer period of time, it might appear that longer program participation leads to better outcomes, when in fact only the youth most able and motivated to improve over the school year decide to stay in the program and thus have longer matches. In either case, this type of bias could contribute to spurious associations between match length and benefits.To help account for selection bias, we tried to use an addi-tional statistical technique—two-stage least squares regres-sion (Heckman, 1976)—that examines the extent to which results are affected by unobserved differences between par-ticipants. However, our results are inconclusive because the goodness of fit of the first-stage regressions was very poor; the data set did not have the information necessary to pre-dict match length well enough for each child.Basic Regression AnalysesThe results presented in Chapter V on the effects of match length on youth outcomes are based on regression analyses. This statistical technique allows us to isolate the effect of match length on individual outcomes by controlling for the effects of other variables, such as race and gender. In cases where the dependent variable is continuous (e.g., school liking, positive classroom behavior), ordinary least squares (OLS) regression was used as follows:Y = a + b1X1 + b2X2 +...bKXK + ewhere: Y = value of the dependent variable (i.e., the    follow-up value for the outcome measure of    interest); Xk = value of kth explanatory variable, k=1 to K; a, bk = coefficients; and e = a stochastic disturbance term with a mean    of zero and a constant variance.In cases where the dependent variable is dichotomous (e.g., fighting in the month prior to the survey, principal’s office visits, absences, tardies) logistic regression analysis was used, using maximum likelihood estimation by specifying a linear function for the logit (the logarithm of the odds) of having a positive response on the dependent variable as follows:log (p/[1–p]) = a + b1X1 + b2X2 +...bKXK + ewhere:  p = the probability of having a positive response   on the dependent variable (i.e., the follow-   up value for the outcome measure of interest); 1–p = the probability of having a negative    response on the dependent variable; and a, b, X and e are defined as in the OLS equation above.All regressions include explanatory variables for ethnicity, gender, program, length of time between administration of the two surveys, the baseline level of the outcome measure (i.e., the value of the outcome measure at the beginning of the test period), and categorical variables indicating total duration of match (i.e., one indicating whether or not a match is at least nine months in duration, and a second indi-cating whether or not a match is six to nine months in dura-tion). Regressions for the academic performance outcomes also include grade level.4 Analyses using teacher-reported outcomes are based on approximately 80 youth. Analyses using youth-reported outcomes are based on approximately 150 youth.40 School-Based Mentoring: A Closer LookWe also tested whether the effects of match length found in our basic results appear within our match-length subgroups (i.e., six to nine months, nine or more months). Our concern was that the positive behaviors demonstrated by youth with more than nine months of mentoring might not have been a result of that mentoring; rather, it could be that the youth who chose to remain in the program from one academic year to the next were more motivated to improve in some unmea-sured way. If longer matches do directly lead to better youth outcomes, then we would expect our findings to replicate within the match-length subgroups. The effects of match length were only present for one of the six outcomes (i.e., school liking) in which we found effects using the entire sample, suggesting that selection bias may be affecting coef-ficient estimates in some of our basic regression analyses. However, these findings may be due, in part, to the small size of the subsamples: in four of the five remaining cases, regres-sion coefficients for either or both of the subsamples were in the same direction and of a larger magnitude than statistically significant coefficients using the full sample.We further investigated the possibility of selection bias by repeating our basic regression analyses, but eliminating those matches that ended before the follow-up survey for which we still have follow-up survey data. Youth who ended their match early may have traits that cause them to experi-ence relatively poor gains over the test period, causing a spu-rious positive association between match length and changes in the outcome measures in the original analysis. On the contrary, removing these early-ending matches, about 20 percent of the original sample, actually yielded stronger pos-itive correlations between overall match length and improve-ments in behavior, indicating that our initial estimates were relatively free of this particular form of selection bias.Truncation Bias. Another potential limitation of the data stems from the numerical scales used to record teacher and youth reports of outcomes. The highest possible score for many of our outcome measures is “4” or “5.” Thus, a child who starts with a fairly high score at the beginning of the school year could not show a large increase by the end of the year. In this way, one might expect to see smaller posi-tive changes for children who start with better behavior and bigger changes for youth starting out with lower scores. If teachers match their most needy students (i.e., youth scor-ing lower on our measures) earlier in the year, then we would see a spurious positive association between match length and improvements. If, on the other hand, youth scor-ing higher at baseline have longer match lengths, then we would see a spurious negative association between match length and improvements.To explore the extent to which truncation bias exists in our results we took several approaches. First, we ran a series of regressions using only those observations for which the outcome measures were not at extreme values at baseline. Second, we conducted analyses using standard methods of working with limited dependent variables, namely tobit regression and ordered logit. These approaches reinforced our original results: for all of the outcome measures with sta-tistically significant match-length effects in the basic analysis, the size and statistical significance of the effects were at least as large in these additional analyses. Moreover, these meth-ods revealed that improvements in an additional outcome measure, emotional disposition in the classroom, were asso-ciated with longer mentoring matches.ConclusionsOur assessment of the effectiveness of SBM programs pre-sented in Chapter V withstands more rigorous testing as reported in this Appendix. While selection bias remains a concern, we could produce no strong evidence that the basic results are biased; this lack of definitive evidence, how-ever, is largely a result of data limitations. Truncation bias is likely muting our results, and the analyses suggest that the effects of mentoring are even stronger than portrayed by the basic analysis. In sum, we must use extreme caution in interpreting the results reported here, and we recommend that future projects utilize a random-assignment design to determine the impacts of school-based mentoring.Appendices 41Appendix D: Table 1Coefficient Estimates for Match Length Variables for 24 Outcome MeasuresOutcome Effect Of Match Length Program  Adjusted   Effects R2 6-9  9+ 6-9 months  months months vs.    9+ monthsSocial Skills and Networks     Peer social network 0.082 0.314**   .48Social skills 0.284 0.482**   .29Relationships with Adults     Adult social support 0.162 -0.100   .14Relationship with parent 0.179 0.383   .14Parent tells youth school is important 0.325 1.480   .12Parent involvement in school 0.171 0.079   .55Perception of teacher’s attitude toward child -0.226 -0.169  Prog. 1 > Prog. 3** .23Academic Performance     Study skills 0.156 0.274  Prog. 3 > Prog. 1* .59Language -0.145 -0.048  Prog. 3 > Prog. 1*** .45    Prog. 3 > Prog. 2**  Social Studies 0.373 0.362   .40Math 0.598 0.524   .36Science 0.811 0.510  Prog. 3 > Prog. 2* .26Percentage of in-class  0.117 0.103   .21assignments not completedPercentage of homework  0.111 0.066   .12assignments not completedClassroom Behavior and Attitude     Fought in last 4 weeks 0.692 -0.728 9+<6-9*  .21Positive classroom behavior 0.235 0.592*** 9+>6-9* Prog. 3 > Prog. 2* .55Principal’s office visit in last 4 weeks -0.373 -2.318** 9+<6-9** Prog. 3 < Prog. 2* .24Classroom effort 0.115 0.182  Prog. 3 > Prog. 2* .43Academic engagement 0.199 0.350*  Prog. 3 > Prog. 2** .49Classroom emotional disposition 0.042 0.214  Prog. 3 > Prog. 1** .44School liking 0.350** 0.386**   .39Attendance     Absence in last 4 weeks -0.553 -1.024   .10Tardy in last 4 weeks -0.021 -0.426   .13Hygiene     Hygiene/Appearance -0.078 0.225  Prog. 3 > Prog. 2* .42Notes:* Estimated coefficient is statistically different from zero at p < .10 significance level.** Estimated coefficient is statistically different from zero at p < .05 significance level.*** Estimated coefficient is statistically different from zero at p < .01 significance level.42 School-Based Mentoring: A Closer Look1 Youth whose teachers completed surveys at the second time point differed from youth without a teacher survey in only three ways: they were less likely to have parents who helped them with their school work, they had case managers who spoke more often with their parent or guardian, and they felt slightly less close to their mentor at the second time point.2 Youth whose mentors completed the survey differed from youth without a mentor survey in the following ways: they were more likely to be female, had met with their mentor longer during the test period and, at the second time point, reported lower levels of adult support and perceived that their teachers had a less positive attitude toward them. Case managers reported that, relative to youth without mentor surveys, these youth had closer relationships with their mentors. Their mentors enjoyed spend-ing time with them more, engaged in more positive behaviors toward the youth and participated more often in agency events. Case managers also reported that these youth had less direct supervision from the agency but benefited from more communi-cation between the case manager and their parents. These youth did not differ in age, grade, ethnicity, single-parent status or any other outcome of interest in the study.3 This is a measure (ranging from 0 to 1.00) of how well a set of variables reflects a single unidimensional construct. In this case, these alphas (or “reliability coefficients”) reflect how well the three items listed intercorrelate to measure “classroom emo-tional disposition.”4 In an initial set of regressions, we included grade level in all analyses. These analyses revealed significant effects for this vari-able only when predicting academic performance. Thus, it was only retained in this subset of analyses.Appendices Endnotes

https://works.swarthmore.edu/cgi/viewcontent.cgi?article=1365&context=fac-economics

Appendix D: The Econometric Analysis Of The Benefits Of School-Based Mentoring

Abstract

Similar works

Full text

Available Versions

Works

Works

Works