Experience Corps: Methodology of Member Outcomes Stud

Blinne, Wayne

Dessen, Emily

Halvorsen, Cal

Hong, Song-Iee

McCrary, Stacey

Morrow-Howell, Nancy

English

Washington University St. Louis: Open Scholarship

  Campus Box 1196  One Brookings Drive  St. Louis, MO  63130-9906  •  (314) 935.7433  •  www.gwbweb.wustl.edu/csd                         October 3, 2008   Experience Corps: Methodology of Member Outcomes Study           Experience Corps® (EC) is a program that brings older adults into public elementary schools to improve academic achievement of students, through one-to-one tutoring, small group academic help, and assisting teachers.  It has been in existence for over 13 years and currently operates in 20 cities across the country.  The Atlantic Philanthropies provided funding to Washington University in St. Louis to conduct a national evaluation.  Mathematica Policy Research (MPR) provided data collection services. The evaluation was designed to provide information about the EC members, their experiences in the program, and outcomes associated with participation.  Two cohorts of new members, one joining in Fall 2006 and the other in Fall 2007, were included in the sample.  A total of 730 members participated in a survey prior to beginning EC service; and a sub-sample of 213 members were followed for two years.                           Pre-test            Post-test             Pre-test            Post-test                              Aug-Nov 06          June 07           Aug-Sept 07          June 08  COHORT 1  COHORT 2  Referral of new members  In Fall 2006 and 2007, all EC sites were asked to send names and contact information of all new EC members to the evaluation team at WU.  Eligibility criteria included:  being at least 50 years of age, able to conduct the interview in English, and not dropping the EC program prior to the study interview.  Members could be full or part time, stipended or unstipended.   The following chart demonstrates that all sites participated in the study, and numbers of new recruits varied by the size of the program and the number of new tutors needed in these years.  ‘06 ‘07  ‘06 ‘07  Baltimore  32  22  New York  14  4  Bay Area  59  28  New Haven  -  29  Boston  60  42  Philadelphia  95  40  Chicago  8  -  Port Arthur  26  10  Cleveland  75  22  Portland  23  12  Grand Rapids  15  8  St. George  20  7  Indianapolis  39  -  Tempe  30  12  Mesa  6  17  Tucson  10  20  Minneapolis  33  32  Washington DC  56  29   Contacting and consenting Each new EC member was mailed a letter of introduction, consent form, informational brochure, and survey cards.  Contact information was forwarded to the research team at MPR, who attempted to reach each member by phone. The goal was to survey members before they received EC training or were placed in schools.   All interviews were conducted by phone and lasted approximately 45 minutes. Members were remunerated with a $20 gift card (Cohort 1 2nd post test subjects received a $30 gift card).  Training  Interviews were conducted by trained interviewers from MPR and WU.  Throughout the evaluation, there was constant contact between MPR and WU to maintain similar evaluation processes. Interviewers took part in a 2-day training and completed a certification interview.  Refresher classes were given prior to the start of each data collection period, and emphasis was placed on new questions added to the survey.  Interviewers were monitored by supervisors to ensure quality. Survey pre-test  The survey is largely comprised of questions that are standardized and used extensively in other national studies.  Most measures come from the Midlife in the US study (MIDUS) and the Health and Retirement Survey (HRS).  Other measures were selected and modified carefully for the evaluation.  The survey was pilot tested on 30 current EC members prior to finalizing.   2 Survey post-test  EC members were reminded of the follow-up phone interview through a letter and phone call to schedule the interview.  The post-test survey was identical to the pre-test, with the exception of added questions about program participation and open-ended questions regarding satisfaction with the program, perceived benefits of participation and intent to remain involved. 2nd post test  This survey was similar to the post-test, with the addition of questions about views/outlook on public education and more specific questions regarding stipends.   Flow chart of recruitment and interviewing                               Cohort 1  In total, 601 names were sent to us between August 06 and March 07.  Those individual who were not 50 years or older, who did not follow through with training, or could not conduct the interview in English were classified as ineligible.  There were 508 (84%) members eligible for the evaluation; and 463 (91%) surveys were completed. This larger sample was broken into two subsamples: those who started EC before mid-November (n=306) and were post-tested; and those who volunteered after mid-November (n=157) and were not included in the first post-test.  Initially, we planned to only recruit members who signed up through November; but many sites recruit throughout the academic year; thus, we extended our enrollment period.  Those members who were interviewed Aug-Nov were asked to participate in a post-test conducted after their tutoring services ended for the academic year (n=319).  We conducted interviews from May-July 2007.  We attained an 85% post-test completion rate (n=271).   A second post-test was completed on members of Cohort 1 who signed up for a second year of service (n=230). During May-July 2008, 214 (93%) members participated in this survey.  Cohort 2  From Aug-Nov, 334 names were sent to us. There were 293 (88%) eligible members; 267 (89%) members were interviewed.  Post-testing on this group was conducted May-July, 2008.   Quality control  All completed surveys were reviewed by trained editors. If necessary, additional phone calls were made to the member to clarify information or obtain missing responses.  Data entry and cleaning  To ensure accuracy, each survey was entered twice, and any discrepancies fixed. One final database exists for each round of data collection. Data analysis  Although there were little missing data, the data sets were completed using a Markov Chain Monte Carlo (MCMC) multiple imputation method.  Univariate analysis of all study variables were performed, and appropriate statistical analyses completed to answer study questions. Given that the observations are nested within cities, regression techniques were used to correct for this clustering.   Open-ended questions were content analyzed, and categories for responses were developed.  Two coders worked independently and then compared answers. Discrepancies were discussed and final codes were agreed upon.   In sum, a rigorous methodology was accomplished to study the EC members and their experiences in the program.  All new members across the country in each of two cohorts were recruited, and response rates were very high. Measurement was solid; and interviewers were well-trained and monitored. Findings are generalizable to all EC programs and will help guide program improvement and expansion.    Research team at Washington University in St. Louis:  Nancy Morrow-Howell, Stacey McCrary, Song-Iee Hong, Wayne Blinne, Emily Dessem, Cal Halvorsen Mathematica Policy Research provided data collection services:  Barbara Rogers, Jennifer McGovern This work was funded by The Atlantic Philanthropies Cohort 1  Fall 2006 601 names sent   508 eligible (84%) 463 interviews (91%) 306 interviewed Aug-Nov 06 157 interviewed Nov-April 271 post-tested May-July 07 213 2nd post tested May-July 08 Cohort 2 Fall 2007 334 names sent 293 eligible (88%) 267 complete (89%) 237 post-tested May-July 08 

Experience Corps: Methodology of Member Outcomes Study

https://openscholarship.wustl.edu/cgi/viewcontent.cgi?article=1855&context=csd_research

Experience Corps: Methodology of Member Outcomes Study

Abstract

Similar works

Full text

Available Versions

Washington University St. Louis: Open Scholarship