Moderator: Katie Barghaus
Presenters:   Sonya R. Porter:  Demonstrating the Use of Linked Data to Advance Education Research  Chenna Reddy Cotla:  Heterogeneous causal effects of perception of school safety on absenteeism and performance: A causal decision tree approach using administrative and survey data from a large school district  Sally Wallace:  Returns to Late Aged College Degrees  Douglas Lauen:  Early College High Schools at Scale: Using Administrative Data to Assess the Impacts of an Educational Intervention on Voting and Crime  Wesley Greenblatt:  Long-Term Effects from Early Exposure to Research: Evidence from the NIH \u27Yellow Berets\u2

Barghaus (Moderator), Katie

Greenblatt, Wesley

Lauen, Douglas

Porter, Sonya R

Reddy Cotla, Chenna

Wallace, Sally

English

Kosmopolis

C1: Uses of Administrative Data for Supporting Education Policy

ScholarlyCommons@Penn

Long-term Effects from Early Exposure to Research: Evidence from the NIH “Yellow Berets”Pierre Azoulay (MIT & NBER)Wesley Greenblatt (MIT)Misty L. Heggeness (U.S. Bureau of the Census)@m_heggenessDisclaimerThis presentation was prepared for the 2018 ADRF Conference in Washington, DC. It was developed to promote research and advancements in our understanding of the use of administrative records in household and person-level statistics. In that spirit and to encourage discussion and thoughtful feedback at early stages of our work, this presentation has undergone a more limited review than official Census Bureau reports. All views and any errors are solely those of the author and do not necessarily reflect any official position of the Bureau. Do not cite or distribute without author permission.Chart of Doom #1: Rising Research Effort, Flat Productivity0501001502002503001950 1955 1960 1965 1970 1975 1980 1985 1990 1995YearLevel (1975=100)-0.15-0.1-0.0500.050.10.15Growth Rateproductivity growth(right axis)real US R&Dexpenditures(left axis)US R&D workers(left axis)Chart of Doom #2: The Graying of the Scientific Workforce0.000.050.100.15Fraction of Investigators 0 5 10 15 20 25 30 19651975198519952005The question¾ Early careers in general, and early scientific careers in particular can be fragile (Oreopoulos et al. 2006, Hill 2018)¾ But fragility might go hand in hand with malleability (Higgins 2005)¾ Can one induce young people to become innovators?The supply of innovators: a brief review¾ “Innovating in Science and Engineering or ‘Cashing in’ on Wall Street? Evidence on Elite STEM Talent” (Shu 2016)– The marginal financier (i.e., MIT grad who pursued engineering rather than finance in the doldrums of the great recession) has a relatively low grades in STEM classes.¾ Recent paper by Graff Zivin and Lyons (2018)– No evidence of crowd out in a student innovation contest¾ Evidence from US inventors: exposure effects in childhood have long run impacts on the probability to patent (Bell et al. 2016)¾ Hill (2018) documents the fragility of early careers in astronomy using weather shocks to seeing conditionsThe notional experiment¾ Find a population of “naïve to research” individuals who nonetheless possess much of the human capital required to propel themselves to the research frontier¾ Provide to a (random?) subset of them a short but intense exposure to research in a rarefied intellectual environment¾ Wait 50 years to gauge the full effects of this short-term interventionA serendipitous find…The NIH Associate Training Programs¾ “Doctor Draft” initiated during the Korean War¾ Started in 1953 with a few dozen medical graduates– Two years in the US PHS Commissioned Corps– PHS CC also include CDC and IHS¾ Escalated during the Vietnam War– 1967: restrictions on exemptions available to physicians seeking deferment– Leads to increased selectivity of the program– But even in 1963, 53 of 1,464 physician applicants were selected (NIH Office of Research Information 1963)¾ Three sub-programs: RA, CA, SAProgram content and objectives¾ Turn physicians into independent medical investigators well grounded in modern scientific knowledge and methods. Associates should:– learn how to do research more than to do research itself– be brought into close contact with accomplished scientists in specialized research fields¾ By 1970, the NIH ATP was recognized as the place to get thorough training in biomedical research in the US (Broder 2001)“The importance of having the Research Associate[s] work on problems of [their] own choice rather than be ‘servants’ in the research problems of the preceptor, and the importance of providing the student[s] with some integrated and organized basic knowledge as a foundation that would permit them to do their own integrating of knowledge later.”—Christian Anfinsen (1963)Existing evidence¾ Khot et al. 2011 compares ATP attendees with a control sample of non-ATP medical school academics¾ Klein (1998) provides an historical analysis of the Yellow Berets’ “legacy”¾ None of these writings leverage data on the unsuccessful applicants, whose index cards were thought to have been destroyedPros and cons of using Medicine/NIH ATP as a setting¾ MDs acquire a lot of human capital over the course of their training, but face the choice of deploying it across two different settings:– The production (aka clinical) setting, where their HC generates mostly private returns– The research setting, where the same HC also generates social returns– Long-standing goal of the medical elite: steer a larger number of physicians towards research careers (Wyngaarden 1979; Rosenberg 1999)¾ Key limitation– external validity: NIH is a pretty unique place, and it became unique in large part because the alternative was VietnamThe (very) raw data…1967 1968 1969 1970 1971 1972 1973Vietnam War Lottery Draft YearsN=3,075 Second Round Applicants1,887 (61.37%) Attendees; 1,188 (38.63%) Non-attendees0100200300400500Nb. of Applicants1965 1966 1967 1968 1969 1970 1971 1972 1973 1974 1975Non-AttendeesAttendees7 Nobel Prize Winners, 32 HHMIs, 88 Members of the National Academies…Anthony Fauci, 1968Stanley N. Cohen, 1962 Robert C. Gallo, 1965Richard Axel, 1972Harold Varmus, 1968Michael S. Brown &Joseph L. Goldstein, 1968Phil Leder, 1962Edward M. Scolnick, 1967Vincent T. DeVita, Jr., 1963Data sources¾ NIH ATP index cards¾ NIH Compound Grant Applicant File¾ NIH telephone directories¾ AAMC Faculty Roster¾ AMA Physician Master File¾ USPTO patent data¾ PubMed/WoS¾ Google, doximity, etc.Descriptive Statistics:Pre-Application DataDescriptive Statistics: Career ChoiceDescriptive Statistics: Research OutcomesCareer Publications0.000.100.200.300.400.500.60Fraction of ATP Fellows0 100 200 300 400 500 Career Nb. of PublicationsNon-AttendeesAttendees Note: 61 outliers with more than 500 career publications omitted.Career NIH Funding0.000.100.200.30Fraction of ATP Fellows0 5,000,000 10,000,000 15,000,000 20,000,000 25,000,000 Career NIH Grant Funding ($ 2015)Non-AttendeesAttendees Note: 1071 fellows receive at least some NIH funding during their career. 270 outliers with more than $25 mln. in career funding omitted.Career Citations0.000.100.200.300.400.500.600.700.80Fraction of ATP Fellows0 10,000 20,000 30,000 40,000 50,000 Career Nb. of CitationsNon-AttendeesAttendees Note: 32 outliers with more than 50,000 career citations omitted.Applicants’ Medical SchoolsDistribution of First & Last PositionsExtraordinary achievements concentrated in the group of treated scientistsATP selection process in theoryIn practice, selection was often ad hoc…“During my long interview day, I met with several well-known laboratory chiefs, most of whom were not especially encouraging. But one sympathetic senior scientist, the endocrinologist Jack Robbins, saw that my limited experience would probably keep me from being selected, and he suggested that I speak with Ira Pastan, a young NIH investigator who had recently established his own laboratory to study the production of hormones by the thyroid gland.This recommendation proved to be wise and fateful. My schooling in literature turned out to be more important than my interest in endocrinology, Ira's field, because Ira's wife Linda, a poet, had often complained that Ira's colleagues seldom talked about books. Ira, himself an enthusiastic reader, thought it might be helpful to have someone with my background in his lab.When the matches were announced, I was told I would become Ira's first clinical associate, having been passed over by the more famous senior investigators I had ranked higher on my list. This outcome could not have been more fortunate.”Research Design (or lack thereof…)¾ Poor man’s identification strategy– No IV (draft lottery not binding on this population)– No RDD either¾ Selection on observables– Recall these are second-round applicants, and a lot of weeding out has already taken place– They are selected on the basis of a relatively short (30 minutes) interview, and psychologists have documented that the process is dominated by noise (e.g., Dana et al. 2013)– The observables we do have (med school, internship hospital, prior research record) do predict selection, but not strongly• Incorporating selection under ignorability does not shrink the naïve cross-sectional estimates by muchEconometric Modeling¾ Step one: estimate a propensity score¾ Step two: create inverse probability of treatment weights. For the case of the Average Treatment Effect (ATE):¾ Estimate outcome equation by weighted least squares (or weighted logit, or weighted Poisson…) where the weights are equal to each observation’s IPT.Modeling selection into the ATPsResearch Outcomes [IPTW Poisson estimates]Career Choice [IPTW Poisson estimates]Publ. Quality [IPTW Poisson estimates]Concluding thoughts/questions¾ Are we just “shooting fish in a barrel”? Probably not…– Institutional details surrounding the selection process– Refined outcomes (e.g., share of translational research publications)¾ So is the only thing we need another war?– Some current experiments in training aim to reproduce the hothouse environment (HHMI’s Janelia Farm Campus, e.g., Rubin 2006)¾ But reasons to be pessimistic– The effects might have been large and long-lasting precisely because the exposure received was intense• How much dilution is allowable before results start to fade?– Not just about shifting aspiration levels; actual skill building is needed to become a frontier innovator– Potentially high returns to designing and testing exposure interventions 

https://repository.upenn.edu/cgi/viewcontent.cgi?article=1016&amp;context=admindata_conferences

C1: Uses of Administrative Data for Supporting Education Policy

Abstract

Similar works

Full text

Available Versions

Kosmopolis

ScholarlyCommons@Penn