When You Say Nothing at All: The Predictive Power of Student Effort on Surveys

Character traits and noncognitive skills are important for human capital development and longrun life outcomes. Research in economics and psychology now shows this clearly. But research into the exact determinants of noncognitive skills have been slowed by a common data limitation: most large-scale datasets do not contain adequate measures of noncognitive skills. This is a particularly acute problem in education policy evaluation. We demonstrate that there are important latent data within any survey dataset that can be used as proxy measures of noncognitive skills. Specifically, we examine the amount of conscientious effort that students exhibit on surveys, as measured by their item response rates. We use six nationally representative, longitudinal surveys of American youth. We find that the percentage of questions left unanswered during the baseline year, when respondents were adolescents, is a significant predictor of later-life outcomes. Respondents with higher item response rates are more likely to attain higher levels of education. The pattern of findings gives compelling reasons to view item response rates as a promising behavioral measure of noncognitive skills for use in future research in education. We posit that response rates are a partial measure of conscientiousness, though additional research from the field of psychology is required to determine what exact noncognitive skills are being captured by item response rate

When Students Don\u27t Care: Reexamining International Differences in Achievement and Non-Cognitive Skills

Policy debates in education are often framed by using international test scores, such as the Programme for International Student Assessment (PISA). The obvious presumption is that observed differences in test scores within and across countries reflect differences in cognitive skills and general content knowledge, the things which achievement tests are designed to measure. We challenge this presumption, by studying how much of the within-country and between-country variation in PISA test scores is associated with student effort, rather than true academic content knowledge. Drawing heavily on recent literature, we posit that our measures of student effort are actually proxy measures of relevant non-cognitive skills related to conscientiousness. Completing surveys and tests takes effort and students may actually reveal something about their conscientiousness by the amount of effort they show during these tasks. Our previous work, and that of others validates this claim (e.g. Boe, May and Boruch, 2002; Borghans and Schils, 2012; Hitt, Trivitt and Cheng, 2016; Hitt, 2016; Zamarro et al., 2016). Using parametrizations of measures of survey and test effort we find that these measures help explain between 32 and 38 percent of the observed variation in test scores across countries, while explaining only a minor share of the observed variation within countries

Character Assessment: Three Essays

I propose a new approach to measuring character skills. In the following three essays, my co-‎authors and I measure the effort that adolescent students appear to put forward on surveys ‎and tests. First, I examine the extent to which students simply skip questions or plead ‎ignorance on surveys. Second, I develop new methods for detecting careless answers, those ‎instances in which students appear to be just filling in the bubbles. I show, using ‎longitudinal datasets, that both measures are predictive of educational degree attainment, ‎independent of measured cognitive ability and other demographic factors. Finally, I ‎demonstrate that international differences in reading, math and science test scores appear in ‎fact to partially reflect international differences in student effort on assessments. Just as some ‎students skip questions and carelessly answer surveys, some students do the same on tests. To ‎the extent that effort on surveys and tests reflects noncognitive skills, presumed international ‎differences in cognitive ability (as measured by standardized tests) might in fact be driven by ‎differences in noncognitive ability. Altogether, the measures explored in the paper present ‎three new methods for quantifying student character skills, which can be used in future ‎research. Throughout, my co-authors and I posit that the character skills that our measures ‎capture are related to conscientiousness and self-control.

Comparisons of Student Perceptions of Teacher\u27s Performance in the Classroom: Using Parametric Anchoring Vignette Methods for Improving Comparability

Self-reports are an indispensable source of information in education research but might be affected by reference group bias if the frame of reference (i.e. implicit standards), used to answer the questions, differs across students. The anchoring vignettes method was introduced, in other areas of social science, precisely to correct for this source of bias. However, studies that make use of this approach in education are rare and more research is needed to study its potential. This paper uses data from PISA 2012 to investigate the use of the parametric model of the anchoring vignettes method to correct for differential implicit standards in cross-country comparisons of student’s perceptions of an important dimension of teacher quality: teacher’s classroom management. Our results show significant heterogeneity in implicit standards across countries. We also show how correlations between countries’ average teacher classroom management levels and external variables can be improved substantially when heterogeneity in implicit standards is adjusted for. We conclude that the anchoring vignettes method shows a good potential to enhance the validity and comparability of self-reported measures in education

Comparing and Validating Measures of Character Skills: Findings from a Nationally Representative Sample

Though researchers now are aware of the potential importance of character skills, such as conscientiousness, grit, self-control, and a growth mindset, researchers struggle to find reliable measures of these skills. In this paper, we use data collected from the Understanding America Study, a nationally representative internet panel to study the validity of innovative measures of character skills based on measures of survey effort. We believe surveys themselves can be seen as a behavioral tasks and that respondents provide meaningful information about their character skills by way of the effort they put forward on surveys. In particular, we compare measures of grit, conscientiousness and other personality traits, and growth mindset, based on self-reports, and survey effort measures of character. We study the relationship across each other and their relationship with academic and life outcomes such as income and labor-market outcomes, after controlling for cognitive ability and other relevant demographic characteristics. Our results show that survey effort measures of character skills, in particular measures of careless answering in surveys, show great promise for being good proxy measures of relevant non-cognitive skills

Non-Cognitive Abilities and Spanish Regional Differences in Student Performance in PISA 2009

The goal of this paper is to analyze the role that non-cognitive skills and, in particular, regional differences in those skills, play on the observed differences in 15-year-old student’s academic performance, across Spanish regions, on PISA 2009. Previous research has shown the relevance of differences in student’s personal, family and school characteristics in accounting for academic differences across Spanish regions but it has also found that a sizeable part of the observed differences remained unexplained. We have found that differences in the distribution of certain non-cognitive skills associated to academic performance like focus, perseverance and resilience play a prominent role in accounting for differences in student performance in PISA 2009. We observe these skills by developing new measures of student effort on standardized tests. In particular, our estimates suggest that a standard deviation reduction in the dispersion of non-cognitive skills across Spanish regions would lead to a 25% reduction in the magnitude of the observed differences in student performance across regions. This is a relevant effect as, for example, a one standard deviation reduction in the regional dispersion of parent’s educational levels or occupational status would only lead to at most a 2% reduction in the magnitude of observed differences in performance on PISA across Spanish regions. Put plainly, a substantial portion of the regional variation in test scores appears attributable to effort on the PISA test, and not necessarily just differences in actual knowledge

Does the Timing of Money Matter? A Case Study of the Arkansas Academic Challenge Scholarship

This paper examines the effect of a state-financed merit-aid scholarship—the Arkansas Academic Challenge Scholarship (ACS)—on post-secondary outcomes at a large university in Arkansas. Exploiting scholarship eligibility requirements, we implement a fuzzy regression discontinuity design to identify the scholarship’s causal impacts on college outcomes. The analysis focuses on currently enrolled sophomores, juniors, and seniors who receive the scholarship to investigate the broad impacts of receiving money at nontraditional points in an individual’s college trajectory. Findings indicate small, negative impacts of scholarship receipt on short-run outcomes such as GPA and credit accumulation, but large statistically significant declines in the likelihood of graduating within four, five, or six years of matriculation. The youngest cohort, who began receiving funding during their sophomore year of enrollment, primarily drives these findings. However, cohort analysis also reveals that seniors who do not graduate on time are 54 percentage points more likely to graduate within 6 years of matriculation when they receive the scholarship. These results highlight the fact that the timing of receiving money may heavily influence student behavior and postsecondary outcomes

Measures of Student Non-cognitive Skills and Political Tolerance after Two Years of the Louisiana Scholarship Program

This report examines the short-term effects of the Louisiana Scholarship Program (LSP) on students’ non-cognitive skills and civic values. While a growing number of studies have evaluated K-12 school voucher programs along academic dimensions, few have focused on the development of non-cognitive skills and civic values. This study aims to address that gap by providing the first analysis of differences in self-reported measures of grit, locus of control, self esteem, and political tolerance associated with the LSP. Using results from a phone survey of applicants to the program, we find little evidence of differences between LSP scholarship recipients and non-recipients. Nevertheless, diagnostics assessing the precision of our instruments to detect differences between subjects indicate that several of the scales measuring non-cognitive skills performed poorly in our sample. Moreover, our relatively low survey response rate of 11 percent raises concerns about the representativeness of our sample. Given these issues, we caution that our results are best understood as descriptive and not necessarily conclusive: they do not represent reliable estimates of the causal impact of the LSP on student non-cognitive skills and political tolerance

No Excuses Charter Schools: A Meta-Analysis of the Experimental Evidence on Student Achievement

While charter schools differ widely in philosophy and pedagogical views, the United States’s most famous urban charter schools typically use the No Excuses approach. Enrolling mainly poor and minority students, these schools feature high academic standards, strict disciplinary codes, extended instructional time, and targeted supports for low-performing students. The strenuous and regimented style is controversial amongst some scholars, but others contend that the No Excuses approach is needed to rapidly close the achievement gap. We conduct the first meta-analysis of the achievement impacts of No Excuses charter schools. Focusing on experimental studies, we find that No Excuses charter schools significantly improve math scores and reading scores. We estimate gains of 0.25 and 0.16 standard deviations on math and literacy achievement, respectively, as the effect of attending a No Excuses charter school for one year. Though the effect is large and meaningful, we offer some caveats to this finding and discuss policy implications for the United States as well as other countries

Mapping reef fish and the seascape: using acoustics and spatial modeling to guide coastal management

Reef fish distributions are patchy in time and space with some coral reef habitats supporting higher densities (i.e., aggregations) of fish than others. Identifying and quantifying fish aggregations (particularly during spawning events) are often top priorities for coastal managers. However, the rapid mapping of these aggregations using conventional survey methods (e.g., non-technical SCUBA diving and remotely operated cameras) are limited by depth, visibility and time. Acoustic sensors (i.e., splitbeam and multibeam echosounders) are not constrained by these same limitations, and were used to concurrently map and quantify the location, density and size of reef fish along with seafloor structure in two, separate locations in the U.S. Virgin Islands. Reef fish aggregations were documented along the shelf edge, an ecologically important ecotone in the region. Fish were grouped into three classes according to body size, and relationships with the benthic seascape were modeled in one area using Boosted Regression Trees. These models were validated in a second area to test their predictive performance in locations where fish have not been mapped. Models predicting the density of large fish (≥29 cm) performed well (i.e., AUC = 0.77). Water depth and standard deviation of depth were the most influential predictors at two spatial scales (100 and 300 m). Models of small (≤11 cm) and medium (12–28 cm) fish performed poorly (i.e., AUC = 0.49 to 0.68) due to the high prevalence (45–79%) of smaller fish in both locations, and the unequal prevalence of smaller fish in the training and validation areas. Integrating acoustic sensors with spatial modeling offers a new and reliable approach to rapidly identify fish aggregations and to predict the density large fish in un-surveyed locations. This integrative approach will help coastal managers to prioritize sites, and focus their limited resources on areas that may be of higher conservation value