The Role of Theory in Quantitative Data Analysis

If you take theory and models seriously, then (a) you need to elaborate clearly for yourself ‘what counts’ and how things supposedly fit together, and (b) you must hold yourself accountable to data.… From my perspective, theory is – or should be – the lifeblood of the empirical scientist. (Schoenfeld, 2010, p. 105) The process of theorizing, collecting evidence, testing theory, revising theory, and then working through the cycle again is the basis of science. Theory is no less important when conducting a data analysis using quantitative methods. All statistical textbooks spend significant space on discussing the assumptions of statistical tests, the data requirements for a given estimation procedure, and the boundaries around the conclusions that can be drawn from results. Theory about the phenomenon that one is examining through quantitative data analysis is not only the driver for the methods used to collect evidence, but most importantly, the decisions made about how to model and test that theory

The Use of Meta-Analytic Statistical Significance Testing

Meta-analysis multiplicity, the concept of conducting multiple tests of statistical significance within one study, is an underdeveloped literature (Tendal, Nüesch, Higgins, Jüni, & Gøtzsche, 2011). We address this issue by considering how Type I errors can impact meta-analytic results, suggest how statistical power may be affected through the use of multiplicity corrections, and propose how meta-analysts should analyze multiple tests of statistical significance. The context for this study is a meta-review of meta-analyses published in two leading review journals in education and psychology. Our review of 130 meta-analyses revealed a strong reliance on statistical significance testing without considering of Type I errors or the use of multiplicity corrections. In order to provide valid conclusions, meta-analysts must consider these issues prior to conducting the study

Validation of the Employment Hope Scale: Measuring Psychological Self-Sufficiency Among Low-Income Jobseekers

The Employment Hope scale (EHS) was designed to measure the empowerment-based self-sufficiency (SS) outcome among low-income job-seeking clients. This measure captures the psychological SS dimension as opposed to the more commonly used economic SS in workforce development and employment support practice. The study validates the EHS and reports its psychometric properties. Method: An exploratory factor analysis (EFA) was conducted using an agency data from the Cara Program in Chicago, United States. The principal axis factor extraction process was employed to identify the factor structure. Results: EFA resulted in a 13-item two-factor structure with Factor 1 representing “Psychological Empowerment” and Factor 2 representing “Goal-Oriented Pathways.” Both factors had high internal consistency reliability and construct validity. Conclusions: While findings may be preliminary, this study found the EHS to be a reliable and valid measure, demonstrating its utility in assessing psychological SS as an empowerment outcome among low-income jobseekers

Quality of Research Evidence in Education: How Do We Know?

The persistence of inequitable education is the fundamental fact facing education researchers as we reflect on the quality and value of the evidence we produce (American Educational Research Association & National Academy of Education, 2020; Educational Opportunity Monitoring Project, 2020). As a field, we must critically examine what it means for us to develop increasingly sophisticated research tools and research design models while disparate outcomes along familiar lines of race and class continue apace. This issue’s importance has been laid bare by the COVID-19 pandemic and the global protests for racial justice in the wake of George Floyd’s murder. If our research endeavors are not effectively combating racism in education, providing help as our schools refashion themselves for remote and hybrid teaching, or supporting schools in other ways to address the myriad of equity gaps they face, then what are we doing? What are we generating evidence of and for

The Question of School Resources and Student Achievement: A History and Reconsideration

One question posed continually over the past century of education research is to what extent school resources affect student outcomes. From the turn of the century to the present, a diverse set of actors, including politicians, physicians, and researchers from a number of disciplines, have studied whether and how money that is provided for schools translates into increased student achievement. The authors discuss the historical origins of the question of whether school resources relate to student achievement, and report the results of a meta- analysis of studies examining that relationship. They find that policymakers, researchers, and other stakeholders have addressed this question using diverse strategies. The way the question is asked, and the methods used to answer it, is shaped by history, as well by the scholarly, social, and political concerns of any given time. The diversity of methods has resulted in a body of literature too diverse and too inconsistent to yield reliable inferences through meta-analysis. The authors suggest that a collaborative approach addressing the question from a variety of disciplinary and practice perspectives may lead to more effective interventions to meet the needs of all students

AHRQ series on complex intervention systematic reviews-paper 5: advanced analytic methods.

BACKGROUND AND OBJECTIVE: Advanced analytic methods for synthesizing evidence about complex interventions continue to be developed. In this paper, we emphasize that the specific research question posed in the review should be used as a guide for choosing the appropriate analytic method. METHODS: We present advanced analytic approaches that address four common questions that guide reviews of complex interventions: (1) How effective is the intervention? (2) For whom does the intervention work and in what contexts? (3) What happens when the intervention is implemented? and (4) What decisions are possible given the results of the synthesis? CONCLUSION: The analytic approaches presented in this paper are particularly useful when each primary study differs in components, mechanisms of action, context, implementation, timing, and many other domains

Outcome-Reporting Bias in Education Research

Outcome reporting bias occurs when primary studies do not include information about all outcomes measured in a study. When studies omit findings on important measures, efforts to synthesize the research using systematic review techniques will be biased and interpretations of individual studies will be incomplete. Outcome reporting bias has been well-documented in medicine, and has been shown to lead to inaccurate assessments of the effects of medical treatments and, in some cases, to omission of reports of harms. This study examines outcome reporting bias in educational research by comparing the reports of educational interventions from dissertations to their published versions. We find that non-significant outcomes were 30% more likely to be omitted from a published study than statistically significant ones

Treatment for School Refusal Among Children and Adolescents: A Systematic Review and Meta-Analysis

Objective: School refusal is a psychosocial problem associated with adverse short- and long-term consequences for children and adolescents. The authors conducted a systematic review and meta-analysis to examine the effects of psychosocial treatments for children and adolescents with school refusal. Method: A comprehensive search process was used to find eligible randomized controlled trials and quasi-experimental studies assessing the effects of psychosocial treatments on anxiety or attendance outcomes. Data were quantitatively synthesized using meta-analytic methods. Results: Eight studies including 435 children and adolescents with school refusal were included in this review. Significant effects were found for attendance but not for anxiety. Conclusions: Evidence indicates that improvements in school attendance occur for children and adolescents with school refusal who receive psychosocial treatment. The lack of evidence of short-term effects on anxiety points to the need for long-term follow-up studies to determine whether increased attendance ultimately leads to reduced anxiety