Re-examining the implications of systems thinking for evaluation

Over the last twenty years, many prominent evaluators have been borrowing and using ideas, theories, and methods from the systems and complexity fields with little research on the implications of this trend for the evaluation field. This thesis examines this borrowing to identify over-arching implications for evaluation theory and practice. The first paper reviews inter-disciplinary literature on systems thinking and complexity science with regards to evaluating social policies and programs and identifies major implications for how evaluators theorize evaluation practice. The second paper reports on an analysis of eight cases of evaluation practice that use systems and complexity ideas and techniques and presents findings regarding how evaluators conceive of and practice evaluation. The third paper advances an argument for how evaluators can use critical systems heuristics to surface, reflect on, and make explicit the values that influence and should influence an evaluation. Collectively, these papers support the potential of borrowing from the systems and complexity fields to expand on and re-define evaluation theory and practice

Equity “On the Sideline”: A Mixed Methods Study of New England Evaluation Practice in 2020

Background: Centering equity in evaluations is increasingly recognized as an important professional responsibility of evaluators. While some theoretical and practical guidance exists, the evaluation field has limited empirical research on equity within evaluation practice. Purpose: This paper explores whether and how evaluators address inequities and advance equity throughout evaluation phases drawing on select findings from a larger study. Setting: The study focuses on American Evaluation Association-affiliated evaluators in the New England region of the United States who work in a variety of areas (e.g., health, education).   Intervention: Not applicable Research Design: The study uses a complementarity, sequential mixed methods design comprised of a researcher-developed online questionnaire administered to a census and snowball sample of practicing evaluators (n=82) and individual, semi-structured interviews with a subset of this sample selected to maximize variation (n=21). Quantitative data were analyzed using descriptive statistics (i.e., means and standard deviations, frequencies). Qualitative data were analyzed using a collaborative process of deductive and inductive coding followed by thematic analysis. Findings: Eight overarching findings suggest that despite evaluators’ attempts to center equity, it remains largely “on the sideline.” This is due to evaluators’ need to work against some conventional professional and methodological norms, within contractual and contextual constraints, and with limited professional preparation. &nbsp

Equity “On the Sideline”

Background: Centering equity in evaluations is increasingly recognized as an important professional responsibility of evaluators. While some theoretical and practical guidance exists, the evaluation field has limited empirical research on equity within evaluation practice. Purpose: This paper explores whether and how evaluators address inequities and advance equity throughout evaluation phases drawing on select findings from a larger study. Setting: The study focuses on American Evaluation Association-affiliated evaluators in the New England region of the United States who work in a variety of areas (e.g., health, education).   Intervention: Not applicable Research Design: The study uses a complementarity, sequential mixed methods design comprised of a researcher-developed online questionnaire administered to a census and snowball sample of practicing evaluators (n=82) and individual, semi-structured interviews with a subset of this sample selected to maximize variation (n=21). Quantitative data were analyzed using descriptive statistics (i.e., means and standard deviations, frequencies). Qualitative data were analyzed using a collaborative process of deductive and inductive coding followed by thematic analysis. Findings: Eight overarching findings suggest that despite evaluators’ attempts to center equity, it remains largely “on the sideline.” This is due to evaluators’ need to work against some conventional professional and methodological norms, within contractual and contextual constraints, and with limited professional preparation.

National trends in total cholesterol obscure heterogeneous changes in HDL and non-HDL cholesterol and total-to-HDL cholesterol ratio : a pooled analysis of 458 population-based studies in Asian and Western countries

Background: Although high-density lipoprotein (HDL) and non-HDL cholesterol have opposite associations with coronary heart disease, multi-country reports of lipid trends only use total cholesterol (TC). Our aim was to compare trends in total, HDL and nonHDL cholesterol and the total-to-HDL cholesterol ratio in Asian and Western countries. Methods: We pooled 458 population-based studies with 82.1 million participants in 23 Asian and Western countries. We estimated changes in mean total, HDL and non-HDL cholesterol and mean total-to-HDL cholesterol ratio by country, sex and age group. Results: Since similar to 1980, mean TC increased in Asian countries. In Japan and South Korea, the TC rise was due to rising HDL cholesterol, which increased by up to 0.17 mmol/L per decade in Japanese women; in China, it was due to rising non-HDL cholesterol. TC declined in Western countries, except in Polish men. The decline was largest in Finland and Norway, at similar to 0.4 mmol/L per decade. The decline in TC in most Western countries was the net effect of an increase in HDL cholesterol and a decline in non-HDL cholesterol, with the HDL cholesterol increase largest in New Zealand and Switzerland. Mean total-to-HDL cholesterol ratio declined in Japan, South Korea and most Western countries, by as much as similar to 0.7 per decade in Swiss men (equivalent to similar to 26% decline in coronary heart disease risk per decade). The ratio increased in China. Conclusions: HDL cholesterol has risen and the total-to-HDL cholesterol ratio has declined in many Western countries, Japan and South Korea, with only a weak correlation with changes in TC or non-HDL cholesterol.Peer reviewe

Contributions of mean and shape of blood pressure distribution to worldwide trends and variations in raised blood pressure: A pooled analysis of 1018 population-based measurement studies with 88.6 million participants

© The Author(s) 2018. Background: Change in the prevalence of raised blood pressure could be due to both shifts in the entire distribution of blood pressure (representing the combined effects of public health interventions and secular trends) and changes in its high-blood-pressure tail (representing successful clinical interventions to control blood pressure in the hypertensive population). Our aim was to quantify the contributions of these two phenomena to the worldwide trends in the prevalence of raised blood pressure. Methods: We pooled 1018 population-based studies with blood pressure measurements on 88.6 million participants from 1985 to 2016. We first calculated mean systolic blood pressure (SBP), mean diastolic blood pressure (DBP) and prevalence of raised blood pressure by sex and 10-year age group from 20-29 years to 70-79 years in each study, taking into account complex survey design and survey sample weights, where relevant. We used a linear mixed effect model to quantify the association between (probittransformed) prevalence of raised blood pressure and age-group- and sex-specific mean blood pressure. We calculated the contributions of change in mean SBP and DBP, and of change in the prevalence-mean association, to the change in prevalence of raised blood pressure. Results: In 2005-16, at the same level of population mean SBP and DBP, men and women in South Asia and in Central Asia, the Middle East and North Africa would have the highest prevalence of raised blood pressure, and men and women in the highincome Asia Pacific and high-income Western regions would have the lowest. In most region-sex-age groups where the prevalence of raised blood pressure declined, one half or more of the decline was due to the decline in mean blood pressure. Where prevalence of raised blood pressure has increased, the change was entirely driven by increasing mean blood pressure, offset partly by the change in the prevalence-mean association. Conclusions: Change in mean blood pressure is the main driver of the worldwide change in the prevalence of raised blood pressure, but change in the high-blood-pressure tail of the distribution has also contributed to the change in prevalence, especially in older age groups

Effect of Hydrocortisone on Mortality and Organ Support in Patients With Severe COVID-19: The REMAP-CAP COVID-19 Corticosteroid Domain Randomized Clinical Trial.

Importance: Evidence regarding corticosteroid use for severe coronavirus disease 2019 (COVID-19) is limited. Objective: To determine whether hydrocortisone improves outcome for patients with severe COVID-19. Design, Setting, and Participants: An ongoing adaptive platform trial testing multiple interventions within multiple therapeutic domains, for example, antiviral agents, corticosteroids, or immunoglobulin. Between March 9 and June 17, 2020, 614 adult patients with suspected or confirmed COVID-19 were enrolled and randomized within at least 1 domain following admission to an intensive care unit (ICU) for respiratory or cardiovascular organ support at 121 sites in 8 countries. Of these, 403 were randomized to open-label interventions within the corticosteroid domain. The domain was halted after results from another trial were released. Follow-up ended August 12, 2020. Interventions: The corticosteroid domain randomized participants to a fixed 7-day course of intravenous hydrocortisone (50 mg or 100 mg every 6 hours) (n = 143), a shock-dependent course (50 mg every 6 hours when shock was clinically evident) (n = 152), or no hydrocortisone (n = 108). Main Outcomes and Measures: The primary end point was organ support-free days (days alive and free of ICU-based respiratory or cardiovascular support) within 21 days, where patients who died were assigned -1 day. The primary analysis was a bayesian cumulative logistic model that included all patients enrolled with severe COVID-19, adjusting for age, sex, site, region, time, assignment to interventions within other domains, and domain and intervention eligibility. Superiority was defined as the posterior probability of an odds ratio greater than 1 (threshold for trial conclusion of superiority >99%). Results: After excluding 19 participants who withdrew consent, there were 384 patients (mean age, 60 years; 29% female) randomized to the fixed-dose (n = 137), shock-dependent (n = 146), and no (n = 101) hydrocortisone groups; 379 (99%) completed the study and were included in the analysis. The mean age for the 3 groups ranged between 59.5 and 60.4 years; most patients were male (range, 70.6%-71.5%); mean body mass index ranged between 29.7 and 30.9; and patients receiving mechanical ventilation ranged between 50.0% and 63.5%. For the fixed-dose, shock-dependent, and no hydrocortisone groups, respectively, the median organ support-free days were 0 (IQR, -1 to 15), 0 (IQR, -1 to 13), and 0 (-1 to 11) days (composed of 30%, 26%, and 33% mortality rates and 11.5, 9.5, and 6 median organ support-free days among survivors). The median adjusted odds ratio and bayesian probability of superiority were 1.43 (95% credible interval, 0.91-2.27) and 93% for fixed-dose hydrocortisone, respectively, and were 1.22 (95% credible interval, 0.76-1.94) and 80% for shock-dependent hydrocortisone compared with no hydrocortisone. Serious adverse events were reported in 4 (3%), 5 (3%), and 1 (1%) patients in the fixed-dose, shock-dependent, and no hydrocortisone groups, respectively. Conclusions and Relevance: Among patients with severe COVID-19, treatment with a 7-day fixed-dose course of hydrocortisone or shock-dependent dosing of hydrocortisone, compared with no hydrocortisone, resulted in 93% and 80% probabilities of superiority with regard to the odds of improvement in organ support-free days within 21 days. However, the trial was stopped early and no treatment strategy met prespecified criteria for statistical superiority, precluding definitive conclusions. Trial Registration: ClinicalTrials.gov Identifier: NCT02735707

Strengthening Capacity and Equity in New England Evaluation

This report presents results from the Strengthening Capacity and Equity in New England Evaluation (SCENE) study. The SCENE study was conducted between August 2020 and August 2021 by a research group at Boston College with funding from the Barr Foundation.The purpose was three-fold: (1) to characterize evaluation providers, evaluation practices, and evaluators working in the New England area; (2) to explore whether and how evaluators address inequities and advance equity; and (3) to identify ways to strengthen capacity and equity among evaluators within the region. By providing an initial assessment to inform future capacity building and learning initiatives, we aim to inform regional evaluation practitioners and commissioners interested in advancing equity in evaluation. Findings are particularly relevant to local affiliates of the American Evaluation Association, including: Greater Boston Evaluators Network, Vermont Evaluation Network, Connecticut Area Evaluators' Network, Maine Evaluation Society. We also sought to address gaps in the evaluation literature regarding what equity and equity-focused evaluation mean and look like in practice. We plan to submit study results for publication/dissemination in academic journals

Effects of leg flags on nest survival of four species of Arctic-breeding shorebirds

Marking wild birds is an integral part of many field studies. However, if marks affect the vital rates or behavior of marked individuals, any conclusions reached by a study might be biased relative to the general population. Leg bands have rarely been found to have negative effects on birds and are frequently used to mark individuals. Leg flags, which are larger, heavier, and might produce more drag than bands, are commonly used on shorebirds and can help improve resighting rates. However, no one to date has assessed the possible effects of leg flags on the demographic performance of shorebirds. At seven sites in Arctic Alaska and western Canada, we marked individuals and monitored nest survival of four species of Arctic-breeding shorebirds, including Semipalmated Sandpipers (Calidris pusilla), Western Sandpipers (C. mauri), Red-necked Phalaropes (Phalaropus lobatus), and Red Phalaropes (P. fulicarius). We used a daily nest survival model in a Bayesian framework to test for effects of leg flags, relative to birds with only bands, on daily survival rates of 1952 nests. We found no evidence of a difference in nest survival between birds with flags and those with only bands. Our results suggest, therefore, that leg flags have little effect on the nest success of Arctic-breeding sandpipers and phalaropes. Additional studies are needed, however, to evaluate the possible effects of flags on shorebirds that use other habitats and on survival rates of adults and chicks.acceptedVersio

Development of an antigen detection assay for early point-of-care diagnosis of Zaire ebolavirus.

The 2013-2016 Ebola virus (EBOV) outbreak in West Africa and the ongoing cases in the Democratic Republic of the Congo have spurred development of a number of medical countermeasures, including rapid Ebola diagnostic tests. The likelihood of transmission increases as the disease progresses due to increasing viral load and potential for contact with others. Early diagnosis of EBOV is essential for halting spread of the disease. Polymerase chain reaction assays are the gold standard for diagnosing Ebola virus disease (EVD), however, they rely on infrastructure and trained personnel that are not available in most resource-limited settings. Rapid diagnostic tests that are capable of detecting virus with reliable sensitivity need to be made available for use in austere environments where laboratory testing is not feasible. The goal of this study was to produce candidate lateral flow immunoassay (LFI) prototypes specific to the EBOV glycoprotein and viral matrix protein, both targets known to be present during EVD. The LFI platform utilizes antibody-based technology to capture and detect targets and is well suited to the needs of EVD diagnosis as it can be performed at the point-of-care, requires no cold chain, provides results in less than twenty minutes and is low cost. Monoclonal antibodies were isolated, characterized and evaluated in the LFI platform. Top performing LFI prototypes were selected, further optimized and confirmed for sensitivity with cultured live EBOV and clinical samples from infected non-human primates. Comparison with a commercially available EBOV rapid diagnostic test that received emergency use approval demonstrates that the glycoprotein-specific LFI developed as a part of this study has improved sensitivity. The outcome of this work presents a diagnostic prototype with the potential to enable earlier diagnosis of EVD in clinical settings and provide healthcare workers with a vital tool for reducing the spread of disease during an outbreak