Simple principies for the evaluation of complex programmes

Uma das mais dramáticas mudanças nos últimos anos tem sido o desenvolvimento de programas complexos, com múltiplos objectivos, gerados por uma grande multiplicidade de entidades e com uma grande diversidade de objectivos. A razão para esta mudança é clara. A génese dos problemas sociais está entrelaçada. Os decisores interrogam-se sobre os impactes de medidas unívocas, intervenções focalizadas que estão apenas a tratar os sintomas na melhor das hipóteses, tendo ganhos de curto prazo, sem conseguir alcançar a génese mais profunda das questões. Esta complexidade inspirou "super-intervenções" de iniciativa local (area-based initiatives -ABis). O objectivo do autor é discutir do ponto de vista do investigador, a melhor forma de avaliar estes programas. Propõe-se alguns princípios base para essa avaliação que contribuem para estabelecer prioridades no interior de uma grande diversidade de opções problemáticas e metodológicas. Esses princípios decorrem de uma avaliação enquadrada teoricamente cujo objectivo é explicitar os pressupostos subjacentes às intervenções que se apelida de "teoria do programa". É com base nessa teoria que o programa pode ser avaliadoFundação para a Ciência e a TecnologiaMinistério da CulturaInstituto Português do Livro e das Biblioteca

Protocol - realist and meta-narrative evidence synthesis: Evolving Standards (RAMESES)

<p>Abstract</p> <p>Background</p> <p>There is growing interest in theory-driven, qualitative and mixed-method approaches to systematic review as an alternative to (or to extend and supplement) conventional Cochrane-style reviews. These approaches offer the potential to expand the knowledge base in policy-relevant areas - for example by explaining the success, failure or mixed fortunes of complex interventions. However, the quality of such reviews can be difficult to assess. This study aims to produce methodological guidance, publication standards and training resources for those seeking to use the realist and/or meta-narrative approach to systematic review.</p> <p>Methods/design</p> <p>We will: [a] collate and summarise existing literature on the principles of good practice in realist and meta-narrative systematic review; [b] consider the extent to which these principles have been followed by published and in-progress reviews, thereby identifying how rigour may be lost and how existing methods could be improved; [c] using an online Delphi method with an interdisciplinary panel of experts from academia and policy, produce a draft set of methodological steps and publication standards; [d] produce training materials with learning outcomes linked to these steps; [e] pilot these standards and training materials prospectively on real reviews-in-progress, capturing methodological and other challenges as they arise; [f] synthesise expert input, evidence review and real-time problem analysis into more definitive guidance and standards; [g] disseminate outputs to audiences in academia and policy. The outputs of the study will be threefold:</p> <p>1. Quality standards and methodological guidance for realist and meta-narrative reviews for use by researchers, research sponsors, students and supervisors</p> <p>2. A 'RAMESES' (Realist and Meta-review Evidence Synthesis: Evolving Standards) statement (comparable to CONSORT or PRISMA) of publication standards for such reviews, published in an open-access academic journal.</p> <p>3. A training module for researchers, including learning outcomes, outline course materials and assessment criteria.</p> <p>Discussion</p> <p>Realist and meta-narrative review are relatively new approaches to systematic review whose overall place in the secondary research toolkit is not yet fully established. As with all secondary research methods, guidance on quality assurance and uniform reporting is an important step towards improving quality and consistency of studies.</p

RAMESES publication standards: realist syntheses

PMCID: PMC3558331This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited

Carbon Monitoring System Flux Estimation and Attribution: Impact of ACOS-GOSAT X(CO2) Sampling on the Inference of Terrestrial Biospheric Sources and Sinks

Using an Observing System Simulation Experiment (OSSE), we investigate the impact of JAXA Greenhouse gases Observing SATellite 'IBUKI' (GOSAT) sampling on the estimation of terrestrial biospheric flux with the NASA Carbon Monitoring System Flux (CMS-Flux) estimation and attribution strategy. The simulated observations in the OSSE use the actual column carbon dioxide (X(CO2)) b2.9 retrieval sensitivity and quality control for the year 2010 processed through the Atmospheric CO2 Observations from Space algorithm. CMS-Flux is a variational inversion system that uses the GEOS-Chem forward and adjoint model forced by a suite of observationally constrained fluxes from ocean, land and anthropogenic models. We investigate the impact of GOSAT sampling on flux estimation in two aspects: 1) random error uncertainty reduction and 2) the global and regional bias in posterior flux resulted from the spatiotemporally biased GOSAT sampling. Based on Monte Carlo calculations, we find that global average flux uncertainty reduction ranges from 25% in September to 60% in July. When aggregated to the 11 land regions designated by the phase 3 of the Atmospheric Tracer Transport Model Intercomparison Project, the annual mean uncertainty reduction ranges from 10% over North American boreal to 38% over South American temperate, which is driven by observational coverage and the magnitude of prior flux uncertainty. The uncertainty reduction over the South American tropical region is 30%, even with sparse observation coverage. We show that this reduction results from the large prior flux uncertainty and the impact of non-local observations. Given the assumed prior error statistics, the degree of freedom for signal is approx.1132 for 1-yr of the 74 055 GOSAT X(CO2) observations, which indicates that GOSAT provides approx.1132 independent pieces of information about surface fluxes. We quantify the impact of GOSAT's spatiotemporally sampling on the posterior flux, and find that a 0.7 gigatons of carbon bias in the global annual posterior flux resulted from the seasonally and diurnally biased sampling when using a diagonal prior flux error covariance

Do reviews of healthcare interventions teach us how to improve healthcare systems?

Planners, managers and policy makers in modern health services are not without ingenuity e they will always try, try and try again. They face deep-seated or ‘wicked’ problems, which have complex roots in the labyrinthine structures though which healthcare is delivered. Accordingly, the interventions devised to deal with such stubborn problems usually come in the plural. Many different reforms are devised to deal with a particular stumbling block, which may be implemented sequentially, simultaneously or whenever policy fashion or funding dictates. This paper examines this predicament from the perspective of evidence based policy. How might researchers go about reviewing the evidence when they are faced with multiple or indeed competing interventions addressing the same problem? In the face of this plight a rather unheralded form of research synthesis has emerged, namely the ‘typological review’. We critically review the fortunes of this strategy. Separating the putative reforms into series of subtypes and producing a scorecard of their outcomes has the unintended effect of divorcing them all from an understanding of how organisations change. A more fruitful approach may lie in a ‘theory-driven review’ underpinned by an understanding of dynamics of social change in complex organisations. We test this thesis by examining the primary and secondary research on the many interventions designed to tackle a particularly wicked problem, namely the inexorable rise in demand for healthcare

Internet-based medical education: a realist review of what works, for whom and in what circumstances

http://creativecommons.org/licenses/by/2.0

Jet characterization in the upper troposphere/lower stratosphere (UTLS): applications to climatology and transport studies

A method of classifying the upper tropospheric/lower stratospheric (UTLS) jets has been developed that allows satellite and aircraft trace gas data and meteorological fields to be efficiently mapped in a jet coordinate view. A detailed characterization of multiple tropopauses accompanies the jet characterization. Jet climatologies show the well-known high altitude subtropical and lower altitude polar jets in the upper troposphere, as well as a pattern of concentric polar and subtropical jets in the Southern Hemisphere, and shifts of the primary jet to high latitudes associated with blocking ridges in Northern Hemisphere winter. The jet-coordinate view segregates air masses differently than the commonly-used equivalent latitude (EqL) coordinate throughout the lowermost stratosphere and in the upper troposphere. Mapping O3 data from the Aura Microwave Limb Sounder (MLS) satellite and the Winter Storms aircraft datasets in jet coordinates thus emphasizes different aspects of the circulation compared to an EqL-coordinate framework: the jet coordinate reorders the data geometrically, thus highlighting the strong PV, tropopause height and trace gas gradients across the subtropical jet, whereas EqL is a dynamical coordinate that may blur these spatial relationships but provides information on irreversible transport. The jet coordinate view identifies the concentration of stratospheric ozone well below the tropopause in the region poleward of and below the jet core, as well as other transport features associated with the upper tropospheric jets. Using the jet information in EqL coordinates allows us to study trace gas distributions in regions of weak versus strong jets, and demonstrates weaker transport barriers in regions with less jet influence. MLS and Atmospheric Chemistry Experiment-Fourier Transform Spectrometer trace gas fields for spring 2008 in jet coordinates show very strong, closely correlated, PV, tropopause height and trace gas gradients across the jet, and evidence of intrusions of stratospheric air below the tropopause below and poleward of the subtropical jet; these features are consistent between instruments and among multiple trace gases. Our characterization of the jets is facilitating studies that will improve our understanding of upper tropospheric trace gas evolution

Can We Systematically Review Studies That Evaluate Complex Interventions?

In three Viewpoints, Sasha Shepperd and colleagues, Geoff Wong, and Aziz Sheikh explore various approaches to help systematic reviewers who wish to review complex health interventions