Some Evaluation Questions

Accessed 90,154 times on https://pareonline.net from November 13, 1999 to December 31, 2019. For downloads from January 1, 2020 forward, please click on the PlumX Metrics link to the right

Scatteract: Automated extraction of data from scatter plots

Charts are an excellent way to convey patterns and trends in data, but they do not facilitate further modeling of the data or close inspection of individual data points. We present a fully automated system for extracting the numerical values of data points from images of scatter plots. We use deep learning techniques to identify the key components of the chart, and optical character recognition together with robust regression to map from pixels to the coordinate system of the chart. We focus on scatter plots with linear scales, which already have several interesting challenges. Previous work has done fully automatic extraction for other types of charts, but to our knowledge this is the first approach that is fully automatic for scatter plots. Our method performs well, achieving successful data extraction on 89% of the plots in our test set.Comment: Submitted to ECML PKDD 2017 proceedings, 16 page

Causality

Making correct causal claims is important for research and practice. This article explains what causality is, and how it can be established via experimental design. Because experiments are infeasible in many applied settings, researchers often use "observational" methods to estimate causal models. In these situations, it is likely that model estimates are compromised by endogeneity. The article discusses the conditions that engender endogeneity and methods that can eliminate it

Reprint of “The Single-Case Reporting Guideline In BEhavioural interventions (SCRIBE) 2016: explanation and elaboration”

There is substantial evidence that research studies reported in the scientific literature do not provide adequate information so that readers know exactly what was done and what was found. This problem has been addressed by the development of reporting guidelines which tell authors what should be reported and how it should be described. Many reporting guidelines are now available for different types of research designs. There is no such guideline for one type of research design commonly used in the behavioral sciences, the single-case experimental design (SCED). The present study addressed this gap. This report describes the Single-Case Reporting guideline In BEhavioural interventions (SCRIBE) 2016, which is a set of 26 items that authors need to address when writing about SCED research for publication in a scientific journal. Each item is described, a rationale for its inclusion is provided, and examples of adequate reporting taken from the literature are quoted. It is recommended that the SCRIBE 2016 is used by authors preparing manuscripts describing SCED research for publication, as well as journal reviewers and editors who are evaluating such manuscripts.Published versio

The use of segmented regression in analysing interrupted time series studies : an example in pre-hospital ambulance care

Peer reviewedPublisher PD

Changing the ideological roots of prejudice: Longitudinal effects of ethnic intergroup contact on social dominance orientation

Social Dominance Orientation (SDO) has been reported to be strongly related to a multitude of intergroup phenomena, but little is known about situational experiences that may influence SDO. Drawing from research on intergroup contact theory, we argue that positive intergroup contact is able to reduce SDO-levels. The results of an intergroup contact intervention study among high school students (Study 1, N=71) demonstrated that SDO-levels were indeed attenuated after the intervention. Furthermore, this intervention effect on SDO was especially pronounced among students reporting a higher quality of contact. A cross-lagged longitudinal survey among adults (Study 2, N=363) extended these findings by demonstrating that positive intergroup contact is able to decrease SDO over time. Moreover, we did not obtain evidence for the idea that people high in SDO would engage less in intergroup contact. These findings indicate that intergroup contact erodes one of the important socio-ideological bases of generalized prejudice and discrimination

RCTs: How compatible are they with policy-making?

Randomised controlled trials (RCTs) have been promoted as a means of improving policy-making by testing new policies. While testing before full-scale roll-out is commendable, this paper discusses the challenges of using RCTs in contemporary (national) health policy-making in England. There are at least two challenges in particular that are currently underrepresented in the debate: The first arises from the complexity of many policies which are often too diffuse and unclear in focus to allow for the clear distinction between a policy ‘mechanism’ and its context to be drawn that is required for a RCT. The second challenge relates to the timing of RCTs, which tend to take place either too early in the life of a policy to be meaningful or too late to have an effect on policy formulation. We therefore encourage policy-makers and researchers to be clear about the types of uncertainties ‘field experiments’ are meant to address which may be addressed better by other types of knowledge generation

Properties of bootstrap tests for N-of-1 studies

N-of-1 study designs involve the collection and analyses of repeated measures data from an individual not using an intervention and using an intervention. This study explores the use of semi-parametric and parametric bootstrap tests in the analysis of N-of-1 studies under a single time series framework in the presence of autocorrelation. When the Type I error rates of bootstrap tests are compared to Wald tests, our results show the bootstrap tests have more desirable properties. We compare the results for normally distributed errors with those for contaminated normally distributed errors and find that, except for when there is relatively large autocorrelation, there is little difference between the power of the parametric and semi-parametric bootstrap tests. We also experiment with two intervention designs: ABAB and AB, and show the ABAB design has more power. The results provide guidelines for designing N-of-1 studies, in the sense of how many observations and how many intervention changes are needed to achieve a certain level of power and which test should be performed

Putting theory oriented evaluation into practice

Evaluations of gaming simulations and business games as teaching devices are typically end-state driven. This emphasis fails to detect how the simulation being evaluated does or does not bring about its desired consequences. This paper advances the use of a logic model approach which possesses a holistic perspective that aims at including all elements associated with the situation created by a game. The use of the logic model approach is illustrated as applied to Simgame, a board game created for secondary school level business education in six European Union countries