Comments on Thomas C. Chalmers's Address: Evaluating Clinical Trials

Peer Reviewedhttp://deepblue.lib.umich.edu/bitstream/2027.42/66701/2/10.1177_109821408400500111.pd

Assessing Factors Influencing Acceptance of No-Difference Research

In this randomized study 160 members of the Evaluation Research Society acted as judges to assess the attributes of research that produce credibility. The study focused on acceptability of no-difference findings, a long ignored but important domain of research. In the context of a hypothetical study, four factors were tested to determine their influence on acceptability of both no-difference and difference findings: randomization/nonran domization, one/three outcomes, power = .80/.60, and equivalence on baseline measures of all eight/all but two of eight. Experts were asked to judge degree of acceptability and to reject or accept findings in both a no-difference and a difference study. Randomization consistently enhanced the believability of outcomes whereas other factors exerted a less consistent influence. Limitations of the study were discussed.Peer Reviewedhttp://deepblue.lib.umich.edu/bitstream/2027.42/66931/2/10.1177_0193841X8701100108.pd

A critique of the effectiveness of applied behavior analysis research

Since its inception, applied behavior analysis has required that solutions to socially significant problems be effective, though criteria for this dimension have remained largely implicit. This paper reviews three explicit technique for determining the effectiveness of behavioral research: graphical, social validational, and cost analyses. The concept of effect size is introduced as an additional means of comparing the effectiveness of various treatment alternatives. Survey data are utilized to support a bothersome implication of this review, namely that the contingency to produce large effects placed on behavioral researchers may actually decrease the likelihood that a useful technology of application will be produced. Finally, strategies are offered for preserving the effectiveness of behavioral procedures when existing technologies are disseminated to settings of relevance.Peer Reviewedhttp://deepblue.lib.umich.edu/bitstream/2027.42/24090/1/0000346.pd

Evaluating Understanding of Popular Press Reports of Health Research

This research assessed the ability of a sample of persons on a college campus to understand media reports of health research. Three or four articles on each of five contemporary health topics (dietary cholesterol and heart disease, treatment for breast cancer, starch blockers, drug treatment for heart disease, test tube skin) were selected from widely circulated newspapers (e.g., New York Times ) and magazines (e.g., News- week). A sample of 144 college students responded to content-based and application- based questions derived from photocopies of these popular press articles. The overall rate of reader misunderstanding approached 40% and generally fell beween one third and one half for each of 16 articles representing five health topics. Several strengths and weaknesses of the research are considered as they relate to the accuracy of estimated error rates and to the generality of study findings. The implications of these findings for other areas of health (e.g., AIDS risk factor research) are also discussed.Peer Reviewedhttp://deepblue.lib.umich.edu/bitstream/2027.42/67147/2/10.1177_109019819001700208.pd

Differential Attrition

The differential attrition of persons from comparison groups severely restricts the inferences that can be made from results of evaluative research. This problem is particularly troublesome in the evaluation of medical technologies, such as coronary artery bypass graft surgery, since a substantial percentage of medical or control patients cross over to the surgical group. A procedure using worst case assumptions is developed that allows researchers to estimate the maximum effect of differential attrition, and therefore enhance the quality of their inferences. The article first illustrates theprocedure, then concludes with a discussion of the generality of the estimation procedure to other instances in which differential attrition is a problem, and points out the limitations of the approach.Peer Reviewedhttp://deepblue.lib.umich.edu/bitstream/2027.42/67927/2/10.1177_0193841X8300700607.pd

Cumulating quality of life results in controlled trials of coronary artery bypass graft surgery

Many studies evaluating the effectiveness of coronary artery bypass graft surgery allude to the quality of life benefit resulting from surgery. However, no comprehensive empirical estimate of the absolute or relative magnitude of this benefit is currently available. This paper presents a data synthesis of the research literature on bypass surgery to derive such an estimate. It uses follow-up measures of the percent of patients who were angina-free within both the surgical and medical groups of 14 controlled trials to estimate the quality of life benefit following surgery. Results based on the longest reported follow-up period suggest that the chances are approximately 25 to 40% greater than patients will be angina-free if they receive surgery rather than medical treatment. Estimates of benefit are about 15% less in randomized controlled trials compared to controlled trials that used a matching strategy. These results are unlikely to be affected by related factors such as the percentage of patients who crossover from the medical group to the surgical group or the specific method of calculating anginal relief used in this research report. However, differential patient selection may account for the observed design effect.Peer Reviewedhttp://deepblue.lib.umich.edu/bitstream/2027.42/25475/1/0000015.pd

Research synthesis reviews: an illustrated critique of "hidden" judgments, choices, and compromises

This paper takes a pragmatic view of the steps involved in conducting a quantitative literature review. It emphasizes the multitude of judgments, choices, and compromises commonly encountered. Examination of a meta-analysis study of implosion therapy outcome research is provided as a structured means of illustrating and constructively considering questions, decisions, and issues likely to arise at each stage of the review process. The discussion should prove useful to prospective meta-analysis researchers and consumers. An extensive set of references that characterize contemporary meta-analysis research is also included.Peer Reviewedhttp://deepblue.lib.umich.edu/bitstream/2027.42/26981/1/0000548.pd

On the Reliability of Meta-Analytic Reviews

The article addresses the issue of intercoder reliability in meta-analyses. The current practice of reporting a single, mean intercoder agreement score in meta-analytic research leads to systematic bias and overestimates the true reliability. An alternative approach is recommended in which average intercoder agreement scores or other reliability statistics are calculated within clusters of coded variables. These clusters form a hierarchy in which the correctness of coding decisions at a given level of the hierarchy is contingent on decisions made at higher levels. Two separate studies of intercoder agreement in meta-analysis are presented to assess the validity of the model.Peer Reviewedhttp://deepblue.lib.umich.edu/bitstream/2027.42/67840/2/10.1177_0193841X9301700303.pd

Lost and Found, Letters and Methods: Assessing Attitudes toward Chiropractic and Medical Care

<p>Attitudes toward traditional and chiropractic medicine were compared using Milgram's lost letter technique. A total of 192 letters were placed on the windshields of vehicles in parking lots at six restaurants and department stores in each of four quadrants of a medium-sized, Southeastern city. These letters were addressed to "Admissions" at either a fictitious Institute of Medicine or Institute of Chiropractic Care. Return addresses included either a male or a female name. Thus, those who found a lost letter were faced with the option of returning or not returning a letter from either a male or a female, addressed to an Institute of traditional or non-traditional medicine. After examining previous studies which had used the lost letter technique, numerous methodological improvements were implemented. For example, letters were randomly assigned to potential drop spots for each of 24 study locations (six study locations in each of four city quadrants), and a Latin square design was used to control for possible order effects in the four study conditions that were implemented. Nearly 65% of the letters (124 of 192) were returned. We found: 1) letters addressed to a fictitious Institute of Chiropractic Care were just as likely to be returned as those addressed to a fictitious Institute of Medicine; 2) letters with female return addresses were as likely to be returned as those with male return addresses; 3) there was no interaction between study conditions; 4) based on what was essentially a replication study, a comparison of the pattern of returns using the first and second cycle of lost letters (n = 96 for each cycle) revealed an equivalent pattern of no-difference findings.</p><p> </p><p>DOI: 10.2458/azu_jmmss.v1i1.78</p

Use and Misuse of No-Difference Findings in Eliminating Threats To Validity

Donald Campbell and his colleagues have elaborated an approach to research based on the elimination of threats to four validity types: internal, external, construct, and statistical conclusion. The central thesis of this article is that the process of eliminating validity threats depends fundamentally on no-difference findings, a fact that, unfor tunately, has not been made explicit by researchers. The implications of this neglect are explored using examples from a number of different substantive areas such as psychology, health, and medicine. Finally, the intrinsic role of no-difference findings is described in the context of all four validity types, and suggestions for improving the process are offered.Peer Reviewedhttp://deepblue.lib.umich.edu/bitstream/2027.42/67577/2/10.1177_0193841X8601000608.pd