Objectives: Assess the degree to which research project findings were published and explore factors that influenced publication. Methods: Questionnaire to project leaders. Classification of publications and findings. Chi-squared; univariate and multivariate Cox regression analyses. Results: Forty percent of projects published in peer-reviewed journal; highly statistically significant relationships between publication in peer-reviewed journals and (1) projects in Responsive/Fellowships streams (p = .045); and (2) projects awarded >pound22,713 (p = .02); influence of study findings not statistically significant. Conclusions: Funders should consider the significant number of studies that did not result in publication and the higher rate of publication in peer-reviewed journals from some programs

Cronin, E

Sheldon, T

English

White Rose Research Online

This is a repository copy of Factors influencing the publication of health research.White Rose Research Online URL for this paper:http://eprints.whiterose.ac.uk/189/Article:Cronin, E and Sheldon, T orcid.org/0000-0002-7479-5913 (2004) Factors influencing the publication of health research. International Journal of Technology Assessment in Health Care. pp. 351-355. ISSN 0266-4623 eprints@whiterose.ac.ukhttps://eprints.whiterose.ac.uk/Reuse Items deposited in White Rose Research Online are protected by copyright, with all rights reserved unless indicated otherwise. They may be downloaded and/or printed for private study, or other acts as permitted by national copyright laws. The publisher or other rights holders may allow further reproduction and re-use of the full text version. This is indicated by the licence information on the White Rose Research Online record for the item. Takedown If you consider content in White Rose Research Online to be in breach of UK law, please notify us by emailing eprints@whiterose.ac.uk including the URL of the record and the reason for the withdrawal request. International Journal of Technology Assessment in Health Care, 20:3 (2004), 351–355.Copyright c© 2004 Cambridge University Press. Printed in the U.S.A.RESEARCH REPORTSFactors influencing thepublication of health researchEugenia CroninSouth East London Strategic Health AuthorityTrevor SheldonUniversity of YorkObjectives: Assess the degree to which research project findings were published andexplore factors that influenced publication.Methods: Questionnaire to project leaders. Classification of publications and findings.Chi-squared; univariate and multivariate Cox regression analyses.Results: Forty percent of projects published in peer-reviewed journal; highly statisticallysignificant relationships between publication in peer-reviewed journals and (1) projects inResponsive/Fellowships streams (p = .045); and (2) projects awarded >£22,713 (p = .02);influence of study findings not statistically significant.Conclusions: Funders should consider the significant number of studies that did notresult in publication and the higher rate of publication in peer-reviewed journals from someprograms.Keywords: Publication bias, Meta-analysis, Public policyThere is ongoing interest in publication bias, because of itspotential effects on the cumulative, accessible knowledgebase on which health-care decisions are made. This studyexamines outputs from a National Health Service (NHS) Re-search and Development (R&D) program initiated in 1993that supported health research. In 1991, the NHS establishedthe program in response to concerns about the state of basicand clinical research in England and the push for a health-service culture embracing evidence-based medicine, high-lighted by an influential parliamentary review. This pro-gram is publicly funded; therefore, investigating outputsthat may have affected its impact and payback is impor-tant from the point of view of accountability. There is evi-dence that studies with statistically significant or favorableresults are more likely to be published than those with statis-tically nonsignificant or unfavorable results (4). PublicationThis study was undertaken by EC and supervised by TS, as part of a Mastersof Science in Public Health degree at St. George’s Hospital Medical School,University of London. The support of NHS Executive London, R&D Direc-torate is acknowledged.bias, or the differential rate of submission and acceptanceof studies for publication according to their results (1), canresult in misleading summaries of the research evidence (4).Research funders are very interested in achieving optimumimpact from funded projects. Conditions of funding usu-ally include a requirement for dissemination of results aswidely as possible, using such avenues as publication inpeer-reviewed journals, books, and presentation at confer-ences. The NHS R&D program represents one of the firstattempts by any country to establish a coherent R&D in-frastructure to support a knowledge-based health service (3).Little is known about publication patterns from this program.This study examines factors influencing publication in a co-hort of studies commissioned by the North Thames RegionalOffice from 1993 to 1998.METHODSThe aim of the study is to assess the degree to which find-ings from R&D projects funded by this NHS program were351Cronin and SheldonTable 1. R&D Program DescriptionsProgram DescriptionResponsive This program began in 1993 with the remit of funding projects in healthresearch that were of particular value to the NHS. There were norestrictions on the size of funding, and the program did not stipulatetopics to be supported.Education and Training (ET)/Fellowships This program supported Masters and PhD students in order to build researchcapacity. Between 1995 and 2000 it supported approximately32 Masters and 45 PhD students.Commissioned—Research Implementation This program also began in 1993 with the aim of identifying barriers andhighlighting potential levers to the process of getting research intopractice. Over 5 years, it commissioned 78 projects with a value of£3.5 million.Commissioned—Organization & Management (O&M) The original focus of this program was organizational and behavioralaspects of change. Funding of £0.5 million per year was available tocommission projects and the program was first publicized inJuly 1994.Commissioned—Sexual Health This program was advertised nationally in May 1994. The commissioninggroup facilitated bids from the voluntary sector, linking them to academicunits. The total value of the program was just over £1 million.Commissioned—Health of the Nation The R&D directorate issued calls for proposals in Health of the Nationpriority areas in the autumn of 1993 and spring 1994. 16 projects werecommissioned, with a total value of £500 k.Commissioned—Disabilities Priority areas in physical and complex disabilities were defined for thisprogram in 1993. A budget of £1 million was available to cover3 years.Commissioned—Mental Health Priority areas were defined for this program in May 1993. A budgetof £1 million was available to cover 3 years.R&D, Research and Development.published and to explore factors that influenced publication.It uses a cohort of funded research projects to assess publica-tion bias, which is recommended as better than indirect meth-ods, for estimating publication bias (4). The current study wasconducted in three parts: (i) Questionnaire to project leaderswhose projects finished in the period July 1995–December1998; (ii) Classification of publications and findings; and(iii) Analysis; Data on a cohort of 101 research projects thatfinished between 1995 and 1998 were collected from projectfiles and questionnaires completed by project leaders. Thesample selected was markedly heterogeneous, reflecting thebreadth of the various health research funding programs thathad been operating when the studies were commissioned.These programs are described in Table 1.There were eighty-four responses to the questionnaire.Fourteen of these were excluded because, on review, projectend dates had been extended beyond the sample period, pub-lication had not been required at the time, or the project hadterminated prematurely. The remaining sample was of sev-enty studies for which there were completed questionnaireswith the required outcome data.Establishing a classification system proved to be a sig-nificant challenge, as much of the previous work in this areahas used research cohorts with much greater homogeneity.Like the study by Dickersin and colleagues (2), the currentstudy allowed for classification of findings for which statis-tical tests for significance had not been used. The eventualclassification system followed a decision tree, allowing re-search studies to be grouped first into research or develop-ment, whether they were quantitative or qualitative studies,and finally according to the type of findings.Descriptive statistical analyses were undertaken, andchi-squared analysis was performed to detect associationsbetween publication and various factors. Next, a multivari-ate analysis of the rate of publication was undertaken usingsurvival analysis, a method used frequently in epidemiolog-ical research. This method required recording the length offollow-up (time between completion of study and the ques-tionnaire) and time until main peer-reviewed journal publi-cation. Cox proportional hazards regression was then usedto study the relationship between the time to publication andthe set of potential influencing factors (1).This form of regression accounts for situations where theevent of interest (in this case publication in a peer reviewedjournal) does not occur for all cases; in other words, thereare censored data.The graphs are presented in the format “1 minus survivalfunction,” reflecting that survival analysis has been appliedto a situation for which the subjects (in this case researchprojects) start off being unpublished and gradually becomepublished, in contrast to the more usual application to patientsurvival, where subjects start off being alive. This form ofpresentation reflects cumulative gain rather than cumulativeloss.352 INTL. J. OF TECHNOLOGY ASSESSMENT IN HEALTH CARE 20:3, 2004Factors influencing the publication of health researchTable 2. Univariate Associations between Study CharacteristicsVariable % (nos) χ 2 p value1. “Study showed effect”Did not publish in peer reviewed journal 43% (18/42) 1.37 .24Published in peer reviewed journal 57% (16/28)Did not publish at all 38% (8/21) 1.32 .25Published 53% (26/49)Published in peer reviewed journala 57% (16/28) 1.74 .19Published in other form 38% (8/21)Did not use statistical tests for significance 39% (19/49) 6.27 .01Used statistical tests for significance 71% (15/21)2. “Study used statistical tests of significance”Did not publish in peer reviewed journal 66% (28/42) .56 .46Published in peer reviewed journal 75% (21/28)Did not publish at all 76% (16/21) .55 .46Published 67% (33/49)Published in peer reviewed journala 24% (5/21) 1.71 .19Published in other form 43% (9/21)3. “Study contained quantitative element”Did not publish in peer reviewed journal 48% (20/42) 1.16 .28Published in peer reviewed journal 60% (17/28)Did not publish at all 52% (11/21) .003 .96Published 53% (26/49)a These include only projects published in some form.For each of the analyses, performed on SPSS, the vari-able “months to main publication” was used as the “time”element, and “publication in peer reviewed journal” was usedas the “status” or outcome element. Publication in peer-reviewed journal was used rather than “published at all”because the questionnaire respondents had not consistentlyreported month of publication in their responses. For peerreviewed journal articles, this date could be ascertained fromjournal databases, but for book chapters, reports, and otherpublications, this dating was considered more difficult. As aresult, it was possible only to explore timing of publicationin the case of studies which resulted in peer-reviewed journalpublications.FINDINGSComplete data were available for seventy projects. The me-dian study duration was 16 months with a median level offunding per project of £44,361. The median length of follow-up after project completion was 35 months, by which time40 percent had published in a peer-reviewed journal and70 percent had published in some form. A total of 12 per-cent of research projects were responsive (researcher-led ortraining fellowships), and 78 percent were commissionedaccording to national or locally set R&D priorities. Ninety-three percent of projects were classified as research and theremaining 7 percent as development projects. A total of 47.1percent included some quantitative analysis, whereas the re-maining 52.9 percent did not.Analyses of the two-by-two tables undertaken to ex-plore possible associations between a range of independentvariables and the proportion that resulted in publication aresummarized in Table 2. These tests did not find evidenceof an association between publication and whether a study(i) showed an effect, (ii) used statistical tests for significance,or (iii) contained a quantitative element. Two interesting as-sociations between characteristics of studies were found.First, there was a positive association between whether astudy showed an effect, and whether it used statistical testsof significance (χ2 (1) 6.275, p = .01), indicating that stud-ies which used statistical tests of significance are more likelyto have reported an effect in the study (bottom of Table 2).Second, we also found a strong linear correlation betweenthe length of the project (in months) and the amount offunding the project received (r = .6; p = .01). Thus, whendeveloping the multivariate model, we excluded “duration ofproject” and “use of statistical tests,” including instead “sizeof funding” and ”showing an effect,” respectively.The variables were explored one by one (univariate Coxregression). Several variables showed a statistically signifi-cant association with publication in peer-reviewed journals(Table 3). Backward stepwise Cox regression was used toexplore which of the variables were the most important in-fluencers of publication pattern. All the factors explored sofar were entered, and the best model selected by progres-sively removing the variable showing the least evidence ofan independent association (Table 3).Beginning with the eight funding programs, variationwas noted between the curves, indicating a difference inpublication patterns and suggesting that there is a relationshipbetween funding program and publication outcome.The output showed that, overall, funding program wasmarginally significantly related (p = .057) to eventual pub-lication in a peer-reviewed journal; however, there appearedINTL. J. OF TECHNOLOGY ASSESSMENT IN HEALTH CARE 20:3, 2004 353Cronin and SheldonTable 3. Results of Univariate and Multivariate Survival AnalysesVariable name in Univariate OR Multivariate ORCharacteristic of the research project Figures 1 and 2 (95% confidence interval) (95% confidence interval)Funding program: Responsive; All commissioned PROG2programs, including Research Implementation;O&M; ET/FellowshipsFunding program: Responsive & PROG4 3.5 (1.36–9.1) p = .009 2.65; (1.02–6.8) p = .045Fellowships versus CommissionedAmount of funding: <£22,713 vs >£22,713 £TWOCAT .16 (.04–.69) p = .01 .18 (.04–.8) p = .02Whether the study showed an effect or not — .53 (.25–1.1) p = .1 —Whether tests for statistical significance were used — 1.01 (.43–2.4) p = .98 —Whether the project was research or development — 2.8 (.96–8.32) p = .06 —Whether the research was quantitative — 1.07 (.5–2.31) p = .85 —or exclusively qualitativeOR, odds ratio; O&M, Organization & Management; ET, Education and Training.Table 4. Final Output Statistics95% CI for Exp(B)Variable B SE Wald df Sig R Exp(B) Lower Upper£TWOCAT - 1.6856 .7445 5.1257 1 .0236 −.1218 .1853 .043 .797PROG4 .9745 .4868 4.0071 1 .0453 .0976 2.6500 1.021 6.881SE, standard error; Sig, significance; CI, confidence interval.to be little difference between individual funding programs.Therefore, the five Commissioned programs were groupedtogether so that four categories of program remained: Re-sponsive; All Commissioned programs, including ResearchImplementation; Commissioned—Organization & Mana-gement (O&M); Education and Training (ET)/Fellowships(PROG2).1.0.8.6.4.20.0-2-10 0 10Months to publication of MAIN pub./or to questionnaire response20 30 40 50 6021PROG4One Minus Cum SurvivalFigure 1. “1 minus survival function” for patterns 1–2. PROG4: 1, Responsive and Education and Training/Fellowships;2, Commissioned programs and Organization & Management.The survival curve indicated a natural break between re-sponsive and ET/Fellowships; and Commissioned programsincluding O&M. The final best Cox regression model isshown at Table 4, and in Figures 1 and 2. It indicates that thesize of the funding (£TWOCAT, which differentiates projectsthat received up to £22,713 from those that received overthis amount) and type of research program under which the354 INTL. J. OF TECHNOLOGY ASSESSMENT IN HEALTH CARE 20:3, 2004Factors influencing the publication of health research.8.6.4.20.0-2-10 0 10Months to publication of MAIN pub./or to questionnaire response20 30 40 50 6021£TWOCATOne Minus Cum SurvivalFigure 2. “1 minus survival function” for patterns 1–2. £TWOCAT: 1, Up to £22,713 (minimum £1000); 2, Over £22,713(maximum $141,124).project was funded (PROG4, which differentiates Responsiveand ET/Fellowship projects from those in Commissionedprograms) are both associated with publication rate. Othervariables were not shown to be associated with the rate ofpublication, but a larger sample is needed to explore this.COMMENTStatistically significant independent relationships were foundbetween the rate of publication in peer-reviewed journals andboth the type of program (Responsive/Commissioned) andthe cost of the research. Whether studies showed an effect orthe direction of this effect was not statistically significantlyassociated with rate of publication, although this could bedue to type II error.This is one of the first studies exploring publication ofoutputs from the NHS R&D program. Two factors clearlyaffected the rate of publication over a median follow-upperiod of 35 months after study completion. Researcher-initiated projects were more likely to be published. Thisfinding may be important, given the gradual replacement ofresponsive funding streams in the NHS in favor of researchcommissioned in areas of nationally determined priorities.In addition, better-funded projects had a higher rate of pub-lication; however, it is not clear whether this finding reflectsa tendency for larger projects to result in peer-reviewed pub-lications or whether researchers capable of attracting greaterfunds are more successful at publishing.Importantly, there was no evidence of publication bias,although the numbers were small. This study should be re-peated in other parts of the national and regional NHS R&Dprogram and pooled.REFERENCES1. Dickersin K, Chan S, Chalmers TC, Sacks HS, Smith H. Pub-lication bias and clinical trials. ControlClin Trials. 1987;8:343-353.2. Dickersin K, Min YL, Meinert CL. Factors influencing publica-tion of research results. Follow-up of applications submitted totwo institutional review boards. JAMA. 1992;267:374-378.3. Peckham M. Research for health. London: Department of Health;1993.4. Song F, Eastwood AJ, Gilbody S, Duley L, Sutton AJ. Publicationand related biases. Health Technol Assess. 2000;4: No. 10.INTL. J. OF TECHNOLOGY ASSESSMENT IN HEALTH CARE 20:3, 2004 355

Factors influencing the publication of health research

https://eprints.whiterose.ac.uk/189/1/sheldonta4.pdf

Factors influencing the publication of health research

Abstract

Similar works

Full text

Available Versions

White Rose Research Online

White Rose Research Online