Background Compulsory submission of a checklist from the relevant reporting guideline is one of the most widespread journal requirements aiming to improve completeness of reporting. However, the current suboptimal levels of adherence to reporting guidelines observed in the literature may indicate that this journal policy is not having a significant effect. Findings We explored whether authors provided the appropriate CONSORT checklist extension for their study and whether there were inconsistencies between what authors claimed on the submitted checklist and what was actually reported in the published paper. We randomly selected 12 randomized trials from three journals that provide the originally submitted checklist and analyzed six core CONSORT items. Only one paper used the appropriate checklist extension and had no inconsistencies between what was claimed in the submitted checklist and what was reported in the published paper. Conclusion Journals should take further actions to take full advantage of the requirement for the submission of fulfilled CONSORT checklists, thus ensuring that these checklists reflect what is reported in the manuscript.</p

Altman, D

Bertizzolo, L

Biggane, AM

Blanco, D

Boutron, I

Cobo, E

Gkioni, E

Glonti, K

Kirkham, J

Olarte, C

Olsen, M

Superchi, C

Background Compulsory submission of a checklist from the relevant reporting guideline is one of the most widespread journal requirements aiming to improve completeness of reporting. However, the current suboptimal levels of adherence to reporting guidelines observed in the literature may indicate that this journal policy is not having a significant effect. Findings We explored whether authors provided the appropriate CONSORT checklist extension for their study and whether there were inconsistencies between what authors claimed on the submitted checklist and what was actually reported in the published paper. We randomly selected 12 randomized trials from three journals that provide the originally submitted checklist and analyzed six core CONSORT items. Only one paper used the appropriate checklist extension and had no inconsistencies between what was claimed in the submitted checklist and what was reported in the published paper. Conclusion Journals should take further actions to take full advantage of the requirement for the submission of fulfilled CONSORT checklists, thus ensuring that these checklists reflect what is reported in the manuscript

Altman, Douglas

Oxford University Research Archive

COMMENTARY Open AccessAre CONSORT checklists submitted byauthors adequately reflecting whatinformation is actually reported inpublished papers?David Blanco1,3* , Alice M. Biggane2,3, Erik Cobo1 and MiRoR networkAbstractBackground: Compulsory submission of a checklist from the relevant reporting guideline is one of the mostwidespread journal requirements aiming to improve completeness of reporting. However, the current suboptimallevels of adherence to reporting guidelines observed in the literature may indicate that this journal policy is nothaving a significant effect.Findings: We explored whether authors provided the appropriate CONSORT checklist extension for their study andwhether there were inconsistencies between what authors claimed on the submitted checklist and what wasactually reported in the published paper. We randomly selected 12 randomized trials from three journals thatprovide the originally submitted checklist and analyzed six core CONSORT items. Only one paper used theappropriate checklist extension and had no inconsistencies between what was claimed in the submitted checklistand what was reported in the published paper.Conclusion: Journals should take further actions to take full advantage of the requirement for the submission offulfilled CONSORT checklists, thus ensuring that these checklists reflect what is reported in the manuscript.Keywords: Reporting guidelines, Completeness of reporting, Reporting inconsistencies, CONSORT, EndorsementBackgroundCompleteness of reporting is a critical issue in health re-search. It enhances the transparency of researchmethods and findings, thus promoting their credibilityand reproducibility. In an attempt to improve complete-ness of reporting, several reporting guidelines for differ-ent study designs and clinical areas were developed inthe last two decades [1]. The Consolidated Standards ofReporting Trials (CONSORT), was created in 1996 tohelp authors report the methods and findings of ran-domized trials [2]. It has been revised and updated twice[3, 4]. ‘Endorsement’ of CONSORT by medical journalshas been one of the most widespread actions imple-mented to improve the levels of completeness of report-ing of randomized trials. It is defined as any of thefollowing situations: (a) journal editorial statement en-dorsing CONSORT; (b) requirement or recommenda-tion in journal’s instructions to authors to followCONSORT when preparing their manuscript; or (c) re-quirement for authors to submit a CONSORT checklistwith their manuscript [5]. Existing evidence shows that,despite modest improvements when CONSORT is en-dorsed by journals, the completeness of reporting of tri-als remains suboptimal [5].In recent years, dozens of medical journals have optedfor policy (c) in the previous paragraph, as it has themost potential to improve completeness of reporting. Inaddition, in an effort to make the editorial process moretransparent and credible, some journals following this* Correspondence: david.blanco.tena@upc.edu1Statistics and Operations Research Department, Barcelona Tech, Barcelona,Spain3INSERM, U1153 Epidemiology and Biostatistics Sorbonne Paris Cité ResearchCenter (CRESS), Methods of Therapeutic Evaluation of Chronic Diseases(METHODS) Team, Paris Descartes University, Sorbonne Paris Cité, ParisF-75014, FranceFull list of author information is available at the end of the article© The Author(s). 2018 Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, andreproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link tothe Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver(http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.Blanco et al. Trials  (2018) 19:80 DOI 10.1186/s13063-018-2475-0policy, such as PLoS One, BMJ Open and Trials, alsomake the original CONSORT checklist submitted by theauthors accessible for their readers as supplementarymaterial. However, the current suboptimal levels of ad-herence to reporting guidelines observed in the literatureacross different research areas and study designs [6] mayindicate that this journal policy is not having the desiredeffect. To date, there has been no investigation intowhether or not there are inconsistencies between whatauthors claim to have reported in the submitted check-list and what is actually reported in the published paper.A number of specific study designs (such as clusterdesigns) or interventions (such as non-pharmacologicalinterventions) have additional specific extensions [7, 8].In the journals mentioned, authors are required to sub-mit the CONSORT extension that applies to their study.Thus, it is essential to assess whether authors actuallyprovide the appropriate extensions for their studies.In this commentary, we explore (1) whether authorsprovide the appropriate CONSORT checklist and (2)whether there are inconsistencies between what authorsclaim to have reported in the submitted checklist andwhat they have actually reported in the published paper,for those papers submitted with the appropriate check-list. Our intention is to illustrate whether CONSORTchecklist fulfilment by authors should be considered aguarantee that CONSORT items are effectively satisfied.Our findingsWe looked at 12 randomly selected randomized trials pub-lished in either Trials, BMJ Open or PLoS One between 1January 2016 and 30 June 2017 (see Additional file 1: Point1, search strategy and study selection). We chose thosejournals because they require authors to submit aCONSORT checklist with their manuscript and make thischecklist accessible for their readers as supplementary ma-terial. For each paper selected, we retrieved the initialCONSORT checklist and manuscript submitted by the au-thors. First, we independently determined whether theCONSORT checklist originally submitted by the authorswas the appropriate extension for the study design. Then,for papers using the appropriate checklist, we compared itwith what was actually reported in the published paper toidentify any inconsistencies (see Additional file 1: Point 2,analysis of inconsistencies). We focused on six core CON-SORT items of the ‘Methods’ and ‘Results’ sections: (6a)outcomes; (8a) randomization or sequence generation; (9)allocation concealment mechanism; (11a) blinding; (13a)flow of participants and (13b) losses and exclusions. Weselected those items because they are essential for system-atic reviewers to evaluate the risk of bias. For each item,the CONSORT explanation and elaboration document [9]was used to determine what information should bereported.We graded the consistency between what authors saidand what they reported for each item as follows: Completely consistent: There was no divergencebetween what authors claimed to have reportedthrough the originally submitted CONSORT checklistand what they reported in the published paper. Partially consistent: This may include either (a) partialabsence of relevant information that was expected tobe reported or (b) information corresponding to thatitem was reported in a different part of the paperfrom that specified in the checklist. Not consistent: Authors claimed to have reportedthat item through the CONSORT checklist but didnot adequately report the information in thepublished paper.From the 12 randomly selected randomized trials, thestandard CONSORT checklist was appropriate for six pa-pers (four of which were standard parallel trials covered byCONSORT and two of which were crossover trials, forwhich there is not an extension yet). The other six re-quired CONSORT extensions (for cluster trials, three; forpragmatic trials, two; and for non-pharmacological inter-ventions, one) but authors did not use them in any case,despite the extensions being available at the time of sub-mission. The aforementioned extensions were publishedbetween 2008 and 2012 [7, 8, 10], yet the six papers requir-ing their uptake were all submitted later than May 2015.For the six papers for which an appropriate CON-SORT checklist was submitted, only one paper hadcomplete consistency between the checklist and the pub-lished paper. The most concerning problems centred onitems 8 and 9 (see Fig. 1). For example, a possible incon-sistency identified regarding CONSORT item 9 (Alloca-tion concealment mechanism – mechanism used toimplement the random allocation sequence, describingany steps taken to conceal the sequence until interven-tions were assigned) is the following: authors claimthrough the checklist that they have reported the item;however, we found that the paper stated “The sequenceof the test conditions was randomized for each partici-pant by LB and KDK. A card was made for each possiblesequence and a card was picked blindly for each partici-pant.” This statement does not make it clear how the au-thors implemented the random allocation sequence norhow they kept the assignment concealed. Picking a carddoes not guarantee that allocation used in the analysishas preceded treatment, nor does it allow readers to re-produce the mechanism used to implement the randomallocation sequence.A full summary of the evaluations for all six itemsacross all six papers is shown in Additional file 2. Thelevel of reporting inconsistencies found for every itemBlanco et al. Trials  (2018) 19:80 Page 2 of 4among the six papers considered in the analysis is pro-vided in Fig. 1. Illustrative examples of inconsistenciesfound for each item are shown in Additional file 3.Why could reporting inconsistencies occur?Among the numerous potential reasons for the presenceof reporting inconsistencies, we underline two explana-tions. Firstly, it is possible that authors are not attentiveto the requirements of CONSORT or, despite their ef-forts to be compliant with the requirements, they arestruggling to interpret certain items or the level of detailthat is required. Examples include: Item 8a ‘Method used to generate the randomallocation sequence’. This item was adequatelyreported in just one of the papers screened. Withinthis item, there were various reasons for theinconsistencies, including: lack of thorough andcomplete reporting from the authors (see examplefor this item in Additional file 3) and thenontechnical use of the term ‘random’ (‘The studynurse randomly opened a preformed envelopecontaining the allocated treatment regimen’). Item 11a ‘If done, who was blinded after assignmentto interventions (for example, participants, careproviders, those assessing outcomes) and how’. Theinitial phrase ‘if done’ may have caused confusionabout whether or not the authors have to reportwhat groups of individuals involved in the trial wereunblinded. To avoid authors not disclosing the lackof blinding if the trial was unmasked, we suggestthat future versions of the CONSORT checklistdelete the phrase ‘if done’.The misinterpretation of CONSORT is a major con-cern, as it means that essential information regardingstudy conduct is miscommunicated. This is particularlyrelevant for item 11a, as according to PRISMA item 19[11], when assessing the risk of bias of a study it is neces-sary to know whether or not patients, health care pro-viders, data collectors and outcome assessors are blinded.Secondly, the issues described in this study might alsolie with the reviewers and editors. It is possible that theyare falsely reassured with regard to the reporting qualityof the manuscripts, merely by the presence of a com-pleted checklist. Moreover, the fact that the reporting in-consistencies persist throughout the editorial processmight mean that editors and reviewers are not usingreporting guidelines as a method to review manuscripts[12] although the CONSORT explanation and elabor-ation document suggests that: ‘Readers, peer reviewersand editors can also use CONSORT to help them critic-ally appraise and interpret reports of randomizedcontrolled trials.’Possible solutionsIn an effort to take full advantage of requiring the sub-mission of checklists, journals should consider clarifyingtheir stance on whether the full checklists, or at least thechecklists’ core items should be examined by editors orreviewers, or even by trained editorial assistants [13].As the page numbers reported by authors in thechecklist are not updated after the peer review processand the typesetting process, they do not correspond tothe page where the information is placed in the pub-lished paper. Having to update the page numbers in thechecklist from original submission to published papercould act as a checkpoint for editors or reviewers to re-mind them to verify whether authors are appropriatelyreporting the key information in the latest version of themanuscript. An alternative solution could be to askauthors to address the section and the paragraph wherethe information corresponding to each item is reported.This would reduce the risk of overburdening the authors01234566a 8a 9 11a 13a 13bNumber of articlesCONSORT item numberNot consistentPartially consistentCompletely consistentFig. 1 Reporting inconsistencies found for the six papers that used the appropriate CONSORT checklistBlanco et al. Trials  (2018) 19:80 Page 3 of 4and could potentially help deter the misconception thatthese checklists are merely bureaucratic. Furthermore,making available the updated checklist could help sys-tematic reviewers easily and quickly find the relevant in-formation to assess the risk of bias of the studiesincluded in the reviews [11].ConclusionsPoor-quality reporting in health research critically affectsthe credibility, reproducibility and generalizability of themethods and findings of randomized trials. For thesereasons, further exploration of methods that will obligateauthors to consistently and accurately fulfil and submitCONSORT checklists is required. Moreover, journalsshould consider making clear whether the checklistsshould be examined by editors, peer reviewers or atrained editorial assistant.Additional filesAdditional file 1: Search strategy, study selection and analysis ofinconsistencies. (DOCX 15 kb)Additional file 2: Summary of evaluations. (XLSX 77 kb)Additional file 3: Examples of inconsistencies. (DOCX 15 kb)AbbreviationsCONSORT: Consolidated Standards of Reporting Trials; MiRoR: Methods inResearch on Research; PRISMA: Preferred Reporting Items for SystematicReviews and Meta-AnalysesAcknowledgementsWe would like to thank the members of the Methods in Research onResearch (MiRoR) Project (http://miror-ejd.eu/) and Marie Sklodowska-CurieActions for their support.FundingThis study is part of the June 2017 Journal Club, organized by David Blanco,Alice Biggane and Erik Cobo within the MiRoR project. This project receivedfunding from the European Union’s Horizon 2020 research and innovationprogramme under the Marie Sklodowska-Curie grant agreement, no 676207.Availability of data and materialsAll data generated or analyzed during this study are included in thispublished paper (and its additional files).Authors’ contributionsDB, AMB, IB, JK and EC contributed to conceptualizing and designing thestudy. DB, AMB and EC carried out the study, while LB, EG, KG, CO, MO andCS acted as second reviewers of the included papers. DB drafted themanuscript and AMB and EC made several major revisions. DA, LB, EG, KG,CO, MO and CS made minor revisions. All authors read and approved thefinal version of the manuscript.Authors’ informationMembers of the MiRoR network involved: Doug Altman, Lorenzo Bertizzolo,Isabelle Boutron, Efstathia Gkioni, Ketevan Glonti, Jamie Kirkham, CamilaOlarte, Maria Olsen, Cecilia Superchi.Ethics approval and consent to participateNot required.Consent for publicationNot required.Competing interestsEC is a senior editor of Trials. Two members of the MiRoR network involvedin this commentary also play a role in Trials: Isabelle Boutron is a senioreditor and Douglas Altman is editor-in-chief.Publisher’s NoteSpringer Nature remains neutral with regard to jurisdictional claims inpublished maps and institutional affiliations.Author details1Statistics and Operations Research Department, Barcelona Tech, Barcelona,Spain. 2Department of Biostatistics, University of Liverpool, Liverpool, UK.3INSERM, U1153 Epidemiology and Biostatistics Sorbonne Paris Cité ResearchCenter (CRESS), Methods of Therapeutic Evaluation of Chronic Diseases(METHODS) Team, Paris Descartes University, Sorbonne Paris Cité, ParisF-75014, France.Received: 20 December 2017 Accepted: 12 January 2018References1. EQUATOR Network. Library for health research reporting. http://www.equator-network.org/resource-centre/library. Accessed 5 Jan 2018.2. Begg C, Cho M, Eastwood S, Horton R, Moher D, Olkin I, et al. Improving thequality of reporting of randomized controlled trials. The CONSORTstatement. JAMA. 1996;276(8):637–9.3. Moher D, Schulz KF, Altman DG. The CONSORT statement: revisedrecommendations for improving the quality of reports of parallel-grouprandomised trials. Lancet (London, England). 2001;357(9263):1191–4.4. Schulz KF, Altman DG, Moher D, CONSORT Group. CONSORT 2010statement: Updated guidelines for reporting parallel group randomisedtrials. Int J Surg. 2011;9(8):672–7.5. Turner L, Shamseer L, Altman DG, Schulz KF, Moher D, et al. Does use of theCONSORT Statement impact the completeness of reporting of randomisedcontrolled trials published in medical journals? A Cochrane review. Syst Rev.2012;1(1):60.6. Samaan Z, Mbuagbaw L, Kosa D, Borg Debono V, Dillenburg R, Zhang S, etal. A systematic scoping review of adherence to reporting guidelines inhealth care literature. J Multidiscip Healthc. 2013;6:169–88.7. Campbell MK, Piaggio G, Elbourne DR, Altman DG, CONSORT Group.Consort 2010 statement: extension to cluster randomised trials. BMJ. 2012;345:e5661.8. Boutron I, Moher D, Altman DG, Schulz KF, Ravaud P, CONSORT Group.Extending the CONSORT statement to randomized trials ofnonpharmacologic treatment: explanation and elaboration. Ann Intern Med.2008;148(4):295–309.9. Moher D, Hopewell S, Schulz KF, Montori V, Gøtzsche PC, Devereaux PJ, et al.CONSORT 2010 explanation and elaboration: updated guidelines for reportingparallel group randomised trials. J Clin Epidemiol. 2010;63(8):e1–37.10. Zwarenstein M, Treweek S, Gagnier JJ, Altman DG, Tunis S, Haynes B, et al.Improving the reporting of pragmatic trials: an extension of the CONSORTstatement. BMJ. 2008;337:a2390.11. Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gøtzsche PC, Ioannidis JPA, et al.The PRISMA statement for reporting systematic reviews and meta-analysesof studies that evaluate health care interventions: explanation andelaboration. J Clin Epidemiol. 2009;62(10):e1–34.12. Hirst A, Altman DG. Are peer reviewers encouraged to use reporting guidelines?A survey of 116 health research journals. PLoS One. 2012;7(4):e35621.13. Hopewell S, Collins GS, Boutron I, Yu L-M, Cook J, Shanyinde M, et al. Impact ofpeer review on reports of randomised trials published in open peer reviewjournals: retrospective before and after study. BMJ. 2014;349:g4145.Blanco et al. Trials  (2018) 19:80 Page 4 of 4

English

Are CONSORT checklists submitted by authors adequately reflecting what information is actually reported in published papers?

Are CONSORT checklists submitted by authors adequately reflecting what information is actually reported in published papers?

Abstract

Similar works

Full text

Available Versions

Oxford University Research Archive