Guidance for the design and reporting of studies evaluating the clinical performance of tests for present or past SARS-CoV-2 infection

Testing for SARS-CoV-2 infection is key in managing the current pandemic. More than 1700 preprints and peer reviewed journal articles evaluating tests for SARS-CoV-2 infection have been published as of January 2021. However, evaluations of these studies have identified many methodological issues, leading to a high risk of bias and difficulties applying the results in practice. Better guidance is urgently needed on the conduct and interpretation of these studies. This article outlines the principles for defining the intended purpose of the test; study population selection; reference standard, test timing; and other critical considerations for the design, reporting, and interpretation of diagnostic accuracy studies. The implementation and accuracy of SARS-CoV-2 tests have major implications for individuals and communities, balancing the potential consequences of continued infection against the need for public health measures, such as the restriction of movements and social activities. Decision making in the current pandemic requires a clear understanding of the clinical performance and limitations of testing. This article provides guidance to assist researchers design robust diagnostic accuracy studies, assist publishers and peer reviewers to assess such studies, and support clinicians and policy makers in their evaluation of the evidence on SARS-CoV-2 testing for clinical and public health decisions. The guidance aims to ensure that studies evaluating the diagnostic accuracy of SARS-CoV-2 tests are conducted as rigorously as possible, in an efficient and timely way

Developing risk prediction models for type 2 diabetes: a systematic review of methodology and reporting

<p>Abstract</p> <p>Background</p> <p>The World Health Organisation estimates that by 2030 there will be approximately 350 million people with type 2 diabetes. Associated with renal complications, heart disease, stroke and peripheral vascular disease, early identification of patients with undiagnosed type 2 diabetes or those at an increased risk of developing type 2 diabetes is an important challenge. We sought to systematically review and critically assess the conduct and reporting of methods used to develop risk prediction models for predicting the risk of having undiagnosed (prevalent) or future risk of developing (incident) type 2 diabetes in adults.</p> <p>Methods</p> <p>We conducted a systematic search of PubMed and EMBASE databases to identify studies published before May 2011 that describe the development of models combining two or more variables to predict the risk of prevalent or incident type 2 diabetes. We extracted key information that describes aspects of developing a prediction model including study design, sample size and number of events, outcome definition, risk predictor selection and coding, missing data, model-building strategies and aspects of performance.</p> <p>Results</p> <p>Thirty-nine studies comprising 43 risk prediction models were included. Seventeen studies (44%) reported the development of models to predict incident type 2 diabetes, whilst 15 studies (38%) described the derivation of models to predict prevalent type 2 diabetes. In nine studies (23%), the number of events per variable was less than ten, whilst in fourteen studies there was insufficient information reported for this measure to be calculated. The number of candidate risk predictors ranged from four to sixty-four, and in seven studies it was unclear how many risk predictors were considered. A method, not recommended to select risk predictors for inclusion in the multivariate model, using statistical significance from univariate screening was carried out in eight studies (21%), whilst the selection procedure was unclear in ten studies (26%). Twenty-one risk prediction models (49%) were developed by categorising all continuous risk predictors. The treatment and handling of missing data were not reported in 16 studies (41%).</p> <p>Conclusions</p> <p>We found widespread use of poor methods that could jeopardise model development, including univariate pre-screening of variables, categorisation of continuous risk predictors and poor handling of missing data. The use of poor methods affects the reliability of the prediction model and ultimately compromises the accuracy of the probability estimates of having undiagnosed type 2 diabetes or the predicted risk of developing type 2 diabetes. In addition, many studies were characterised by a generally poor level of reporting, with many key details to objectively judge the usefulness of the models often omitted.</p

Overview of data-synthesis in systematic reviews of studies on outcome prediction models

Background: Many prognostic models have been developed. Different types of models, i.e. prognostic factor and outcome prediction studies, serve different purposes, which should be reflected in how the results are summarized in reviews. Therefore we set out to investigate how authors of reviews synthesize and report the results of primary outcome prediction studies. Methods: Outcome prediction reviews published in MEDLINE between October 2005 and March 2011 were eligible and 127 Systematic reviews with the aim to summarize outcome prediction studies written in English were identified for inclusion. Characteristics of the reviews and the primary studies that were included were independently assessed by 2 review authors, using standardized forms. Results: After consensus meetings a total of 50 systematic reviews that met the inclusion criteria were included. The type of primary studies included (prognostic factor or outcome prediction) was unclear in two-thirds of the reviews. A minority of the reviews reported univariable or multivariable point estimates and measures of dispersion from the primary studies. Moreover, the variables considered for outcome prediction model development were often not reported, or were unclear. In most reviews there was no information about model performance. Quantitative analysis was performed in 10 reviews, and 49 reviews assessed the primary studies qualitatively. In both analyses types a range of different methods was used to present the results of the outcome prediction studies. Conclusions: Different methods are applied to synthesize primary study results but quantitative analysis is rarely performed. The description of its objectives and of the primary studies is suboptimal and performance parameters of the outcome prediction models are rarely mentioned. The poor reporting and the wide variety of data synthesis strategies are prone to influence the conclusions of outcome prediction reviews. Therefore, there is much room for improvement in reviews of outcome prediction studies. (aut.ref.

Minimal information about T cell assays: the process of reaching the community of T cell immunologists in cancer and beyond

Many assays to evaluate the nature, breadth, and quality of antigen-specific T cell responses are currently applied in human medicine. In most cases, assay-related protocols are developed on an individual laboratory basis, resulting in a large number of different protocols being applied worldwide. Together with the inherent complexity of cellular assays, this leads to unnecessary limitations in the ability to compare results generated across institutions. Over the past few years a number of critical assay parameters have been identified which influence test performance irrespective of protocol, material, and reagents used. Describing these critical factors as an integral part of any published report will both facilitate the comparison of data generated across institutions and lead to improvements in the assays themselves. To this end, the Minimal Information About T Cell Assays (MIATA) project was initiated. The objective of MIATA is to achieve a broad consensus on which T cell assay parameters should be reported in scientific publications and to propose a mechanism for reporting these in a systematic manner. To add maximum value for the scientific community, a step-wise, open, and field-spanning approach has been taken to achieve technical precision, user-friendliness, adequate incorporation of concerns, and high acceptance among peers. Here, we describe the past, present, and future perspectives of the MIATA project. We suggest that the approach taken can be generically applied to projects in which a broad consensus has to be reached among scientists working in fragmented fields, such as immunology. An additional objective of this undertaking is to engage the broader scientific community to comment on MIATA and to become an active participant in the project

Extending an evidence hierarchy to include topics other than treatment: revising the Australian 'levels of evidence'

Background: In 1999 a four-level hierarchy of evidence was promoted by the National Health and Medical Research Council in Australia. The primary purpose of this hierarchy was to assist with clinical practice guideline development, although it was co-opted for use in systematic literature reviews and health technology assessments. In this hierarchy interventional study designs were ranked according to the likelihood that bias had been eliminated and thus it was not ideal to assess studies that addressed other types of clinical questions. This paper reports on the revision and extension of this evidence hierarchy to enable broader use within existing evidence assessment systems. Methods: A working party identified and assessed empirical evidence, and used a commissioned review of existing evidence assessment schema, to support decision-making regarding revision of the hierarchy. The aim was to retain the existing evidence levels I-IV but increase their relevance for assessing the quality of individual diagnostic accuracy, prognostic, aetiologic and screening studies. Comprehensive public consultation was undertaken and the revised hierarchy was piloted by individual health technology assessment agencies and clinical practice guideline developers. After two and a half years, the hierarchy was again revised and commenced a further 18 month pilot period. Results: A suitable framework was identified upon which to model the revision. Consistency was maintained in the hierarchy of "levels of evidence" across all types of clinical questions; empirical evidence was used to support the relationship between study design and ranking in the hierarchy wherever possible; and systematic reviews of lower level studies were themselves ascribed a ranking. The impact of ethics on the hierarchy of study designs was acknowledged in the framework, along with a consideration of how harms should be assessed. Conclusion: The revised evidence hierarchy is now widely used and provides a common standard against which to initially judge the likelihood of bias in individual studies evaluating interventional, diagnostic accuracy, prognostic, aetiologic or screening topics. Detailed quality appraisal of these individual studies, as well as grading of the body of evidence to answer each clinical, research or policy question, can then be undertaken as required.Tracy Merlin, Adele Weston and Rebecca Toohe

Tissue Specific Deletion of Inhibitor of Kappa B Kinase 2 with OX40-Cre Reveals the Unanticipated Expression from the OX40 Locus in Skin Epidermis

NF-κB signalling plays an essential role in T cell activation and generation of regulatory and memory populations in vivo. In the present study, we aimed to investigate the role of NF-κB signalling in post-activation T cells using tissue specific ablation of inhibitor of kappa-B kinase 2 expression, an important component of the inhibitor of kappa-B kinase complex in canonical NF-κB signalling. The OX40 antigen is expressed on activated T cells. Therefore, we used previously described mouse strain expressing Cre recombinase from the endogenous OX40 locus. Ablation of IKK2 expression using OX40Cre mice resulted in the development of an inflammatory response in the skin epidermis causing wide spread skin lesions. The inflammatory response was characterised by extensive leukocytic infiltrate in skin tissue, hyperplasia of draining lymph nodes and widespread activation in the T cell compartment. Surprisingly, disease development did not depend on T cells but was rather associated with an unanticipated expression of Cre in skin epidermis, and activation of the T cell compartment did not require Ikbk2 deletion in T cells. Employment of Cre reporter strains revealed extensive Cre activity in skin epidermis. Therefore, development of skin lesions was rather more likely explained by deletion of Ikbk2 in skin keratinocytes in OX40Cre mice

Assessing health-related quality of life in patients with inflammatory bowel disease, in Crete, Greece

BACKGROUND: Health Related Quality of Life (HRQoL) is an important outcome measure in Inflammatory Bowel Disease (IBD). The aim of our study was to assess HRQoL in a population of 135 Greek patients with IBD. METHODS: A cohort of 135 patients with IBD, 81 with ulcerative colitis (UC) and 54 with Crohn's disease (CD) were enrolled in our study. Demographic and disease-related data were recorded. HRQoL was assessed by a disease-specific and a generic questionnaire, IBDQ and SF-36, respectively. Disease activity was assessed by Harvey-Bradshaw Index and the Colitis Activity Index for CD and UC patients, respectively. RESULTS: Among all variables recorded in our study, only disease activity had a significant effect on HRQoL. Patients with active disease scored significantly lower on both IBDQ and SF-36 when compared to those in remission. Only two among the four IBDQ dimensions, bowel and systemic, had significant ability in distinguishing best patients in remission from those with active disease. CONCLUSIONS: IBD has a negative impact on HRQoL. Patients with active disease are more impaired than patients in remission. In our population of patients bowel and systemic dimensions had a predominant value in patients' perception of quality of life. Patients in our study using the same instrument scored higher than previously reported

Prognostic DNA methylation markers for sporadic colorectal cancer: a systematic review

Background Biomarkers that can predict the prognosis of colorectal cancer (CRC) patients and that can stratify high-risk early stage patients from low-risk early stage patients are urgently needed for better management of CRC. During the last decades, a large variety of prognostic DNA methylation markers has been published in the literature. However, to date, none of these markers are used in clinical practice. Methods To obtain an overview of the number of published prognostic methylation markers for CRC, the number of markers that was validated independently, and the current level of evidence (LoE), we conducted a systematic review of PubMed, EMBASE, and MEDLINE. In addition, we scored studies based on the REMARK guidelines that were established in order to attain more transparency and complete reporting of prognostic biomarker studies. Eighty-three studies reporting on 123 methylation markers fulfilled the study entry criteria and were scored according to REMARK. Results Sixty-three studies investigated single methylation markers, whereas 20 studies reported combinations of methylation markers. We observed substantial variation regarding the reporting of sample sizes and patient characteristics, statistical analyses, and methodology. The median (range) REMARK score for the studies was 10.7 points (4.5 to 17.5) out of a maximum of 20 possible points. The median REMARK score was lower in studies, which reported a p value below 0.05 versus those, which did not (p = 0.005). A borderline statistically significant association was observed between the reported p value of the survival analysis and the size of the study population (p = 0.051). Only 23 out of 123 markers (17%) were investigated in two or more study series. For 12 markers, and two multimarker panels, consistent results were reported in two or more study series. For four markers, the current LoE is level II, for all other markers, the LoE is lower. Conclusion This systematic review reflects that adequate reporting according to REMARK and validation of prognostic methylation markers is absent in the majority of CRC methylation marker studies. However, this systematic review provides a comprehensive overview of published prognostic methylation markers for CRC and highlights the most promising markers that have been published in the last two decades

Reporting Recommendations for Tumor Marker Prognostic Studies (REMARK): Explanation and Elaboration

The REMARK “elaboration and explanation” guideline, by Doug Altman and colleagues, provides a detailed reference for authors on important issues to consider when designing, conducting, and analyzing tumor marker prognostic studies