Quality of meta-analysis in terms of actual statistical analysis

Meta-analysis is an essential tool that facilitates clinicians, medical experts and decision makers to cope with the information overload in the public and healthcare sectors. The publication of meta-analyses is increasing rapidly every day with less or more of methodological rigour. Clinicians and medical experts depend wholeheartedly on the results and conclusions obtained from analyzing meta-analysis in order to assess the clinical effectiveness of healthcare intervention on a daily basis. Meta-analysis provides a specific estimate of a relationship which may also indicate if there is any need for further research. But the foundational problem of performing the clinically powerful meta-analysis is the guideline for how similar the studies must be in order to meet the inclusion criteria of the meta-analysis and the reliability of its conclusions {ERIC(2010)}. When there are discrepancies in the studies being combined and patient populations being studied, meta-analysis may provide results that are wrong. These may mislead potential users of meta-analysis to give wrong prescriptions to their patients. One requirement is to prepare a validated and reliable checklist that can assess the quality of meta-analysis in terms of reporting, methodology, science and most especially, the actual statistical analysis. Literature review reveals that existing checklists mainly focus on other aspects of quality with little or no attention to the quality of statistical methodology. Consequently, this thesis attempts to cover this gap. OBJECTIVE- To construct an appropriate validated quality instrument. Use the instrument to assess the quality of selected meta-analyses in terms of actual statistical analysis. To assess accuracy and consistency of reported estimates using Lee's methods for checking errors in reported relative risks, odds ratios and confidence intervals{Lee(1999)}. STUDIES - Eligible articles (Meta-analysis of randomised controlled trials) identified in the Cochrane database, Web of Knowledge and Medline databases were used in this project. Eligibility criteria include studies published in English language, as a full report between the periods of 2000-2008, have a comprehensive search strategy and have clear methods of selecting studies for inclusion and performed statistical analysis. We developed a checklist that measures the quality of meta-analysis in terms of actual statistical analysis and used the instrument to assess papers published in both Cochrane and Non-Cochrane reviews. RESULTS: A sample size of 100 papers was obtained using an estimated maximum error bound of 0.1. Studies were allocated equally between Cochrane and Non-Cochrane publications and selections were made from electronic databases. Records of meta-analysis of randomised contolled trials published in English, full text and journal articles between the periods of 2000 - 2008 show that there were 515 results out of 5821 records of meta-analysis published in Cochrane library, 507 out of 1434 records and 130 out of 135 records of meta-analysis published in Web of Knowledge and Medline respectively. Simple random sampling, implemented in R statistical package, was used to select random sample of studies from each database. 83 out of the 100 selected studies met the inclusion criteria - 42 studies from Cochrane reviews and 41 from Non-Cochrane reviews. Reporting and methodology quality are high in the two databases. However, in terms of statistical analysis, both databases are unlikely to explicitly state the design of individual studies combined in the meta- analysis. The Cochrane review is more likely to contact authors of published studies than their Paper-base counterparts. Cochrane reviews are less likely also, to use OQAQ(Overview Quality Assessment Questionnaire) and QUOROM (Quality of Reporting of Meta-analyses)in asssessment of validity of studies than paper reviews. There was no double counting of some aspects of studies identified among Paper-base Journals while we discovered four studies in Cochrane reviews that double counted the control arms. However, there was no simple double counting of studies found in both Cochrane and Non-Cochrane reviews. Lee's checks were performed on the twenty selected studies to verify errors on reported odd ratios, relative risks and confidence intervals. Some studies included in the meta-analysis reported zero events either in the treatment or control groups or both which led to a disparity between our calculated results and the estimates reported by the authors. The addition of a continuity correction factor of 0.5 to each cell of the studies with zero events took care of the disparities. Mabinary sas macro designed by Weir and Senn{weir(2008)} was also used to assess and check the validity of reported odd ratios, relative risks and confidence intervals on both reviews. The results obtained using the macro are consistent with the original reported results in most of the studies. Studies reporting relative risks in both Paper-Base Journals and Cochrane reviews are more likely to disagree with the Lee's requirement on minimum subject size and number of diseased subjects in either exposure groups given the CI, than those reporting odd ratios. These studies also have large outcomes. This seems to suggest that Lee's checks are not reliable for studies reporting relative risks, especially when outcomes are relatively large. CONCLUSION: Cochrane Handbooks and scales relating to specific interventions were mostly used to assess quality of studies in Cochrane reviews. Results showed no statistically significant difference between the reporting and methodological quality of Cochrane and non-Cochrane publications. More improvement is needed in the reportage of the design of included studies in both Cochrane and non-Cochrane reviews. This will help establish if the combined studies and the statistical method used in combining them are compatible. However, double counting of some aspects of studies was found in some meta-analysis selected from Cochrane reviews. Analysis suggests that studies reporting odd ratios are likely to be consistent with Lee's checks than those reporting relative risks. We also showed that Peter Lee's checks involving totals cannot be relied on to assess the quality of studies reporting relative risks