Supplemental materials for preprint: Exploring Evidence Selection with the Inclusion Network

Supplemental materials for the preprint, Exploring Evidence Selection with the Inclusion Network. Preprint abstract: Although systematic reviews are intended to provide trusted and actionable scientific knowledge to meet the needs of policy makers and medical professionals, it is common for systematic reviews on the same topic to yield different conclusions. In this paper, we focus on the primary literature actually synthesized, by defining a special purpose citation subnetwork called the inclusion network, which we apply to previously collected data: 27 systematic reviews on exercise and depression and 14 systematic reviews on dietary salt. We develop a suite of analytical methods, utilizing network visualization (static and temporal), one existing network metric (Jaccard similarity), and two customized network metrics (adjusted Jaccard similarity and a measure of evidence eccentricity we call the “dandelion-ness” ratio). Applying these visual and quantitative analytics to the inclusion networks has given us insight into evidence selection and hidden disagreements that cannot be readily derived from simply reading all SRRs, revealing different reasons the evidence selection was discordant as well as evolution in consensus formation over time. More work is required to understand the inclusion network and fully realize its analytical potential; we believe it has promise as a diagnostic tool to identify points of disagreement and facilitate open discussion to resolve them