Quality assessment tools for evidence from environmental science

Assessment of the quality of studies is a critical component of evidence syntheses such as systematic reviews (SRs) that are used to inform policy decisions. To reduce the potential for reviewer bias, and to ensure that the findings of SRs are transparent and reproducible, organisations such as the Cochrane Collaboration, the Campbell Collaboration, and the Collaboration for Environmental Evidence, recommend the use of formal quality assessment tools as opposed to informal expert judgment. However, there is a bewildering array of around 300 formal quality assessment tools that have been identified in the literature, and it has been demonstrated that the use of different tools for the assessment of the same studies can result in different estimates of quality, which can potentially reverse the conclusions of a SR. It is therefore important to consider carefully, the choice of quality assessment tool. We argue that quality assessment tools should: (1) have proven construct validity (i.e. the assessment criteria have demonstrable link with what they purport to measure), (2) facilitate inter-reviewer agreement, (3) be applicable across study designs, and (4) be quick and easy to use. Our aim was to examine current best practice for quality assessment in healthcare and investigate the extent to which these best practices could be useful for assessing the quality of environmental science studies. The feasibility of this transfer is demonstrated in a number of existing SRs on environmental topics. We propose that environmental practitioners should revise, test and adopt the best practice quality assessment tools used in healthcare as a recommended approach for application to environmental science. We provide pilot versions of quality assessment tools, modified from the best practice tools used in healthcare, for application on studies from environmental science.</p

The effects of non-native signal crayfish (Pacifastacus leniusculus) on fine sediment and sediment-biomonitoring

The North American signal crayfish (Pacifastacus leniusculus) has invaded freshwater ecosystems across Europe. Recent studies suggest that predation of macroinvertebrates by signal crayfish can affect the performance of freshwater biomonitoring tools used to assess causes of ecological degradation. Given the reliance on biomonitoring globally, it is crucial that the potential influence of invasive species is better understood. Crayfish are also biogeomorphic agents, and therefore, the aim of this study was to investigate whether sediment-biomonitoring tool outputs changed following signal crayfish invasions, and whether these changes reflected post-invasion changes to deposited fine sediment, or changes to macroinvertebrate community compositions unrelated to fine sediment. A quasi-experimental study design was employed, utilising interrupted time series analysis of long-term environmental monitoring data and a hierarchical modelling approach. The analysis of all sites (n=71) displayed a small, but statistically significant increase between pre- and post-invasion index scores for the Proportion of Sediment-sensitive Invertebrates (PSI) index biomonitoring tool (4.1, p<0.001, 95%CI: 2.1, 6.2), which can range from 0 to 100, but no statistically significant difference was observed for the empirically-weighted PSI (0.4, p=0.742, 95%CI: -2.1, 2.9), or fine sediment (-2.3, p=0.227, 95%CI: -6.0, 1.4). Subgroup analyses demonstrated changes in biomonitoring tool scores ranging from four to 10 percentage points. Importantly, these subgroup analyses showed relatively small changes to fine sediment, two of which were statistically significant, but these did not coincide with the expected responses from biomonitoring tools. The results suggest that sediment-biomonitoring may be influenced by signal crayfish invasions, but the effects appear to be context dependent, and perhaps not the result of biogeomorphic activities of crayfish. The low magnitude changes to biomonitoring scores are unlikely to result in an incorrect diagnosis of sediment pressure, particularly as these tools should be used alongside a suite of other pressure-specific indices

Investigating the influence of multiple pressures on biomonitoring tools

Freshwater ecosystems are among the most impacted ecosystems globally. In order to conserve and restore these environments, monitoring agencies need to be able to identify degraded rivers and streams, and diagnose the causes of degradation. Biomonitoring, defined as 'the use of biota to gauge and track changes in the environment', is one approach that is widely used throughout Europe, including for water management and legislative purposes, driving decisions on remediation methods. Despite the implications of an incorrect diagnosis, many biomonitoring tools lack thorough validation using independent data from sites that (i) cover the full range of environmental characteristics of the rivers and streams to which they will be applied, (ii) are subject to the full gradient of the pressure(s) of concern, and iii) are impacted by multiple pressures, reflecting a realistic assessment of the performance of the tool. We provide examples of a range of approaches for testing biomonitoring tools using long term monitoring data, investigating the influence of confounding factors and the presence of multiple anthropogenically derived pressures. This analysis includes the use of a novel method of interrupted time series analysis to determine the influence of invasions of non-native species on biomonitoring tool outputs. The various approaches are applied to the testing of a sediment-specific biotic index (the Empirically-weighted Proportion of Sediment-sensitive Invertebrates; E-PSI), highlighting the potential for opportunistic data analysis to enable ecologically relevant and hypothesis driven testing, over large spatial and temporal scales

Developing an improved biomonitoring tool for fine sediment: combining expert knowledge and empirical data

AbstractThe Proportion of Sediment-sensitive Invertebrates (PSI) index is a biomonitoring tool that is designed to identify the degree of sedimentation in rivers and streams. Despite having a sound biological basis, the tool has been shown to have only a moderate correlation with fine sediment, which although comparable to other pressure specific indices, limits confidence in its application. The aim of this study was to investigate if the performance of the PSI index could be enhanced through the use of empirical data to supplement the expert knowledge and literature which were used to determine the original four fine sediment sensitivity ratings. The empirical data used, comprised observations of invertebrate abundance and percentage fine sediment, collected across a wide range of reference condition temperate stream and river ecosystems (model training dataset n=2252). Species were assigned sensitivity weights within a range based on their previously determined sensitivity rating. Using a range of weights acknowledges the breadth of ecological niches that invertebrates occupy and also their differing potential as indicators. The optimum species-specific sensitivity weights were identified using non-linear optimisation, as those that resulted in the highest Spearman's rank correlation coefficient between the Empirically-weighted PSI (E-PSI) scores and deposited fine sediment in the model training dataset. The correlation between percentage fine sediment and E-PSI scores in the test dataset (n=252) was eight percentage points higher than the correlation between percentage fine sediment and the original PSI scores (E-PSI rs=−0.74, p<0.01 compared to PSI rs=−0.66, p<0.01). This study demonstrates the value of combining a sound biological basis with evidence from large empirical datasets, to test and enhance the performance of biomonitoring tools to increase confidence in their application

The effects of run-of-river hydroelectric power schemes on invertebrate community composition in temperate streams and rivers

Run-of-river (ROR) hydroelectric power (HEP) schemes are often presumed to be less ecologically damaging than large-scale storage HEP schemes. However, there is currently limited scientific evidence on their ecological impact. The aim of this article is to investigate the effects of ROR HEP schemes on communities of invertebrates in temperate streams and rivers, using a multi-site Before-After, Control-Impact (BACI) study design. The study makes use of routine environmental surveillance data collected as part of long-term national and international monitoring programmes at 22 systematically-selected ROR HEP schemes and 22 systematically-selected paired control sites. Five widely-used family-level invertebrate metrics (richness, evenness, LIFE, E-PSI, WHPT) were analysed using a linear mixed effects model. The analyses showed that there was a statistically significant effect (p<0.05) of ROR HEP construction and operation on the evenness of the invertebrate community. However, no statistically significant effects were detected on the four other metrics of community composition. The implications of these findings are discussed in this article and recommendations are made for best-practice study design for future invertebrate community impact studies

The Effects of Run-of-River Hydroelectric Power Schemes on Fish and Invertebrate Community Composition in Temperate Streams and Rivers

Run-of-river (ROR) hydroelectric power schemes are often presumed to be less environmentally-damaging than large-scale storage schemes. However, there are currently only a limited number of peer-reviewed studies on their physical and ecological impact. This presentation will summarise the findings from a policy secondment, funded by the UK’s Natural Environment Research Council and the Environment Agency of England, which investigated the impacts of ROR hydroelectric power schemes on fish and invertebrate communities in temperate streams and rivers, using Before-After, Control-Impact (BACI) study designs. The study made use of routine environmental surveillance data collected as part of long-term national and international monitoring programmes at systematically-selected ROR hydroelectric power schemes and systematically-selected paired control sites. Five metrics of invertebrate community composition and six area-normalised metrics of fish community composition were analysed using linear mixed effects models. The results are discussed with respect to impacts from other sources of power, and recommendations are made for best-practice study design for future freshwater community impact studies