Selecting and implementing overview methods: implications from five exemplar overviews

This is the final version of the article. Available from BioMed Central via the DOI in this record.Background Overviews of systematic reviews are an increasingly popular method of evidence synthesis; there is a lack of clear guidance for completing overviews and a number of methodological challenges. At the UK Cochrane Symposium 2016, methodological challenges of five overviews were explored. Using data from these five overviews, practical implications to support methodological decision making of authors writing protocols for future overviews are proposed. Methods Methods, and their justification, from the five exemplar overviews were tabulated and compared with areas of debate identified within current literature. Key methodological challenges and implications for development of overview protocols were generated and synthesised into a list, discussed and refined until there was consensus. Results Methodological features of three Cochrane overviews, one overview of diagnostic test accuracy and one mixed methods overview have been summarised. Methods of selection of reviews and data extraction were similar. Either the AMSTAR or ROBIS tool was used to assess quality of included reviews. The GRADE approach was most commonly used to assess quality of evidence within the reviews. Eight key methodological challenges were identified from the exemplar overviews. There was good agreement between our findings and emerging areas of debate within a recent published synthesis. Implications for development of protocols for future overviews were identified. Conclusions Overviews are a relatively new methodological innovation, and there are currently substantial variations in the methodological approaches used within different overviews. There are considerable methodological challenges for which optimal solutions are not necessarily yet known. Lessons learnt from five exemplar overviews highlight a number of methodological decisions which may be beneficial to consider during the development of an overview protocol.The overview conducted by Pollock [19] was supported by a project grant from the Chief Scientist Office of the Scottish Government. The overview conducted by McClurg [21] was supported by a project grant by the Physiotherapy Research Foundation. The overview by Hunt [22] was supported as part of doctoral programme funding by the National Institute for Health Research (NIHR) Collaboration for Leadership in Applied Health Research and Care South West Peninsula (PenCLAHRC). The overview conducted by Estcourt [20] was supported by an NIHR Cochrane Programme Grant for the Safe and Appropriate Use of Blood Components. The overview conducted by Brunton [23] was commissioned by the Department of Health as part of an ongoing programme of work on health policy research synthesis. Alex Pollock is employed by the Nursing, Midwifery and Allied Health Professions (NMAHP) Research Unit, which is supported by the Chief Scientist Office of the Scottish Government. Pauline Campbell is supported by the Chief Nurses Office of the Scottish Government

Assessing the Effectiveness of a Computer Simulation in Introductory Undergraduate Environments

We present studies documenting the effectiveness of using a computer simulation, specifically the Circuit Construction Kit (CCK) developed as part of the Physics Education Technology Project (PhET) [1, 2], in two environments: an interactive college lecture and an inquiry-based laboratory. In the first study conducted in lecture, we compared students viewing CCK to viewing a traditional demonstration during Peer Instruction [3]. Students viewing CCK had a 47% larger relative gain (11% absolute gain) on measures of conceptual understanding compared to traditional demonstrations. These results led us to study the impact of the simulation's explicit representation for visualizing current flow in a laboratory environment, where we removed this feature for a subset of students. Students using CCK with or without the explicit visualization of current performed similarly to each other on common exam questions. Although the majority of students in both groups favored the use of CCK over real circuit equipment, the students who used CCK without the explicit current model favored the simulation more than the other grou

Permutation sampling in Path Integral Monte Carlo

A simple algorithm is described to sample permutations of identical particles in Path Integral Monte Carlo (PIMC) simulations of continuum many-body systems. The sampling strategy illustrated here is fairly general, and can be easily incorporated in any PIMC implementation based on the staging algorithm. Although it is similar in spirit to an existing prescription, it differs from it in some key aspects. It allows one to sample permutations efficiently, even if long paths (e.g., hundreds, or thousands of slices) are needed. We illustrate its effectiveness by presenting results of a PIMC calculation of thermodynamic properties of superfluid Helium-four, in which a very simple approximation for the high-temperature density matrix was utilized

Estimating the degree of saturation in mutant screens

Large-scale screens for loss-of-function mutants have played a significant role in recent advances in developmental biology and other fields. In such mutant screens, it is desirable to estimate the degree of saturation of the screen (i.e., what fraction of the possible target genes has been identified). We applied Bayesian and maximum-likelihood methods for estimating the number of loci remaining undetected in large-scale screens and produced credibility intervals to assess the uncertainty of these estimates. Since different loci may mutate to alleles with detectable phenotypes at different rates, we also incorporated variation in the degree of mutability among genes, using either gamma-distributed mutation rates or multiple discrete mutation rate classes. We examined eight published data sets from large-scale mutant screens and found that credibility intervals are much broader than implied by previous assumptions about the degree of saturation of screens. The likelihood methods presented here are a significantly better fit to data from published experiments than estimates based on the Poisson distribution, which implicitly assumes a single mutation rate for all loci. The results are reasonably robust to different models of variation in the mutability of genes. We tested our methods against mutant allele data from a region of the Drosophila melanogaster genome for which there is an independent genomics-based estimate of the number of undetected loci and found that the number of such loci falls within the predicted credibility interval for our models. The methods we have developed may also be useful for estimating the degree of saturation in other types of genetic screens in addition to classical screens for simple loss-of-function mutants, including genetic modifier screens and screens for protein-protein interactions using the yeast two-hybrid method

Correlating Student Beliefs With Student Learning Using The Colorado Learning Attitudes about Science Survey

A number of instruments have been designed to probe the variety of attitudes, beliefs, expectations, and epistemological frames taught in our introductory physics courses. Using a newly developed instrument -- the Colorado Learning Attitudes about Science Survey (CLASS)[1] -- we examine the relationship between students' beliefs about physics and other educational outcomes, such as conceptual learning and student retention. We report results from surveys of over 750 students in a variety of courses, including several courses modified to promote favorable beliefs about physics. We find positive correlations between particular student beliefs and conceptual learning gains, and between student retention and favorable beliefs in select categories. We also note the influence of teaching practices on student beliefs

Demonstration of non-Markovian process characterisation and control on a quantum processor

In the scale-up of quantum computers, the framework underpinning fault-tolerance generally relies on the strong assumption that environmental noise affecting qubit logic is uncorrelated (Markovian). However, as physical devices progress well into the complex multi-qubit regime, attention is turning to understanding the appearance and mitigation of correlated -- or non-Markovian -- noise, which poses a serious challenge to the progression of quantum technology. This error type has previously remained elusive to characterisation techniques. Here, we develop a framework for characterising non-Markovian dynamics in quantum systems and experimentally test it on multi-qubit superconducting quantum devices. Where noisy processes cannot be accounted for using standard Markovian techniques, our reconstruction predicts the behaviour of the devices with an infidelity of $10^{-3}$. Our results show this characterisation technique leads to superior quantum control and extension of coherence time by effective decoupling from the non-Markovian environment. This framework, validated by our results, is applicable to any controlled quantum device and offers a significant step towards optimal device operation and noise reduction

Practical risk assessment of the relaxation of LOM protection settings in NIE networks' distribution system

This paper presents methodology, experience and practical outcomes of the risk assessment-based revision of Loss-OfMains (LOM) protection settings in NIE Networks’ distribution system. An investigative project has been undertaken by the authors to revise the current LOM practice as recommended by the G59/1/NI regulation, and to propose the settings which would meet the all-Ireland transmission system stability criteria. It is also important to ensure that any increased personal risk is realistically quantified and satisfies the Health and Safety requirements. Both aspects (i.e. LOM protection stability and sensitivity) are covered in the paper. The results and observations included in the paper aim to provide the means and supporting evidence for achieving best compromise in the revision of LOM protection settings

Sustaining Educational Reforms in Introductory Physics

While it is well known which curricular practices can improve student performance on measures of conceptual understanding, the sustaining of these practices and the role of faculty members in implementing these practices are less well understood. We present a study of the hand-off of Tutorials in Introductory Physics from initial adopters to other instructors at the University of Colorado, including traditional faculty not involved in physics education research. The study examines the impact of implementation of Tutorials on student conceptual learning across eight first-semester, and seven second-semester courses, for fifteen faculty over twelve semesters, and includes roughly 4000 students. It is possible to demonstrate consistently high, and statistically indistinguishable, student learning gains for different faculty members; however, such results are not the norm, and appear to rely on a variety of factors. Student performance varies by faculty background - faculty involved in, or informed by physics education research, consistently post higher student learning gains than less-informed faculty. Student performance in these courses also varies by curricula used - all semesters in which the research-based Tutorials and Learning Assistants are used have higher student learning gains than those semesters that rely on non-research based materials and do not employ Learning Assistants.Comment: 21 pages, 4 figures, and other essential inf

Development of quantum perspectives in modern physics

Introductory undergraduate courses in classical physics stress a perspective that can be characterized as realist; from this perspective, all physical properties of a classical system can be simultaneously specified and thus determined at all future times. Such a perspective can be problematic for introductory quantum physics students, who must develop new perspectives in order to properly interpret what it means to have knowledge of quantum systems. We document this evolution in student thinking in part through pre- and post-instruction evaluations using the Colorado Learning Attitudes about Science Survey. We further characterize variations in student epistemic and ontological commitments by examining responses to two essay questions, coupled with responses to supplemental quantum attitude statements. We find that, after instruction in modern physics, many students are still exhibiting a realist perspective in contexts where a quantum-mechanical perspective is needed. We further find that this effect can be significantly influenced by instruction, where we observe variations for courses with differing learning goals. We also note that students generally do not employ either a realist or a quantum perspective in a consistent manner.Comment: 18 pages, plus references; 3 figures; 9 tables. PACS: 01.40.Fk, 03.65._