Study on automatic citation screening in systematic reviews: reporting, reproducibility and complexity

Background: Challenges in the conduct of systematic reviews have led to research into and development of support tools targeting the process or specific stages. There is a growing body of research into the use of text mining methods for citation screening support. However, these studies are reported with insufficient details to support reproducibility and technical comprehensibility of the models. Aim: To investigate transparency in the reporting of citation screening in systematic reviews particularly as it relates to reproducibility and technical comprehensibility of the models. Method: A literature review was conducted to investigate the methods being used for citation screening support and the type of information reported about them. Consequently, a reproducibility assessment of studies was undertaken to systematically assess the level of reproducibility of the studies and the factors responsible. This was followed by two studies to investigate the structural complexity of the models being used. A text mining based tool was developed to support citation screening and tool support research. Results: The review showed a growing body of research but a lack of technical information about models and reproducibility enabling information. The reproducibility assessment identified information essential to study reproduction and suggested a checklist. The complexity assessment and feature enrichment studies reinforced the need for complexity related information in study reports. The citation screening tool demonstrated how a tool can be useful for both practice and research. Conclusions: Research into text mining based tool support for citation screening in systematic reviews is growing. The field has not experienced much independent validation. It is anticipated that more transparency in studies will increase reproducibility and in-depth understanding leading to the maturation of the field. The citation screen tool presented aims to support research transparency, reproducibility and timely evolution of sustainable tools

Determining the effectiveness of three software evaluation techniques through informal aggregation

An accepted fact in software engineering is that software must undergo verification and validation process during development to ascertain and improve its quality level. But there are too many techniques than a single developer could master, yet, it is impossible to be certain that software is free of defects. So, it is crucial for developers to be able to choose from available evaluation techniques, the one most suitable and likely to yield optimum quality results for different products. Though, some knowledge is available on the strengths and weaknesses of the available software quality assurance techniques but not much is known yet on the relationship between different techniques and contextual behavior of the techniques. Objective: This research investigates the effectiveness of two testing techniques ? equivalence class partitioning and decision coverage and one review technique ? code review by abstraction, in terms of their fault detection capability. This will be used to strengthen the practical knowledge available on these techniques

Data extraction methods for systematic review (semi)automation: Update of a living systematic review [version 2; peer review: 3 approved]

Background: The reliable and usable (semi)automation of data extraction can support the field of systematic review by reducing the workload required to gather information about the conduct and results of the included studies. This living systematic review examines published approaches for data extraction from reports of clinical studies. Methods: We systematically and continually search PubMed, ACL Anthology, arXiv, OpenAlex via EPPI-Reviewer, and the dblp computer science bibliography. Full text screening and data extraction are conducted within an open-source living systematic review application created for the purpose of this review. This living review update includes publications up to December 2022 and OpenAlex content up to March 2023. Results: 76 publications are included in this review. Of these, 64 (84%) of the publications addressed extraction of data from abstracts, while 19 (25%) used full texts. A total of 71 (93%) publications developed classifiers for randomised controlled trials. Over 30 entities were extracted, with PICOs (population, intervention, comparator, outcome) being the most frequently extracted. Data are available from 25 (33%), and code from 30 (39%) publications. Six (8%) implemented publicly available tools Conclusions: This living systematic review presents an overview of (semi)automated data-extraction literature of interest to different types of literature review. We identified a broad evidence base of publications describing data extraction for interventional reviews and a small number of publications extracting epidemiological or diagnostic accuracy data. Between review updates, trends for sharing data and code increased strongly: in the base-review, data and code were available for 13 and 19% respectively, these numbers increased to 78 and 87% within the 23 new publications. Compared with the base-review, we observed another research trend, away from straightforward data extraction and towards additionally extracting relations between entities or automatic text summarisation. With this living review we aim to review the literature continually

Data extraction methods for systematic review (semi)automation: Update of a living systematic review [version 2; peer review: 3 approved]

Background: The reliable and usable (semi)automation of data extraction can support the field of systematic review by reducing the workload required to gather information about the conduct and results of the included studies. This living systematic review examines published approaches for data extraction from reports of clinical studies. Methods: We systematically and continually search PubMed, ACL Anthology, arXiv, OpenAlex via EPPI-Reviewer, and the dblp computer science bibliography. Full text screening and data extraction are conducted within an open-source living systematic review application created for the purpose of this review. This living review update includes publications up to December 2022 and OpenAlex content up to March 2023. Results: 76 publications are included in this review. Of these, 64 (84%) of the publications addressed extraction of data from abstracts, while 19 (25%) used full texts. A total of 71 (93%) publications developed classifiers for randomised controlled trials. Over 30 entities were extracted, with PICOs (population, intervention, comparator, outcome) being the most frequently extracted. Data are available from 25 (33%), and code from 30 (39%) publications. Six (8%) implemented publicly available tools Conclusions: This living systematic review presents an overview of (semi)automated data-extraction literature of interest to different types of literature review. We identified a broad evidence base of publications describing data extraction for interventional reviews and a small number of publications extracting epidemiological or diagnostic accuracy data. Between review updates, trends for sharing data and code increased strongly: in the base-review, data and code were available for 13 and 19% respectively, these numbers increased to 78 and 87% within the 23 new publications. Compared with the base-review, we observed another research trend, away from straightforward data extraction and towards additionally extracting relations between entities or automatic text summarisation. With this living review we aim to review the literature continually

The impact of the COVID-19 pandemic on self-harm and suicidal behaviour: a living systematic review

Background: The COVID-19 pandemic has caused widespread morbidity and mortality as well as disruption to people’s lives and livelihoods around the world; this has occurred as a result of both infection with the virus itself and the health protection measures taken to curb its spread. There are concerns that rates of suicide, suicidal behaviours and self-harm may rise during and in the aftermath of the pandemic. Given the likely rapidly expanding research evidence base on the pandemic’s impact on rates of suicide, suicidal behaviours and self-harm and emerging evidence about how best to mitigate such effects, it is important that the best available knowledge is made readily available to policymakers, public health specialists and clinicians as soon as is possible. To facilitate this, we plan to undertake a living systematic review focusing on suicide prevention in relation to COVID-19.Method: Regular automated searches will feed into a web-based screening system which will also host the data extraction form for included articles. Our eligibility criteria are wide and include aspects of incidence and prevalence of suicidal behaviour, effects of exposures and effects of interventions in relation to the COVID-19 pandemic, with minimal restrictions on the types of study design to be included. The outcomes assessed will be death by suicide; self-harm or attempted suicide (including hospital attendance and/or admission for these reasons); and suicidal thoughts/ideation. There will be no restriction on study type, except for single case reports. There will be no restriction on language of publication. The review will be updated at three-monthly intervals if a sufficient volume of new evidence justifies doing so.Conclusions: Our living review will provide a regular synthesis of the most up-to-date research evidence to guide public health and clinical policy to mitigate the impact of COVID-19 on suicide.Protocol registration: PROSPERO CRD42020183326 01/05/202

The impact of the COVID-19 pandemic on self-harm and suicidal behaviour: update of living systematic review

Background: The COVID-19 pandemic has caused considerable morbidity, mortality and disruption to people’s lives around the world. There are concerns that rates of suicide and suicidal behaviour may rise during and in its aftermath. Our living systematic review synthesises findings from emerging literature on incidence and prevalence of suicidal behaviour as well as suicide prevention efforts in relation to COVID-19, with this iteration synthesising relevant evidence up to 19th October 2020.Method: Automated daily searches feed into a web-based database with screening and data extraction functionalities. Eligibility criteria include incidence/prevalence of suicidal behaviour, exposure-outcome relationships and effects of interventions in relation to the COVID-19 pandemic. Outcomes of interest are suicide, self-harm or attempted suicide and suicidal thoughts. No restrictions are placed on language or study type, except for single-person case reports. We exclude one-off cross-sectional studies without either pre-pandemic measures or comparisons of COVID-19 positive vs. unaffected individuals.Results: Searches identified 6,226 articles. Seventy-eight articles met our inclusion criteria. We identified a further 64 relevant cross-sectional studies that did not meet our revised inclusion criteria. Thirty-four articles were not peer-reviewed (e.g. research letters, pre-prints). All articles were based on observational studies.There was no consistent evidence of a rise in suicide but many studies noted adverse economic effects were evolving. There was evidence of a rise in community distress, fall in hospital presentation for suicidal behaviour and early evidence of an increased frequency of suicidal thoughts in those who had become infected with COVID-19.Conclusions: Research evidence of the impact of COVID-19 on suicidal behaviour is accumulating rapidly. This living review provides a regular synthesis of the most up-to-date research evidence to guide public health and clinical policy to mitigate the impact of COVID-19 on suicide risk as the longer term impacts of the pandemic on suicide risk are researched

Systematic review of the impact of the COVID-19 pandemic on suicidal behaviour amongst health and social care workers across the world

BackgroundThe COVID-19 pandemic has had an impact on the mental health of healthcare and social care workers, and its potential effect on suicidal thoughts and behaviour is of particular concern.MethodsThis systematic review identified and appraised the published literature that has reported on the impact of COVID-19 on suicidal thoughts and behaviour and self-harm amongst healthcare and social care workers worldwide up to May 31, 2021.ResultsOut of 37 potentially relevant papers identified, ten met our eligibility criteria. Our review has highlighted that the impact of COVID-19 has varied as a function of setting, working relationships, occupational roles, and psychiatric comorbidities.LimitationsThere have been no completed cohort studies comparing pre- and post-pandemic suicidal thoughts and behaviours. It is possible some papers may have been missed in the search.ConclusionsThe current quality of evidence pertaining to suicidal behaviour in healthcare workers is poor, and evidence is entirely absent for those working in social care. The clinical relevance of this work is to bring attention to what evidence exists, and to encourage, in practice, proactive approaches to interventions for improving healthcare and social care worker mental health

Summarizing the Results of a Series of Experiments : Application to the Effectiveness of Three Software Evaluation Techniques

Software quality has become and persistently remains a big issue among software users and developers. So, the importance of software evaluation cannot be overemphasized. An accepted fact in software engineering is that software must undergo evaluation process during development to ascertain and improve its quality level. In fact, there are too many techniques than a single developer could master, yet, it is impossible to be certain that software is free of defects. Therefore, it may not be realistic or cost effective to remove all software defects prior to product release. So, it is crucial for developers to be able to choose from available evaluation techniques, the one most suitable and likely to yield optimum quality results for different products - it bogs down to choosing the most appropriate for different situations. However, not much knowledge is available on the strengths and weaknesses of the available evaluation techniques. Most of the information related to the techniques available is focused on how to apply the techniques but not on the applicability conditions of the techniques – practical information, suitability, strengths, weaknesses etc. This research focuses on contributing to the available applicability knowledge of software evaluation techniques. More precisely, it focuses on code reading by stepwise abstraction as representative of the static technique, as well as equivalence partitioning (functional technique) and decision coverage (structural technique) as representatives of the dynamic technique. The specific focus of the research is to summarize the results of a series of experiments conducted to investigate the effectiveness of these techniques among other factors. By effectiveness in this research, we mean the potential of each of the techniques to generate test cases capable of revealing software faults in the case of the dynamic techniques or the ability of the static technique to generate abstractions that will aid the detection of faults. The experiments used two versions of three different programs with seven different faults seeded into each of the programs. This work uses the results of the eight different experiments performed and analyzed separately, to explore this fact. The analysis results were pooled together and jointly summarized in this research to extract a common knowledge from the experiments using a qualitative deduction approach created in this work as it was decided not to use formal aggregation at this stage. Since the experiments were performed by different researchers, in different years and in some cases at different site, there were several problems that have to be tackled in order to be able to summarize the results. Part of the problems is the fact that the data files exist in different languages, the structure of the files are different, different names is used for data fields, the analysis were done using different confidence level etc. The first step, taken at the inception of this research was to apply all the techniques to the programs used during the experiments in order to detect the faults. This purpose of this personal experience with the experiment is to be familiarized and get acquainted to the faults, failures, the programs and the experiment situations in general and also, to better understand the data as recorded from the experiments. Afterwards, the data files were recreated to conform to a uniform language, data meaning, file style and structure. A well structured directory was created to keep all the data, analysis and experiment files for all the experiments in the series. These steps paved the way for a feasible results synthesis. Using our method, the technique, program, fault, program – technique, program – fault and technique – fault were selected as main and interaction effects having significant knowledge relevant to the analysis summary result. The result, as reported in this thesis, indicated that the functional technique and the structural technique are equally effective as far as the programs and faults in these experiments are concerned. Both perform better than the code review. Also, the analysis revealed that the effectiveness of the techniques is influenced by the fault type and the program type. Some faults were found to exhibit better behavior with certain programs, some were better detected with certain techniques and even the techniques yield different result in different programs.I can alternatively be contacted through: [email protected]

Summarizing the Results of a Series of Experiments : Application to the Effectiveness of Three Software Evaluation Techniques

Software quality has become and persistently remains a big issue among software users and developers. So, the importance of software evaluation cannot be overemphasized. An accepted fact in software engineering is that software must undergo evaluation process during development to ascertain and improve its quality level. In fact, there are too many techniques than a single developer could master, yet, it is impossible to be certain that software is free of defects. Therefore, it may not be realistic or cost effective to remove all software defects prior to product release. So, it is crucial for developers to be able to choose from available evaluation techniques, the one most suitable and likely to yield optimum quality results for different products - it bogs down to choosing the most appropriate for different situations. However, not much knowledge is available on the strengths and weaknesses of the available evaluation techniques. Most of the information related to the techniques available is focused on how to apply the techniques but not on the applicability conditions of the techniques – practical information, suitability, strengths, weaknesses etc. This research focuses on contributing to the available applicability knowledge of software evaluation techniques. More precisely, it focuses on code reading by stepwise abstraction as representative of the static technique, as well as equivalence partitioning (functional technique) and decision coverage (structural technique) as representatives of the dynamic technique. The specific focus of the research is to summarize the results of a series of experiments conducted to investigate the effectiveness of these techniques among other factors. By effectiveness in this research, we mean the potential of each of the techniques to generate test cases capable of revealing software faults in the case of the dynamic techniques or the ability of the static technique to generate abstractions that will aid the detection of faults. The experiments used two versions of three different programs with seven different faults seeded into each of the programs. This work uses the results of the eight different experiments performed and analyzed separately, to explore this fact. The analysis results were pooled together and jointly summarized in this research to extract a common knowledge from the experiments using a qualitative deduction approach created in this work as it was decided not to use formal aggregation at this stage. Since the experiments were performed by different researchers, in different years and in some cases at different site, there were several problems that have to be tackled in order to be able to summarize the results. Part of the problems is the fact that the data files exist in different languages, the structure of the files are different, different names is used for data fields, the analysis were done using different confidence level etc. The first step, taken at the inception of this research was to apply all the techniques to the programs used during the experiments in order to detect the faults. This purpose of this personal experience with the experiment is to be familiarized and get acquainted to the faults, failures, the programs and the experiment situations in general and also, to better understand the data as recorded from the experiments. Afterwards, the data files were recreated to conform to a uniform language, data meaning, file style and structure. A well structured directory was created to keep all the data, analysis and experiment files for all the experiments in the series. These steps paved the way for a feasible results synthesis. Using our method, the technique, program, fault, program – technique, program – fault and technique – fault were selected as main and interaction effects having significant knowledge relevant to the analysis summary result. The result, as reported in this thesis, indicated that the functional technique and the structural technique are equally effective as far as the programs and faults in these experiments are concerned. Both perform better than the code review. Also, the analysis revealed that the effectiveness of the techniques is influenced by the fault type and the program type. Some faults were found to exhibit better behavior with certain programs, some were better detected with certain techniques and even the techniques yield different result in different programs.I can alternatively be contacted through: [email protected]