A question of trust: can we build an evidence base to gain trust in systematic review automation technologies?

Background Although many aspects of systematic reviews use computational tools, systematic reviewers have been reluctant to adopt machine learning tools. Discussion We discuss that the potential reason for the slow adoption of machine learning tools into systematic reviews is multifactorial. We focus on the current absence of trust in automation and set-up challenges as major barriers to adoption. It is important that reviews produced using automation tools are considered non-inferior or superior to current practice. However, this standard will likely not be sufficient to lead to widespread adoption. As with many technologies, it is important that reviewers see “others” in the review community using automation tools. Adoption will also be slow if the automation tools are not compatible with workflows and tasks currently used to produce reviews. Many automation tools being developed for systematic reviews mimic classification problems. Therefore, the evidence that these automation tools are non-inferior or superior can be presented using methods similar to diagnostic test evaluations, i.e., precision and recall compared to a human reviewer. However, the assessment of automation tools does present unique challenges for investigators and systematic reviewers, including the need to clarify which metrics are of interest to the systematic review community and the unique documentation challenges for reproducible software experiments. Conclusion We discuss adoption barriers with the goal of providing tool developers with guidance as to how to design and report such evaluations and for end users to assess their validity. Further, we discuss approaches to formatting and announcing publicly available datasets suitable for assessment of automation technologies and tools. Making these resources available will increase trust that tools are non-inferior or superior to current practice. Finally, we identify that, even with evidence that automation tools are non-inferior or superior to current practice, substantial set-up challenges remain for main stream integration of automation into the systematic review process

Integration of decision support systems to improve decision support performance

Decision support system (DSS) is a well-established research and development area. Traditional isolated, stand-alone DSS has been recently facing new challenges. In order to improve the performance of DSS to meet the challenges, research has been actively carried out to develop integrated decision support systems (IDSS). This paper reviews the current research efforts with regard to the development of IDSS. The focus of the paper is on the integration aspect for IDSS through multiple perspectives, and the technologies that support this integration. More than 100 papers and software systems are discussed. Current research efforts and the development status of IDSS are explained, compared and classified. In addition, future trends and challenges in integration are outlined. The paper concludes that by addressing integration, better support will be provided to decision makers, with the expectation of both better decisions and improved decision making processes

Knowledge Cartography: Software tools and mapping techniques

Knowledge Cartography is the discipline of mapping intellectual landscapes.The focus of this book is on the process by which manually crafting interactive, hypertextual maps clarifies one’s own understanding, as well as communicating it.The authors see mapping software as a set of visual tools for reading and writing in a networked age. In an information ocean, the primary challenge is to find meaningful patterns around which we can weave plausible narratives. Maps of concepts, discussions and arguments make the connections between ideas tangible and disputable. With 17 chapters from the leading researchers and practitioners, the reader will find the current state–of-the-art in the field. Part 1 focuses on educational applications in schools and universities, before Part 2 turns to applications in professional communitie

A model-driven method for the systematic literature review of qualitative empirical research

This paper explores a model-driven method for systematic literature reviews (SLRs), for use where the empirical studies found in the literature search are based on qualitative research. SLRs are an important component of the evidence-based practice (EBP) paradigm, which is receiving increasing attention in information systems (IS) but has not yet been widely-adopted. We illustrate the model-driven approach to SLRs via an example focused on the use of BPMN (Business Process Modelling Notation) in organizations. We discuss in detail the process followed in using the model-driven SLR method, and show how it is based on a hermeneutic cycle of reading and interpreting, in order to develop and refine a model which synthesizes the research findings of previous qualitative studies. This study can serve as an exemplar for other researchers wishing to carry out model-driven SLRs. We conclude with our reflections on the method and some suggestions for further researc

Testing in the incremental design and development of complex products

Testing is an important aspect of design and development which consumes significant time and resource in many companies. However, it has received less research attention than many other activities in product development, and especially, very few publications report empirical studies of engineering testing. Such studies are needed to establish the importance of testing and inform the development of pragmatic support methods. This paper combines insights from literature study with findings from three empirical studies of testing. The case studies concern incrementally developed complex products in the automotive domain. A description of testing practice as observed in these studies is provided, confirming that testing activities are used for multiple purposes depending on the context, and are intertwined with design from start to finish of the development process, not done after it as many models depict. Descriptive process models are developed to indicate some of the key insights, and opportunities for further research are suggested

SMEs: ERP or virtual collaboration teams

Small firms are indeed the engines of global economic growth. Small and Medium Enterprises (SMEs) play an important role to promote economic development. SMEs in the beginning of implementing new technologies always face capital shortage and need technological assistance. Available ERP systems do not fulfil the specific requirements of Small firms. SMEs has scarce resources and manpower therefore many SMEs don?t have the possessions to buy and operate an ERP System. On the other hand competition and competitiveness of SMEs have to be strengthened. This paper briefly reviews the existing perspectives on virtual teams and their effect on SMEs management. It also discusses the main characteristics of virtual teams and clarifies the differences aspects of virtual team application in SMEs. After outlining some of the main advantages and pitfall of such teams, it concentrates on comparing of ERP and virtual collaborative teams in SMEs. Finally, it provides evidence for the need of ?Software as a Service (SaaS)? where an application is hosted as a service provided to customers across the web for SMEs as an alternative of ERP. It has been widely argued that ERP disadvantage in SMEs such as administrative expenditure and cost, isolated structure, severe lack of software flexibility, insufficient support of SMEs business and high operating cost, lead SMEs to use virtual collaborative team which is net work base solution

Expert Elicitation for Reliable System Design

This paper reviews the role of expert judgement to support reliability assessments within the systems engineering design process. Generic design processes are described to give the context and a discussion is given about the nature of the reliability assessments required in the different systems engineering phases. It is argued that, as far as meeting reliability requirements is concerned, the whole design process is more akin to a statistical control process than to a straightforward statistical problem of assessing an unknown distribution. This leads to features of the expert judgement problem in the design context which are substantially different from those seen, for example, in risk assessment. In particular, the role of experts in problem structuring and in developing failure mitigation options is much more prominent, and there is a need to take into account the reliability potential for future mitigation measures downstream in the system life cycle. An overview is given of the stakeholders typically involved in large scale systems engineering design projects, and this is used to argue the need for methods that expose potential judgemental biases in order to generate analyses that can be said to provide rational consensus about uncertainties. Finally, a number of key points are developed with the aim of moving toward a framework that provides a holistic method for tracking reliability assessment through the design process.Comment: This paper commented in: [arXiv:0708.0285], [arXiv:0708.0287], [arXiv:0708.0288]. Rejoinder in [arXiv:0708.0293]. Published at http://dx.doi.org/10.1214/088342306000000510 in the Statistical Science (http://www.imstat.org/sts/) by the Institute of Mathematical Statistics (http://www.imstat.org

Management issues in systems engineering

When applied to a system, the doctrine of successive refinement is a divide-and-conquer strategy. Complex systems are sucessively divided into pieces that are less complex, until they are simple enough to be conquered. This decomposition results in several structures for describing the product system and the producing system. These structures play important roles in systems engineering and project management. Many of the remaining sections in this chapter are devoted to describing some of these key structures. Structures that describe the product system include, but are not limited to, the requirements tree, system architecture and certain symbolic information such as system drawings, schematics, and data bases. The structures that describe the producing system include the project's work breakdown, schedules, cost accounts and organization