Evolution of statistical analysis in empirical software engineering research: Current state and steps forward

Software engineering research is evolving and papers are increasingly based on empirical data from a multitude of sources, using statistical tests to determine if and to what degree empirical evidence supports their hypotheses. To investigate the practices and trends of statistical analysis in empirical software engineering (ESE), this paper presents a review of a large pool of papers from top-ranked software engineering journals. First, we manually reviewed 161 papers and in the second phase of our method, we conducted a more extensive semi-automatic classification of papers spanning the years 2001--2015 and 5,196 papers. Results from both review steps was used to: i) identify and analyze the predominant practices in ESE (e.g., using t-test or ANOVA), as well as relevant trends in usage of specific statistical methods (e.g., nonparametric tests and effect size measures) and, ii) develop a conceptual model for a statistical analysis workflow with suggestions on how to apply different statistical methods as well as guidelines to avoid pitfalls. Lastly, we confirm existing claims that current ESE practices lack a standard to report practical significance of results. We illustrate how practical significance can be discussed in terms of both the statistical analysis and in the practitioner's context.Comment: journal submission, 34 pages, 8 figure

Alternative causal inference methods in population health research: Evaluating tradeoffs and triangulating evidence.

Population health researchers from different fields often address similar substantive questions but rely on different study designs, reflecting their home disciplines. This is especially true in studies involving causal inference, for which semantic and substantive differences inhibit interdisciplinary dialogue and collaboration. In this paper, we group nonrandomized study designs into two categories: those that use confounder-control (such as regression adjustment or propensity score matching) and those that rely on an instrument (such as instrumental variables, regression discontinuity, or differences-in-differences approaches). Using the Shadish, Cook, and Campbell framework for evaluating threats to validity, we contrast the assumptions, strengths, and limitations of these two approaches and illustrate differences with examples from the literature on education and health. Across disciplines, all methods to test a hypothesized causal relationship involve unverifiable assumptions, and rarely is there clear justification for exclusive reliance on one method. Each method entails trade-offs between statistical power, internal validity, measurement quality, and generalizability. The choice between confounder-control and instrument-based methods should be guided by these tradeoffs and consideration of the most important limitations of previous work in the area. Our goals are to foster common understanding of the methods available for causal inference in population health research and the tradeoffs between them; to encourage researchers to objectively evaluate what can be learned from methods outside one's home discipline; and to facilitate the selection of methods that best answer the investigator's scientific questions

Challenges in impact evaluation of development interventions: opportunities and limitations for randomized experiments

In recent years debates on as well as funding of impact evaluations of development interventions have flourished. Unfortunately, controversy regarding the promotion and application of randomized experiments (RE) has led to a sense of polarization in the development policy and evaluation community. As some proponents claim epistemological supremacy of REs (with respect to attribution) the counter reaction among others has been rejection. Needless to say, such extreme positions are counterproductive to reaching a goal that is commonly endorsed: to learn more about what works and why in development. This paper discusses the prospects and limitations of REs from the perspective of three categories of challenges in impact evaluation: delimitation and scope, attribution versus explanation, and implementation challenges. The implicit lesson is twofold. First of all, the question ‘to randomize or not to randomize’ is overrated in the current debate. Limitations in scope, applicability as well as implementation will necessarily restrict the use of REs in development impact evaluation. There is a risk that the current popularity of REs in certain research and policy circles might lead to a backlash as too high expectations of REs may quicken its demise. More importantly, given the nature and scope of the challenges discussed in the paper, more energy should be devoted to developing and testing ‘rigorous’ mixed method approaches within a framework of theory-driven evaluation.

What is the evidence of the impact of microfinance on the well-being of poor people?

The concept of microcredit was first introduced in Bangladesh by Nobel Peace Prize winner Muhammad Yunus. Professor Yunus started Grameen Bank (GB) more than 30 years ago with the aim of reducing poverty by providing small loans to the country’s rural poor (Yunus 1999). Microcredit has evolved over the years and does not only provide credit to the poor, but also now spans a myriad of other services including savings, insurance, remittances and non-financial services such as financial literacy training and skills development programmes; microcredit is now referred to as microfinance (Armendáriz de Aghion and Morduch 2005, 2010). A key feature of microfinance has been the targeting of women on the grounds that, compared to men, they perform better as clients of microfinance institutions and that their participation has more desirable development outcomes (Pitt and Khandker 1998). Despite the apparent success and popularity of microfinance, no clear evidence yet exists that microfinance programmes have positive impacts (Armendáriz de Aghion and Morduch 2005, 2010; and many others). There have been four major reviews examining impacts of microfinance (Sebstad and Chen, 1996; Gaile and Foster 1996, Goldberg 2005, Odell 2010, see also Orso 2011). These reviews concluded that, while anecdotes and other inspiring stories (such as Todd 1996) purported to show that microfinance can make a real difference in the lives of those served, rigorous quantitative evidence on the nature, magnitude and balance of microfinance impact is still scarce and inconclusive (Armendáriz de Aghion and Morduch 2005, 2010). Overall, it is widely acknowledged that no well-known study robustly shows any strong impacts of microfinance (Armendáriz de Aghion and Morduch 2005, p199-230). Because of the growth of the microfinance industry and the attention the sector has received from policy makers, donors and private investors in recent years, existing microfinance impact evaluations need to be re-investigated; the robustness of claims that microfinance successfully alleviates poverty and empowers women must be scrutinised more carefully. Hence, this review revisits the evidence of microfinance evaluations focusing on the technical challenges of conducting rigorous microfinance impact evaluations

Case Studies in Industry: What We Have Learnt

Case study research has become an important research methodology for exploring phenomena in their natural contexts. Case studies have earned a distinct role in the empirical analysis of software engineering phenomena which are difficult to capture in isolation. Such phenomena often appear in the context of methods and development processes for which it is difficult to run large, controlled experiments as they usually have to reduce the scale in several respects and, hence, are detached from the reality of industrial software development. The other side of the medal is that the realistic socio-economic environments where we conduct case studies -- with real-life cases and realistic conditions -- also pose a plethora of practical challenges to planning and conducting case studies. In this experience report, we discuss such practical challenges and the lessons we learnt in conducting case studies in industry. Our goal is to help especially inexperienced researchers facing their first case studies in industry by increasing their awareness for typical obstacles they might face and practical ways to deal with those obstacles.Comment: Proceedings of the 4th International Workshop on Conducting Empirical Studies in Industry, co-located with ICSE, 201

Ontology-driven conceptual modeling: A'systematic literature mapping and review

All rights reserved. Ontology-driven conceptual modeling (ODCM) is still a relatively new research domain in the field of information systems and there is still much discussion on how the research in ODCM should be performed and what the focus of this research should be. Therefore, this article aims to critically survey the existing literature in order to assess the kind of research that has been performed over the years, analyze the nature of the research contributions and establish its current state of the art by positioning, evaluating and interpreting relevant research to date that is related to ODCM. To understand and identify any gaps and research opportunities, our literature study is composed of both a systematic mapping study and a systematic review study. The mapping study aims at structuring and classifying the area that is being investigated in order to give a general overview of the research that has been performed in the field. A review study on the other hand is a more thorough and rigorous inquiry and provides recommendations based on the strength of the found evidence. Our results indicate that there are several research gaps that should be addressed and we further composed several research opportunities that are possible areas for future research

An Examination of User Detection of Business Email Compromise Amongst Corporate Professionals

With the evolution in technology and increase in utilization of the public Internet, Internet-based mobile applications, and social media, security risks for organizations have greatly increased. While corporations leverage social media as an effective tool for customer advertisements, the abundance of information available via public channels along with the growth in Internet connections to corporate networks including mobile applications, have made cyberattacks attractive for cybercriminals. Cybercrime against organizations is a daily threat and targeting companies of all sizes. Cyberattacks are continually evolving and becoming more complex that make it difficult to protect against with traditional security methods. Cybercriminals utilize email attacks as their most common method to compromise corporations for financial gain. Email attacks on corporations have evolved into very sophisticated scams that specifically target businesses that conduct wire transfers or financial transactions as part of their standard mode of operations. This new evolution of email driven attacks is called Business Email Compromise (BEC) attacks and utilize advanced social engineering, phishing techniques, and email hacking to manipulate employees into conducting fraudulent wire transfers that are intended for actual suppliers and business partners. One of the most common types of BEC attacks is the Chief Executive Officer (CEO) fraud, which are highly customized and targeted attacks aimed to impersonate corporate users that have authority to approve financial transactions and wire transfers in order to influence an employee to unknowingly conduct a fraudulent financial wire transfer. Thus, the main goal of this research study was to assess if there are any significant differences of corporate users’ detection skills of BEC attacks in a simulated test environment based on their personality attributes, using the Myers-Briggs Type Indicator® (MBTI®)’ 16 personalities® framework. BEC attacks have attributed to over $26 billion in corporate financial losses across the globe and are continually increasing. The human aspect in the cybersecurity has been a known challenge and is especially significant in direct interaction with BEC attacks. Furthermore, this research study analyzed corporate users’ attention span levels and demographics to assess if there are any significant differences on corporate users’ BEC attack detection skills. Moreover, this research study analyzed if there are any significant differences for BEC detection skills before and after a BEC awareness training. This research study was conducted by first developing an experiment to measure BEC detection and ensure validity via cybersecurity subject matter experts using the Delphi process. The experiment also collected qualitative and quantitative data for the participants’ performance measures using an application developed for the study. This research was conducted on a group of 45 corporate users in an experimental setting utilizing online surveys and a BEC detection mobile test application. This research validated and developed a BEC detection measure as well as the BEC awareness training module that were utilized in the research experiment. The results of the experiments were analyzed using analysis of variance (ANOVA) and analysis of covariance (ANCOVA) to address the research questions. It was found that there were that no statistically significant mean differences for Business Email Compromise Detection (BECD) skills between personality attributes of corporate professional participants, However, results indicated that there was a significant mean difference for BECD skills and span attention with a p\u3c.0001. Furthermore, there was a significant mean difference for BECD skills and span attention when controlled for gender with a p\u3c0.05. Furthermore, the results indicated that the BEC detection awareness training significantly improved the participant BEC detection skill with a p\u3c.0001. Moreover, following the training, it was found that female BEC detection test scores improved by 45% where the men BECD score improved by 31%. Recommendations for research and industry stakeholders are provided, including to corporations on methods to mitigate BEC attacks