4 research outputs found
The Importance of Accounting for Real-World Labelling When Predicting Software Vulnerabilities
Previous work on vulnerability prediction assume that predictive models are trained with respect to perfect labelling information (includes labels from future, as yet undiscovered vulnerabilities). In this paper we present results from a comprehensive empirical study of 1,898 real-world vulnerabilities reported in 74 releases of three security-critical open source systems (Linux Kernel, OpenSSL and Wiresark). Our study investigates the effectiveness of three previously proposed vulnerability prediction approaches, in two settings: with and without the unrealistic labelling assumption. The results reveal that the unrealistic labelling assumption can profoundly mis- lead the scientific conclusions drawn; suggesting highly effective and deployable prediction results vanish when we fully account for realistically available labelling in the experimental methodology. More precisely, MCC mean values of predictive effectiveness drop from 0.77, 0.65 and 0.43 to 0.08, 0.22, 0.10 for Linux Kernel, OpenSSL and Wiresark, respectively. Similar results are also obtained for precision, recall and other assessments of predictive efficacy. The community therefore needs to upgrade experimental and empirical methodology for vulnerability prediction evaluation and development to ensure robust and actionable scientific findings
Customer Rating Reactions Can Be Predicted Purely Using App Features
In this paper we provide empirical evidence that the rating that an app attracts can be accurately predicted from the features it offers. Our results, based on an analysis of 11,537 apps from the Samsung Android and BlackBerry World app stores, indicate that the rating of 89% of these apps can be predicted with 100% accuracy. Our prediction model is built by using feature and rating information from the existing apps offered in the App Store and it yields highly accurate rating predictions, using only a few (11-12) existing apps for case-based prediction. These findings may have important implications for require- ments engineering in app stores: They indicate that app devel- opers may be able to obtain (very accurate) assessments of the customer reaction to their proposed feature sets (requirements), thereby providing new opportunities to support the requirements elicitation process for app developers
How to Evaluate Solutions in Pareto-based Search-Based Software Engineering? A Critical Review and Methodological Guidance
With modern requirements, there is an increasing tendency of considering
multiple objectives/criteria simultaneously in many Software Engineering (SE)
scenarios. Such a multi-objective optimization scenario comes with an important
issue -- how to evaluate the outcome of optimization algorithms, which
typically is a set of incomparable solutions (i.e., being Pareto non-dominated
to each other). This issue can be challenging for the SE community,
particularly for practitioners of Search-Based SE (SBSE). On one hand,
multi-objective optimization could still be relatively new to SE/SBSE
researchers, who may not be able to identify the right evaluation methods for
their problems. On the other hand, simply following the evaluation methods for
general multi-objective optimization problems may not be appropriate for
specific SE problems, especially when the problem nature or decision maker's
preferences are explicitly/implicitly available. This has been well echoed in
the literature by various inappropriate/inadequate selection and
inaccurate/misleading use of evaluation methods. In this paper, we first carry
out a systematic and critical review of quality evaluation for multi-objective
optimization in SBSE. We survey 717 papers published between 2009 and 2019 from
36 venues in seven repositories, and select 95 prominent studies, through which
we identify five important but overlooked issues in the area. We then conduct
an in-depth analysis of quality evaluation indicators/methods and general
situations in SBSE, which, together with the identified issues, enables us to
codify a methodological guidance for selecting and using evaluation methods in
different SBSE scenarios.Comment: This paper has been accepted by IEEE Transactions on Software
Engineering, available as full OA:
https://ieeexplore.ieee.org/document/925218