Vulnerability Prediction Capability: A Comparison between Vulnerability Discovery Models and Neural Network Models

In this paper, we introduce an approach for predicting the cumulative number of software vulnerabilities that is in most cases more accurate than vulnerability discovery models (VDMs). Our approach uses a neural network model (NNM) to model the nonlinearities associated with vulnerability disclosure. Nine common VDMs were used to compare their prediction capability with our approach. The different models were applied to vulnerabilities associated with eight well-known software (four operating systems and four web browsers). The models were assessed in terms of prediction accuracy and prediction bias. Out of eight software we analyzed, the NNM outperformed the VDMs in all the cases in terms of prediction accuracy, and provided smaller values of absolute average bias in seven cases. This study shows that NNMs are promising for accurate predictions of software vulnerabilities disclosures

Empirical evaluation of defect projection models for widely-deployed production software systems

Defect-occurrence projection is necessary for the development of methods to mitigate the risks of software defect occurrences. In this paper, we examine user-reported software defectoccurrence patterns across twenty-two releases of four widelydeployed, business-critical, production, software systems: a commercial operating system, a commercial middleware system, an open source operating system (OpenBSD), and an open source middleware system (Tomcat). We evaluate the suitability of common defect-occurrence models by first assessing the match between characteristics of widely-deployed production software systems and model structures. We then evaluate how well the models fit real world data. We find that the Weibull model is flexible enough to capture defect-occurrence behavior across a wide range of systems. It provides the best model fit in 16 out of the 22 releases. We then evaluate the ability of the moving averages and the exponential smoothing methods to extrapolate Weibull model parameters using fitted model parameters from historical releases. Our results show that in 50 % of our forecasting experiments, these two naïve parameterextrapolation methods produce projections that are worse than the projection from using the same model parameters as the most recent release. These findings establish the need for further research on parameter-extrapolation methods that take into account variations in characteristics of widely-deployed, production, software systems across multiple releases

Empirical evaluation of defect projection models for widely-deployed production software systems

Defect-occurrence projection is necessary for the development of methods to mitigate the risks of software defect occurrences. In this paper, we examine user-reported software defectoccurrence patterns across twenty-two releases of four widelydeployed, business-critical, production, software systems: a commercial operating system, a commercial middleware system, an open source operating system (OpenBSD), and an open source middleware system (Tomcat). We evaluate the suitability of common defect-occurrence models by first assessing the match between characteristics of widely-deployed production software systems and model structures. We then evaluate how well the models fit real world data. We find that the Weibull model is flexible enough to capture defect-occurrence behavior across a wide range of systems. It provides the best model fit in 16 out of the 22 releases. We then evaluate the ability of the moving averages and the exponential smoothing methods to extrapolate Weibull model parameters using fitted model parameters from historical releases. Our results show that in 50 % of our forecasting experiments, these two naïve parameterextrapolation methods produce projections that are worse than the projection from using the same model parameters as the most recent release. These findings establish the need for further research on parameter-extrapolation methods that take into account variations in characteristics of widely-deployed, production, software systems across multiple releases

Using Time Series Models for Defect Prediction in Software Release Planning

To produce a high-quality software release, sufficient time should be allowed for testing and fixing defects. Otherwise, there is a risk of slip in the development schedule and/or software quality. A time series model is used to predict the number of bugs created during development. The model depends on the previous numbers of bugs created. The model also depends, in an exogenous manner, on the previous numbers of new features resolved and improvements resolved. This model structure would allow hypothetical release plans to be compared by assessing their predicted impact on testing and defect- fixing time. The VARX time series model was selected as a reasonable approach. The accuracy of the model varies for different sampling periods, window sizes, and degree of differencing

Change decision support:extraction and analysis of late architecture changes using change characterization and software metrics

Software maintenance is one of the most crucial aspects of software development. Software engineering researchers must develop practical solutions to handle the challenges presented in maintaining mature software systems. Research that addresses practical means of mitigating the risks involved when changing software, reducing the complexity of mature software systems, and eliminating the introduction of preventable bugs is paramount to today’s software engineering discipline. Giving software developers the information that they need to make quality decisions about changes that will negatively affect their software systems is a key aspect to mitigating those risks. This dissertation presents work performed to assist developers to collect and process data that plays a role in change decision-making during the maintenance phase. To address these problems, developers need a way to better understand the effects of a change prior to making the change. This research addresses the problems associated with increasing architectural complexity caused by software change using a twoold approach. The first approach is to characterize software changes to assess their architectural impact prior to their implementation. The second approach is to identify a set of architecture metrics that correlate to system quality and maintainability and to use these metrics to determine the level of difficulty involved in making a change. The two approaches have been combined and the results presented provide developers with a beneficial analysis framework that offers insight into the change process

Some Guidelines for Risk Assessment of Vulnerability Discovery Processes

Software vulnerabilities can be defined as software faults, which can be exploited as results of security attacks. Security researchers have used data from vulnerability databases to study trends of discovery of new vulnerabilities or propose models for fitting the discovery times and for predicting when new vulnerabilities may be discovered. Estimating the discovery times for new vulnerabilities is useful both for vendors as well as the end-users as it can help with resource allocation strategies over time. Among the research conducted on vulnerability modeling, only a few studies have tried to provide a guideline about which model should be used in a given situation. In other words, assuming the vulnerability data for a software is given, the research questions are the following: Is there any feature in the vulnerability data that could be used for identifying the most appropriate models for that dataset? What models are more accurate for vulnerability discovery process modeling? Can the total number of publicly-known exploited vulnerabilities be predicted using all vulnerabilities reported for a given software? To answer these questions, we propose to characterize the vulnerability discovery process using several common software reliability/vulnerability discovery models, also known as Software Reliability Models (SRMs)/Vulnerability Discovery Models (VDMs). We plan to consider different aspects of vulnerability modeling including curve fitting and prediction. Some existing SRMs/VDMs lack accuracy in the prediction phase. To remedy the situation, three strategies are considered: (1) Finding a new approach for analyzing vulnerability data using common models. In other words, we examine the effect of data manipulation techniques (i.e. clustering, grouping) on vulnerability data, and investigate whether it leads to more accurate predictions. (2) Developing a new model that has better curve filling and prediction capabilities than current models. (3) Developing a new method to predict the total number of publicly-known exploited vulnerabilities using all vulnerabilities reported for a given software. The dissertation is intended to contribute to the science of software reliability analysis and presents some guidelines for vulnerability risk assessment that could be integrated as part of security tools, such as Security Information and Event Management (SIEM) systems