The Importance of Accounting for Real-World Labelling When Predicting Software Vulnerabilities

Previous work on vulnerability prediction assume that predictive models are trained with respect to perfect labelling information (includes labels from future, as yet undiscovered vulnerabilities). In this paper we present results from a comprehensive empirical study of 1,898 real-world vulnerabilities reported in 74 releases of three security-critical open source systems (Linux Kernel, OpenSSL and Wiresark). Our study investigates the effectiveness of three previously proposed vulnerability prediction approaches, in two settings: with and without the unrealistic labelling assumption. The results reveal that the unrealistic labelling assumption can profoundly mis- lead the scientific conclusions drawn; suggesting highly effective and deployable prediction results vanish when we fully account for realistically available labelling in the experimental methodology. More precisely, MCC mean values of predictive effectiveness drop from 0.77, 0.65 and 0.43 to 0.08, 0.22, 0.10 for Linux Kernel, OpenSSL and Wiresark, respectively. Similar results are also obtained for precision, recall and other assessments of predictive efficacy. The community therefore needs to upgrade experimental and empirical methodology for vulnerability prediction evaluation and development to ensure robust and actionable scientific findings

The Importance of Accounting for Real-World Labelling When Predicting Software Vulnerabilities

Previous work on vulnerability prediction assume that predictive models are trained with respect to perfect labelling information (includes labels from future, as yet undiscovered vulnerabilities). In this paper we present results from a comprehensive empirical study of 1,898 real-world vulnerabilities reported in 74 releases of three security-critical open source systems (Linux Kernel, OpenSSL and Wiresark). Our study investigates the effectiveness of three previously proposed vulnerability prediction approaches, in two settings: with and without the unrealistic labelling assumption. The results reveal that the unrealistic labelling assumption can profoundly mis- lead the scientific conclusions drawn; suggesting highly effective and deployable prediction results vanish when we fully account for realistically available labelling in the experimental methodology. More precisely, MCC mean values of predictive effectiveness drop from 0.77, 0.65 and 0.43 to 0.08, 0.22, 0.10 for Linux Kernel, OpenSSL and Wiresark, respectively. Similar results are also obtained for precision, recall and other assessments of predictive efficacy. The community therefore needs to upgrade experimental and empirical methodology for vulnerability prediction evaluation and development to ensure robust and actionable scientific findings

Agile Effort Estimation: Have We Solved the Problem Yet? Insights From A Replication Study

In the last decade, several studies have explored automated techniques to estimate the effort of agile software development. We perform a close replication and extension of a seminal work proposing the use of Deep Learning for Agile Effort Estimation (namely Deep-SE), which has set the state-of-the-art since. Specifically, we replicate three of the original research questions aiming at investigating the effectiveness of Deep-SE for both within-project and cross-project effort estimation. We benchmark Deep-SE against three baselines (i.e., Random, Mean and Median effort estimators) and a previously proposed method to estimate agile software project development effort (dubbed TF/IDF-SVM), as done in the original study. To this end, we use the data from the original study and an additional dataset of 31,960 issues mined from TAWOS, as using more data allows us to strengthen the confidence in the results, and to further mitigate external validity threats. The results of our replication show that Deep-SE outperforms the Median baseline estimator and TF/IDF-SVM in only very few cases with statistical significance (8/42 and 9/32 cases, respectively), thus confounding previous findings on the efficacy of Deep-SE. The two additional RQs revealed that neither augmenting the training set nor pre-training Deep-SE play lead to an improvement of its accuracy and convergence speed. These results suggest that using semantic similarity is not enough to differentiate user stories with respect to their story points; thus, future work has yet to explore and find new techniques and features that obtain accurate agile software development estimates

Aplicación de herramientas de hacking ético para reducir el grado de vulnerabilidad en el sistema web informativo de una pyme - Piura, 2021

El presente trabajo de investigación tuvo como objetivo principal evaluar la reducción de vulnerabilidades en el sitio web de una pyme de matizados de Piura con la aplicación de hacking ético, haciendo uso de sus herramientas que permiten detectar, analizar y evaluar las posibles vulnerabilidades en el sitio web de una determinada empresa. La presente investigación fue de tipo aplicada y de nivel descriptivo, dado que se van a describir las vulnerabilidades, amenazas y riesgos de un sitio web. Como población de estudio se tuvo al sitio web de la empresa, tiene un muestreo no probabilístico intencional ya que se seleccionó 1 sitio web de las pymes de Piura para aplicar las herramientas de hacking ético. Además, se aplicó la prueba testretest, la cual consiste en aplicar la prueba reiteradas veces para verificar la confiabilidad al instrumento de recolección de datos. Como resultado de la investigación se logró reducir las vulnerabilidades de nivel bajo en un 9 % con la aplicación de las herramientas de hacking ético. Este estudio de investigación concluye que las herramientas de hacking ético ayudan a reducir considerablemente las vulnerabilidades en los sitios web, ya que permiten analizar y visualizar los errores que se presentan en un sitio web