Bayesian Graphical Models for Software Testing

This paper describes a new approach to the problem of software testing. The approach is based on Bayesian graphical models and presents formal mechanisms for the logical structuring of the software testing problem, the probabilistic and statistical treatment of the uncertainties to be addressed, the test design and analysis process, and the incorporation and implication of test results. Once constructed, the models produced are dynamic representations of the software testing problem. They may be used to drive test design, answer what-if questions, and provide decision support to managers and testers. The models capture the knowledge of the software tester for further use. Experiences of the approach in case studies are briefly discusse

Bayesian Networks and Gaussian Mixture Models in Multi-Dimensional Data Analysis with Application to Religion-Conflict Data

abstract: This thesis examines the application of statistical signal processing approaches to data arising from surveys intended to measure psychological and sociological phenomena underpinning human social dynamics. The use of signal processing methods for analysis of signals arising from measurement of social, biological, and other non-traditional phenomena has been an important and growing area of signal processing research over the past decade. Here, we explore the application of statistical modeling and signal processing concepts to data obtained from the Global Group Relations Project, specifically to understand and quantify the effects and interactions of social psychological factors related to intergroup conflicts. We use Bayesian networks to specify prospective models of conditional dependence. Bayesian networks are determined between social psychological factors and conflict variables, and modeled by directed acyclic graphs, while the significant interactions are modeled as conditional probabilities. Since the data are sparse and multi-dimensional, we regress Gaussian mixture models (GMMs) against the data to estimate the conditional probabilities of interest. The parameters of GMMs are estimated using the expectation-maximization (EM) algorithm. However, the EM algorithm may suffer from over-fitting problem due to the high dimensionality and limited observations entailed in this data set. Therefore, the Akaike information criterion (AIC) and the Bayesian information criterion (BIC) are used for GMM order estimation. To assist intuitive understanding of the interactions of social variables and the intergroup conflicts, we introduce a color-based visualization scheme. In this scheme, the intensities of colors are proportional to the conditional probabilities observed.Dissertation/ThesisM.S. Electrical Engineering 201

Computational Algorithm for Dynamic Hybrid Bayesian Network in On-line System Health Management Applications

With the increasing complexity of today's engineering systems that contain various component dependencies and degradation behaviors, there has been increasing interest in on-line System Health Management (SHM) capability to continuously monitor (via sensors and other methods of observation) system software, and hardware components for detection and diagnostic of safety-critical systems. Bayesian Network (BN) and their extension for time-series modeling known as Dynamic Bayesian Network (DBN) have been shown by recent studies to be capable of providing a unified framework for system health diagnosis and prognosis. BN has many modeling features, such as multi-state variables, noisy gates, dependent failures, and general posterior analysis. BN also allows a compact representation of the temporal and functional dependencies among system components. However, one of the barriers to applying BN in real-world problems is limitation in adequately handle "hybrid models", which contain both discrete and continuous variables, with both static and time-dependent failure distributions. This research presents a new modeling approach, computational algorithm, and an example application for health monitoring and learning in on-line SHM. A hybrid DBN is introduced to represent complex engineering systems with underlying physics of failure by modeling a theoretical or empirical degradation model with continuous variables. The methodology is designed to be flexible and intuitive, and scalable from small, localized functionality to large complex dynamic systems. Markov Chain Monte Carlo (MCMC) inference is optimized using a pre-computation strategy and dynamic programming for on-line monitoring of system health. Proposed Monitoring and Anomaly Detection algorithm uses pattern recognition to improve failure detection and estimation of Remaining Useful Life (RUL). Pre-computation inference database enables efficient on-line learning and maintenance decision-making. The scope of this research includes a new modeling approach, computation algorithm, and an example application for on-line SHM

Learning to Identify Bugs in Video Games

The use of intelligent software agents promises to revolutionise video game testing. While agents automate the time-consuming task of repeatedly playing a game in search of issues, humans can spend their time on the more creative aspects of game development. Despite the substantial advancements in game-playing that have made this possible, agents are reliant on humans, or hand-crafted guards, to determine whether there are issues with the game's design or functioning. This thesis aimed to develop testing agents that can identify issues with a game's function or bugs with minimal human involvement by learning from their prior experiences. The problem is framed as one of anomaly detection, where bugs correspond to abnormality or novelty in an agent's experience. A series of approaches based on Self-Supervised Learning and Causal Inference have been developed to enable an agent to measure abnormality or otherwise model the game to subsequently identify bugs. The focus was on laying the foundations for testing agents that operate over the same input/output modalities as human testers. The approaches were evaluated by testing a diverse collection of purpose-built video games, where they successfully identified bugs from a broad class. This thesis is among the first work to investigate the use of machine learning in the context of video game bug identification. It presents an exposition of the problem of learning intended behaviour, and then endeavours to develop solutions that demonstrate the benefits of using agents with learning capabilities for testing. Namely, ease of reuse across projects (reusability) and in identifying bugs that would otherwise require human involvement to be found (capability). The use of agents equipped with sophisticated game-playing algorithms and the identification tools outlined in this thesis offers a new framework for video game testing

Intégration du web sémantique dans un système d'aide à la décision pour le génie logiciel

Avoir à sa disposition des données et des connaissances et savoir à quoi elles servent c'est bien, savoir s'en servir c'est encore mieux. La qualité est un critère recherché dans tous les domaines. Dans des domaines qui font allusion aux objets matériels, il est facile de définir, d'observer et de savoir comment obtenir un produit fini de bonne qualité. Dans le domaine du génie logiciel, il est bien plus difficile de définir et d'observer la qualité d'un produit. On fait appel aux métriques, aux normes de qualité, aux modèles de qualité, etc., pour pouvoir déterminer, évaluer et améliorer la qualité d'un logiciel. Les résultats des études empiriques et les connaissances des experts à ce sujet ne sont malheureusement pas partagés avec tous les acteurs du domaine, ce qui entraine des interprétations différentes, la répétition des études ou l'ignorance de certains faits importants pour produire un logiciel de qualité. Dans le souci de permettre et faciliter le partage des connaissances et des données sur la qualité logicielle, nous avons exploité ce que le web sémantique offre (RDF, RDFS, OWL-DL). Nous avons aussi tiré avantage du web sémantique pour encourager la communauté du génie logiciel à unir leur savoir afin d'avoir la même compréhension et interprétation des données et connaissances sur la qualité logicielle. Nous avons réalisé une ontologie qui regroupe ces connaissances et données (modèles de qualité, attributs de qualité, métriques, etc.) indépendamment de leurs formats de sauvegarde. Nous avons mis cette ontologie à la disposition de tous et tout acteur avec le droit d'écriture peut apporter sa contribution à cet effort de centralisation, d'uniformisation et partage des connaissances. En faisant partie intégrante d'un système d'aide à la décision, cette ontologie est destinée à contribuer, dans les phases de conception, d'implémentation et de maintenance.\ud ______________________________________________________________________________ \ud MOTS-CLÉS DE L’AUTEUR : modèles de qualité, métriques, web sémantique, ontologie

Computing system reliability modeling, analysis, and optimization

Ph.DDOCTOR OF PHILOSOPH

Programa Brasileiro da Qualidade e Produtividade em Software

Arquivo condensado em lotes de PDF, sem paginação.A Qualidade no setor de software e serviços de tecnologia da informação constitui tema relevante nas políticas públicas para a área de Tecnologia da Informação –TI desenvolvidas no Brasil nas últimas décadas. A questão relativa à qualidade materializou-se nos últimos anos com a inclusão do tema na Política Industrial do Governo Federal, tanto na Política Industrial, Tecnológica e de Comércio Exterior – PITCE, de 2004, quanto no Plano de Desenvolvimento da Produção – PDP lançada em maio de 2008

Bayesian statistical models for predicting software development effort

Constructing an accurate effort prediction model is a challenge in Software Engineering. This paper presents new Bayesian statistical models, in order to predict development effort of software systems in the International Software Benchmarking Standards Group (ISBSG) dataset. The first model is a Bayesian linear regression (BR) model and the second model is a Bayesian multivariate normal distribution (BMVN) model. Both models are calibrated using subsets randomly sampled from the dataset. The models’ predictive accuracy is evaluated using other subsets, which consist of only the cases unknown to the models. The predictive accuracy is measured in terms of the absolute residuals and magnitude of relative error. They are compared with the corresponding linear regression models. The results show that the Bayesian models have predictive accuracy equivalent to the linear regression models, in general. However, the advantage of the Bayesian statistical models is that they do not require a calibration subset as large as the regression counterpart. In the case of the ISBSG dataset it is confirmed that the predictive accuracy of the Bayesian statistical models, in particular the BMVN model is significantly better than the linear regression model, when the calibration subset consists of only five or smaller number of software systems. This finding justifies the use of Bayesian statistical models in software effort prediction, in particular, when the system of interest has only a very small amount of historical data.UnpublishedC.G. Bai. Bayesian network based software reliability prediction with an operational profile. The Journal of Systems and Software, 77:103–112, 2005. C.G. Bai, Q.P. Hu, M. Xie, and S.H. Ng. Software failure prediction based on a Markov bayesian network model. The Journal of Systems and Software, 74:275–282, 2005. J. Baik, B. Boehm, and B.M. Steece. Disaggregating and calibrating the CASE tool variable in COCOMO II. IEEE Transactions on Software Engineering, 28(11):1009–1022, 2002. S. Chulani, B. Boehm, and B.M. Steece. Bayesian analysis of empirical software engineering cost models. IEEE Transactions on Software Engineering, 25(4):513–583, 1999. P. Congdon. Bayesian Statistical Modelling. John Wiley & Sons., 2001. S.D. Conte, H.E. Dunsmore, and V.Y. Shen. Software Engineering Metrics and Models. Benjamin/Cummings Publishing Company, 1986. C. Fan and Y. Yu. BBN-based software project risk management. The Journal of Systems and Software, 73:193–203, 2004. N. Fenton and M. Neil. A critique of software defect prediction models. IEEE Transactions on Software Engineering, 25(5):675–689, 1999. N.E. Fenton and S.L. Pfleeger. Software Metrics:A Rigorous & Practical Approach. PWS Publishing Company, second edition, 1997. T. Foss, E. Stensrud, B. Kitchenham, and I. Myrtveit. A simulation study of the model evaluation criterion mmre. IEEE Transactions on Software Engineering, 29(11):985–995, 2003. P.J. Green. A primer on markov chain monte carlo. In O.E. Barndorff-Nielsen, D.R. Cox, and C. Klüppelberg, editors, Complex Stochastic Systems, chapter 1, pages 1–62. Chapman & Hall/CRC, 2001. F.V. Jensen. Bayesian Networks and Decision Graphs. Springer–Verlag New York, 2001. B.A. Kitchenham, L.M. Pickard, S.G. MacDonell, and M.J. Shepperd. What accuracy statistics really measure. IEE Proceedings–Software, 148(3):81–85, 2001. S.G. MacDonell. Establishing relationships between specification size and software process effort in case environment. Information and Software Technology, 39:35–45, 1997. J. Moses. Bayesian probability distributions for assessing measurement of subjective software attributes. Information and Software Technology, 42:533–546, 2000. M. Neil, N. Fenton, and L. Nielsen. Building large-scale bayesian networks. The Knowledge Engineering Review, 15(3):257–284, 2000. P.C. Pendharkar, G.H. Subramanian, and J.A. Rodger. A probabilistic model for predicting software development effort. IEEE Transactions on Software Engineering, 31(7):615–624, 2005. I. Stamelos, L. Angelis, P. Dimou, and E. Sakellaris. On the use of Bayesian belief networks for the prediction of software productivity. Information and Software Technology, 45:51–60, 2003. E. Stensrud, T. Foss, B.A. Kitchenham, and I. Myrtveit. An empirical validation of the relationship between the magnitude of relative error and project size. In Proceedings of the 8th IEEE Symposium on Software Metrics (METRICS’02), pages 3–12, 2002. B. Stewart. Predicting project delivery rates using the Naive–Bayes classifier. Journal of Software Maintenance and Evolution: Research and Practice, 14:161–179, 2002. C. van Koten and A.R. Gray. An application of bayesian network for predicting object-oriented software maintainability. Information and Software Technology, in press, 2005. D.A. Wooff, M. Goldstein, and F.P.A. Coolen. Bayesian graphical models for software testing. IEEE Transactions on Software Engineering, 28(5):510–525, 2002

Bayesian statistical models for predicting software development effort

Constructing an accurate effort prediction model is a challenge in Software Engineering. This paper presents new Bayesian statistical models, in order to predict development effort of software systems in the International Software Benchmarking Standards Group (ISBSG) dataset. The first model is a Bayesian linear regression (BR) model and the second model is a Bayesian multivariate normal distribution (BMVN) model. Both models are calibrated using subsets randomly sampled from the dataset. The models’ predictive accuracy is evaluated using other subsets, which consist of only the cases unknown to the models. The predictive accuracy is measured in terms of the absolute residuals and magnitude of relative error. They are compared with the corresponding linear regression models. The results show that the Bayesian models have predictive accuracy equivalent to the linear regression models, in general. However, the advantage of the Bayesian statistical models is that they do not require a calibration subset as large as the regression counterpart. In the case of the ISBSG dataset it is confirmed that the predictive accuracy of the Bayesian statistical models, in particular the BMVN model is significantly better than the linear regression model, when the calibration subset consists of only five or smaller number of software systems. This finding justifies the use of Bayesian statistical models in software effort prediction, in particular, when the system of interest has only a very small amount of historical data.UnpublishedC.G. Bai. Bayesian network based software reliability prediction with an operational profile. The Journal of Systems and Software, 77:103–112, 2005. C.G. Bai, Q.P. Hu, M. Xie, and S.H. Ng. Software failure prediction based on a Markov bayesian network model. The Journal of Systems and Software, 74:275–282, 2005. J. Baik, B. Boehm, and B.M. Steece. Disaggregating and calibrating the CASE tool variable in COCOMO II. IEEE Transactions on Software Engineering, 28(11):1009–1022, 2002. S. Chulani, B. Boehm, and B.M. Steece. Bayesian analysis of empirical software engineering cost models. IEEE Transactions on Software Engineering, 25(4):513–583, 1999. P. Congdon. Bayesian Statistical Modelling. John Wiley & Sons., 2001. S.D. Conte, H.E. Dunsmore, and V.Y. Shen. Software Engineering Metrics and Models. Benjamin/Cummings Publishing Company, 1986. C. Fan and Y. Yu. BBN-based software project risk management. The Journal of Systems and Software, 73:193–203, 2004. N. Fenton and M. Neil. A critique of software defect prediction models. IEEE Transactions on Software Engineering, 25(5):675–689, 1999. N.E. Fenton and S.L. Pfleeger. Software Metrics:A Rigorous & Practical Approach. PWS Publishing Company, second edition, 1997. T. Foss, E. Stensrud, B. Kitchenham, and I. Myrtveit. A simulation study of the model evaluation criterion mmre. IEEE Transactions on Software Engineering, 29(11):985–995, 2003. P.J. Green. A primer on markov chain monte carlo. In O.E. Barndorff-Nielsen, D.R. Cox, and C. Klüppelberg, editors, Complex Stochastic Systems, chapter 1, pages 1–62. Chapman & Hall/CRC, 2001. F.V. Jensen. Bayesian Networks and Decision Graphs. Springer–Verlag New York, 2001. B.A. Kitchenham, L.M. Pickard, S.G. MacDonell, and M.J. Shepperd. What accuracy statistics really measure. IEE Proceedings–Software, 148(3):81–85, 2001. S.G. MacDonell. Establishing relationships between specification size and software process effort in case environment. Information and Software Technology, 39:35–45, 1997. J. Moses. Bayesian probability distributions for assessing measurement of subjective software attributes. Information and Software Technology, 42:533–546, 2000. M. Neil, N. Fenton, and L. Nielsen. Building large-scale bayesian networks. The Knowledge Engineering Review, 15(3):257–284, 2000. P.C. Pendharkar, G.H. Subramanian, and J.A. Rodger. A probabilistic model for predicting software development effort. IEEE Transactions on Software Engineering, 31(7):615–624, 2005. I. Stamelos, L. Angelis, P. Dimou, and E. Sakellaris. On the use of Bayesian belief networks for the prediction of software productivity. Information and Software Technology, 45:51–60, 2003. E. Stensrud, T. Foss, B.A. Kitchenham, and I. Myrtveit. An empirical validation of the relationship between the magnitude of relative error and project size. In Proceedings of the 8th IEEE Symposium on Software Metrics (METRICS’02), pages 3–12, 2002. B. Stewart. Predicting project delivery rates using the Naive–Bayes classifier. Journal of Software Maintenance and Evolution: Research and Practice, 14:161–179, 2002. C. van Koten and A.R. Gray. An application of bayesian network for predicting object-oriented software maintainability. Information and Software Technology, in press, 2005. D.A. Wooff, M. Goldstein, and F.P.A. Coolen. Bayesian graphical models for software testing. IEEE Transactions on Software Engineering, 28(5):510–525, 2002