Delayed failure of software components using stochastic testing

The present research investigates the delayed failure of software components and addresses the problem that the conventional approach to software testing is unlikely to reveal this type of failure. Delayed failure is defined as a failure that occurs some time after the condition that causes the failure, and is a consequence of long-latency error propagation. This research seeks to close a perceived gap between academic research into software testing and industrial software testing practice by showing that stochastic testing can reveal delayed failure, and supporting this conclusion by a model of error propagation and failure that has been validated by experiment. The focus of the present research is on software components described by a request-response model. Within this conceptual framework, a Markov chain model of error propagation and failure is used to derive the expected delayed failure behaviour of software components. Results from an experimental study of delayed failure of DBMS software components MySQL and Oracle XE using stochastic testing with random generation of SQL are consistent with expected behaviour based on the Markov chain model. Metrics for failure delay and reliability are shown to depend on the characteristics of the chosen experimental profile. SQL mutation is used to generate negative as well as positive test profiles. There appear to be few systematic studies of delayed failure in the software engineering literature, and no studies of stochastic testing related to delayed failure of software components, or specifically to delayed failure of DBMS. Stochastic testing is shown to be an effective technique for revealing delayed failure of software components, as well as a suitable technique for reliability and robustness testing of software components. These results provide a deeper insight into the testing technique and should lead to further research. Stochastic testing could provide a dependability benchmark for component-based software engineering

Uso de riscos na validação de sistemas baseados em componentes

Orientadores: Eliane Martins, Henrique Santos do Carmo MadeiraTese (doutorado) - Universidade Estadual de Campinas, Instituto de ComputaçãoResumo: A sociedade moderna está cada vez mais dependente dos serviços prestados pelos computadores e, conseqüentemente, dependente do software que está sendo executado para prover estes serviços. Considerando a tendência crescente do desenvolvimento de produtos de software utilizando componentes reutilizáveis, a dependabilidade do software, ou seja, a segurança de que o software irá funcionar adequadamente, recai na dependabilidade dos componentes que são integrados. Os componentes são normalmente adquiridos de terceiros ou produzidos por outras equipes de desenvolvimento. Dessa forma, os critérios utilizados na fase de testes dos componentes dificilmente estão disponíveis. A falta desta informação aliada ao fato de se estar utilizando um componente que não foi produzido para o sistema e o ambiente computacional específico faz com que a reutilização de componentes apresente um risco para o sistema que os integra. Estudos tradicionais do risco de um componente de software definem dois fatores que caracteriza o risco, a probabilidade de existir uma falha no componente e o impacto que isso causa no sistema computacional. Este trabalho propõe o uso da análise do risco para selecionar pontos de injeção e monitoração para campanhas de injeção de falhas. Também propõe uma abordagem experimental para a avaliação do risco de um componente para um sistema. Para se estimar a probabilidade de existir uma falha no componente, métricas de software foram combinadas num modelo estatístico. O impacto da manifestação de uma falha no sistema foi estimado experimentalmente utilizando a injeção de falhas. Considerando esta abordagem, a avaliação do risco se torna genérica e repetível embasando-se em medidas bem definidas. Dessa forma, a metodologia pode ser utilizada como um benchmark de componentes quanto ao risco e pode ser utilizada quando é preciso escolher o melhor componente para um sistema computacional, entre os vários componentes que provêem a mesma funcionalidade. Os resultados obtidos na aplicação desta abordagem em estudos de casos nos permitiram escolher o melhor componente, considerando diversos objetivos e necessidades dos usuáriosAbstract: Today's societies have become increasingly dependent on information services. A corollary is that we have also become increasingly dependent on computer software products that provide such services. The increasing tendency of software development to employ reusable components means that software dependability has become even more reliant on the dependability of integrated components. Components are usually acquired from third parties or developed by unknown development teams. In this way, the criteria employed in the testing phase of components-based systems are hardly ever been available. This lack of information, coupled with the use of components that are not specifically developed for a particular system and computational environment, makes components reutilization risky for the integrating system. Traditional studies on the risk of software components suggest that two aspects must be considered when risk assessment tests are performed, namely the probability of residual fault in software component, and the probability of such fault activation and impact on the computational system. The present work proposes the use of risk analysis to select the injection and monitoring points for fault injection campaigns. It also proposes an experimental approach to evaluate the risk a particular component may represent to a system. In order to determine the probability of a residual fault in the component, software metrics are combined in a statistical mode!. The impact of fault activation is estimated using fault injection. Through this experimental approach, risk evaluation becomes replicable and buttressed on well-defined measurements. In this way, the methodology can be used as a components' risk benchmark, and can be employed when it is necessary to choose the most suitable among several functionally-similar components for a particular computational system. The results obtained in the application of this approach to specific case studies allowed us to choose the best component in each case, without jeopardizing the diverse objectives and needs of their usersDoutoradoDoutor em Ciência da Computaçã

Caractérisation de la sûreté de fonctionnement de systèmes à base d'intergiciel

Les systèmes critiques sont soumis, comme le reste de l'industrie informatique, à des contraintes de coût de plus en plus sévères. Cette pression pousse les développeurs à privilégier la réutilisation de logiciels, plutôt que de procéder à des développements spécifiques à chaque projet. Cette tendance à l'utilisation de composants logiciels "sur étagère", souvent développés par des tiers, est renforcée par des besoins technologiques de plus en plus complexes, en particulier l'intégration des systèmes dans des réseaux de communication. L'utilisation de composants sur étagère permet aux industriels de se concentrer sur leur domaine de compétence, sans gaspiller de l'effort à redévelopper des fonctions qui ont déjà été implantées dans d'autres secteurs. Cette tendance à la réutilisation, ainsi que l'interconnexion croissante des systèmes, a favorisé l'émergence de standards d'interface, qui permettent l'interopérabilité de systèmes, même lorsqu'ils sont développés par des groupes différents. L'un des standards d'interface pour l'intégration de systèmes est les intergiciels de communication tels que la plate-forme CORBA. Ces intergiciels facilitent l'interaction entre des applications disparates, s'exécutant sur des plates-formes matérielles et logicielles hétérogènes. Pour les intégrateurs de systèmes distribués, ces technologies sont attractives pour plusieurs raisons, autant technologiques qu'économiques : elles constituent un moyen rapide d'intégration de nouvelles technologies, d'augmentation de la souplesse et l'ouverture vers d'autres systèmes. Toutefois, cette attractivité est conditionnée par des craintes concernant la robustesse des composants intergiciels, qui n'ont pas bénéficié de la rigueur du processus de développement utilisé dans le contexte de systèmes critiques. Les intégrateurs de systèmes distribués semi-critiques souhaitent avoir des assurances sur la qualité et la robustesse des composants qu'ils intègrent au sein de leurs systèmes. Ils souhaitent des informations sur les modes de défaillance de l'intergiciel, sur les canaux de propagation d'erreur qu'il introduit. Ils souhaitent avoir des informations quantitatives leur permettant de comparer différentes implémentations candidates du point de vue de la sûreté de fonctionnement, afin de sélectionner le candidat qui est le mieux adapté à leurs besoins. La problématique que nous venons d'énoncer peut se résumer en deux points : - obtenir une meilleure connaissance des types de fautes et d'erreurs qui existent dans les systèmes à base d'intergiciel de communication ; - développer une méthode permettant de caractériser expérimentalement la robustesse de candidats intergiciels. Il existe actuellement très peu de travaux qui permettent de répondre à ces interrogations. L'objectif de cette thèse est de proposer une méthode de caractérisation qui puisse être appliquée à des intergiciels cibles, et de répondre ainsi aux problèmes des intégrateurs de systèmes semi-critiques, ainsi qu'aux développeurs de composants intergiciel. Notre contribution est de proposer une méthodologie pour l'analyse de la sûreté de fonctionnement d'un intergiciel. Notre méthode est basée sur une analyse structurelle des intergiciels de communication, sur l'élaboration d'un modèle de fautes, une classification des modes de défaillance, et le développement d'un ensemble de techniques d'injection de faute adaptées à l'intergiciel. Nous avons validé notre approche en menant des campagnes d'injection de faute ciblant plusieurs implémentations de la norme CORBA. ABSTRACT : We propose a method for the dependability assessment and failure mode characterization of communications middleware. The method is based on the structural analysis of communications-oriented middleware, the identification of a fault model, a failure modes classification, and the development of a number of fault injection techniques that can be used to target middleware implementations. We have applied our method by carrying out fault injection campaigns targeting a number of CORBA implementations, and obtained quantitative measures of the robustness of the different candidates. Our work allows integrators of dependable distributed systems to obtain assurances on the robustness of the software components they place at the heart of their systems, and provides information to middleware vendors regarding robustness failings in their products

Fault Injection Approach Based On Dependence Analysis

Fault Injection is used to validate a system in the presence of faults. Jaca, a software injection tool developed in previous work, is used to inject faults at interfaces between classes of a system written in Java. We present a strategy for fault injection validation based on dependence analysis. The dependence analysis approach is used to help in reducing the number of experiments necessary to cover the system's interfaces. For the experiments we used a system that consists of two integrated components, an ODBMS performance test benchmark, Wisconsin OO7 and an ODBMS, Ozone. The results of some experiments and their analysis are presented. © 2005 IEEE.2181188Arlat, J., Aguera, M., Amat, L., Crouzet, Y., Fabre, J.C., Laprie, J.C., Martins, E., Powell, D., Fault Injection for Dependability Validation-A Methodology and some Applications IEEE Transactions on Software Engineering, 16 (2), pp. 166-182. , Feb/1990Bach, J., Heuristic risk-based testing (1999) Software Testing and Quality Engineering Magazine, , NovCarey, M. J., DeWitt, D. J., Naughton, J. F.: The OO7 Benchmark. http://www.columbia.edu/, 1994, recovered Feb/2005Carreira, J., Madeira, H., Silva, J., Xception, G., Software Fault Injection and Monitoring in Processor Functional Units (1995) 5th IFIP International Working Conference on Dependable Computing for Critical Applications, pp. 135-149. , Urbana-Champaign, EUA, ppChen, Z., Xul, B., Zhao, J., An Overview of Methods for Dependence Analysis of Concurrent Programs ACM SIGPLAN Notices, 37 (8), pp. 45-52. , Aug/2002Chiba, Shigeru. Javassist - A Reflection-based Programming Wizard for Java. Proc of the ACM OOPSLA '98 Workshop on Reflective Programming in C++ and Java, Oct/1998Costa, D., Madeira, H., (1999) Experimental Assessment of COTS DBMS Robustness under Transient Faults, Pacific Rim Dependability Computing, , Hong KongCosta, D., Rilho, T., Madeira, H., Joint Evaluation of Performance and Robustness of a COTS DBMS through Fault Injection, , New York, DSN 2000De Millo, R. A., Li, T., Mathur, A. P.: Architecture of TAMER: A Tool for dependability analysis of distributed fault-tolerant systems, Purdue University, 1994Fetzer, C.Högstedt, K.Felber, P. Automatic Detection and Masking of Non-Atomic Exception Handling, proceedings of DSN 2003, pages 445/454, San Francisco, USA, June/2003Hsueh, Mei-Chen, Tsai, Timothy, Iyer, R.: Fault Injection Techniques and Tools. IEEE Computer, pp. 75-82, Apr/1997Koopman, P., Siewiorek, D., DeVale, K., DeVale, J., Fernsler, K., Guttendorf, D., Kropp, N., Pan, J., Shelton, C., Shi, Y.: Ballista Project:COTS Software Robustness Testing, Carnegie Mellon University, www.ece.cmu.edu/ ~koopman/ballista/, 2003Martins, E., Rubira, C. M. F., Lerne N.G.M.: Jaca: A reflective fault injection tool based on patterns. Proc of the 2002 Intern Conference on Dependable Systems & Networks, Washington D.C. USA, 23-267, pp. 483-487 June/2002Moraes, R, Martins, E.: Testing Component-based Applications in the Presence of Faults. Proc. of the 7th World Multi-conference on Systemic, CyberneticsMoraes, R., Martins, E., A Strategy for Validating an ODBMS Component Using a High-Level Software Fault Injection Tool (2003) Proc. of the First Latin-American Symposium, LADC, pp. 56-68. , São Paulo, BrazilMoraes, R., Martins, E., An Architecture-based Strategy for Interface Fault Injection (2004) Proc. International Conference no Dependable Systems and Networks, , Firenze, ItalyMoraes, R., Martins, E., Fault Injection Approach based on Architectural Dependences. Architecting Dependable Systems III, Lecture Notes in Computer Science, Springer-Verlag Berlin Heidelberg New York to appear(2004) Ozone, Object Oriented Database Management System, , www.ozone-db.orgPressman, R.S., (1997) Software Engineering a Practitioner Approach, , 4 th edition, Me Graw HullRosenberg, L., Stapko, R., Gallo, A., Risk-based Object Oriented Testing (2000) 13th International Software / Internet Quality Week (QW2000), , San Francisco, California USASherer, S.A., A Cost-Effective Approach to Testing (1991) IEEE Software, , MarchStafford, J. A., Richardson, D.J., Wolf, A.L.: Chainning: A Software Architecture Dependence Analysis Technique, Technical Report CU-CS845-97, Department of Computer Science, University of Colorado, September/1997Vieira, M., Madeira, H., Recovery and Performance Balance of COTS DBMS in Presence of Operator Fault. IPDS (2002), Bethesda, Washington DCVoas, J., McGraw, G., (1998) Software Fault Injection: Inoculating Programs against Errors, , John Wiley & Sons, New York, EUAVoas, J. (2003) Marrying Software Fault Injection Technology Results with Software Reliability Growth Models, Fast Abstract ISSRE 2003, Chillarege Pres