Academic Programs Adequate For The Software Profession?

According to the Bureau of Labor Statistics, close to 1.8 million people, or 77% of all computer professionals, were working in the design, development, deployment, maintenance, and management of software in 2006.  The ACM model curriculum for the BS in computer science proposes that about 42% of the core body of knowledge be dedicated to software engineering, including programming.  An examination of the curriculum of a typical computer science department shows that, excluding programming courses, no courses specific to software engineering are required for the BS, although several are available as elective courses.  Academics typically resist the demands of the industry, in part because some of them are for specific software tools, design methods, or programming languages whose use does not last.  Under market pressure, more required software engineering courses may slowly be included in the curriculum.  The usual solution is for businesses to offer their software professionals needed courses in software engineering

A Brief Survey on Oracle-based Test Adequacy Metrics

Even though code coverage is a widespread and popular test adequacy metric, it has several limitations. One of the major limitations is that code coverage does not satisfy the necessary conditions for effective fault detection, as it only cares about executing different parts of a program. Studies showed that code coverage as a test adequacy metric is a poor indicator of the quality of a test suite as it does not consider the quality of test oracles. To address this limitation, researchers proposed extensions to traditional code coverage metrics that explicitly consider test oracle quality. We name these metrics as \textit{oracle-based code coverage}. This survey paper has discussed all oracle-based techniques published so far, starting from 2007.Comment: 7 page

Um algoritmo para o cálculo de cobertura de estados

Dissertação (mestrado) - Universidade Federal de Santa Catarina, Centro Tecnológico, Programa de Pós-Graduação em Ciência da Computação, Florianópolis, 2015.Cobertura de estados é um critério de adequação de testes de software que mede a quantidade de modificações de estados feitas durante a execução dos testes que foram verificadas através de asserções. O presente trabalho propõe um algoritmo para o cálculo de cobertura de estados baseado em construções comuns a linguagens orientadas a objetos, como atribuições, retorno de métodos e chamadas de funções. O algoritmo identifica modificações de estados cobertas por asserções através de um novo cálculo de influências de atributos em métodos de uma classe baseado na extração de dependências entre identificadores existentes no código. São apresentados ainda uma extensão à definição de estado modificado levando em consideração a distinção entre atributos simples e compostos (estruturas de dados) e uma implementação do algoritmo através da instrumentação de bytecode Java. Experimentos feitos em sete projetos de código aberto mostraram a utilidade do algoritmo na identificação de atributos não verificados por asserções mas executados pelo teste. Ao inserir erros propositais nestes atributos os testes não falharam. Em uma situação real, estes erros estariam imperceptíveis pelos desenvolvedores. Os experimentos ainda mostraram que a execução do algoritmo adicionou um overhead de 20% a 33% no tempo de execução dos testes unitários, um valor abaixo dos trabalhos existentes.Abstract : State coverage is a test adequacy criterion that measures the quantity of state modifications made by a test execution that were verified by assertions. This work proposes an algorithm for state coverage based on common constructions of object-oriented languages, such as assignments, method returns and function calls. The algorithm identifies influences of an instance attribute on assertions applying a new method to compute influences of attributes on methods. This work also presents an extension to the modified state definition by applying a distinction between simple and compound types (data-structures) and an implementation of the algorithm based on instrumentation of Java bytecode. Experiments made on seven open source projects showed the validity of the algorithm in identifying attributes not verified by assertions that were executed by tests. Bugs inserted in these attributes were not captured by the tests. The results also showed that the algorithm adds an overhead of 20% and 33% at the test execution. These values are below of existent works

Assessing test quality

When developing tests, one is interested in creating tests of good quality that thoroughly test the program. This work shows how to assess test quality through mutation testing with impact metrics, and through checked coverage. Although there a different aspects that contribute to a test\u27s quality, the most important factor is its ability to reveal defects, because software testing is usually carried out with the aim to detect defects. For this purpose, a test has to provide inputs that execute the defective code under such conditions that it causes an infection. This infection has to propagate and result in a failure, which can be detected by a check of the test. In the past, the aspect of test input quality has been extensively studied while the quality of checks has received less attention. The traditional way of assessing the quality of a test suite\u27s checks is mutation testing. Mutation testing seeds artificial defects (mutations) into a program, and checks whether the tests detect them. While this technique effectively assesses the quality of checks, it also has two drawbacks. First, it places a huge demand on computing resources. Second, equivalent mutants, which are mutants that are semantically equivalent to the original program, dilute the quality of the results. In this work, we address both of these issues. We present the JAVALANCHE framework that applies several optimizations to enable automated and efficient mutation testing for real-life programs. Furthermore, we address the problem of equivalent mutants by introducing impact metrics to detect non-equivalent mutants. Impact metrics compare properties of tests suite runs on the original program with runs on mutated versions, and are based on abstractions over program runs such as dynamic invariants, covered statements, and return values. The intention of these metrics is that mutations that have a graver influence on the program run are more likely to be non-equivalent. Moreover, we introduce checked coverage, an alternative approach to measure the quality of a test suite\u27s checks. Checked coverage determines the parts of the code that were not only executed, but that actually contribute to the results checked by the test suite, by computing dynamic backward slices from all explicit checks of the test suite.Diese Arbeit stellt dar, wie die Qualität von Software Tests durch mutationsbasiertes Testen in Verbindung mit Auswirkungsmaßen und durch Checked Coverage beurteilt werden kann. Obwohl unterschiedliche Faktoren die Qualität eines Tests beeinflussen, ist der wichtigste Aspekt die Fähigkeit Fehler aufzudecken. Dazu muss ein Test Eingaben bereitstellen, die den fehlerhaften Teil des Programms so ausführen, dass eine Infektion entsteht, d.h. der Programmzustand fehlerhaft wird. Diese Infektion muss sich so fortpflanzen, dass sie in einem fehlerhaften Ergebnis resultiert, welches dann von einer Test-Prüfung erkannt werden muss. Die herkömmliche Methode um die Qualität von Test-Prüfungen zu beurteilen ist mutationsbasiertes Testen. Hierbei werden künstliche Fehler (Mutationen) in ein Programm eingebaut und es wird überprüft, ob diese von den Tests erkannt werden. Obwohl diese Technik die Qualität von Test-Prüfungen beurteilen kann, weist sie zwei Nachteile auf. Erstens hat sie einen großen Bedarf an Rechenkapazitäten. Zweitens verwässern äquivalente Mutationen, welche zwar die Syntax eines Programms ändern, jedoch nicht seine Semantik, die Qualität der Ergebnisse. In dieser Arbeit werden Lösungen für beide Probleme aufgezeigt. Wir präsentieren JAVALANCHE, ein System, das effizientes und automatisiertes mutationsbasiertes Testen für realistische Programme ermöglicht. Des Weiteren wird das Problem von äquivalenten Mutationen mittels Auswirkungsmaßen angegangen. Auswirkungsmaße vergleichen Eigenschaften zwischen einem normalen Programmlauf und einem Lauf eines mutierten Programms. Hierbei werden verschiedene Abstraktionen über den Programmlauf benutzt. Die zugrunde liegende Idee ist, dass eine Mutation, die eine große Auswirkung auf den Programmlauf hat, weniger wahrscheinlich äquivalent ist. Darüber hinaus stellen wir Checked Coverage vor, ein neuartiges Abdeckungsmaß, welches die Qualität von Test-Prüfungen misst. Checked Coverage bestimmt die Teile im Programmcode, die nicht nur ausgeführt, sondern deren Resultate auch von den Tests überprüft werden