Qualitative and semi-quantitative modelling and simulation of the software engineering processes

Software process modelling has been identified as being a vehicle for understanding development processes, controlling development costs, duration, and achieving product quality. In recent years, software process simulation has been becoming one of the essential techniques for effectively investigating and managing software development processes. Till now, most researches focus on the quantitative aspects of process simulation and modelling. Nevertheless, purely quantitative process modelling requires a very detailed understanding and accurate measurement of the software process, which relies on reliable and precise historical data. When such data are lacking or the quality is dubious, quantitative models have to impose severe constraints that restrict the model's value. Unfortunately, these data are not readily available in most cases, especially in the organisations at low process maturity levels. In addition, software development is a highly complex, human-centred endeavour, which involves many uncertain factors in the course of development process. Facing the inherent uncertainty and contingency, though quantitative modelling employs statistic techniques, its conditional capability and underlying assumptions limit its performance on large scale problems. As the alternatives of quantitative approaches, qualitative modelling can cope with a lack of complete knowledge, and predicts qualitative process behaviours. Furthermore, semi-quantitative modelling offers the capability of handling process uncertainty with limited knowledge, and achieves tradeoff between quantitative and qualitative approaches. However, most previous researches omitted these approaches, and the associated methods and applications are far from developed. The main contribution of this research lies in the pioneering work on the models, methods, and applications of qualitative and semi-quantitative software process modelling and simulation, and their relations with the conventional, quantitative modelling approaches. This dissertation produces its novelty from twofold research. Firstly, it explores methods and techniques to qualitatively and semi-quantitatively model and simulate software processes at different levels, i.e. project, portion of development process, and product evolution. Secondly, Some exclusive applications of these modelling approaches are also developed for aspects of software engineering practice. Moreover, a proposed framework integrates these approaches with typical quantitative paradigms to guide the adoption of process simulation modelling in software organisations. As a comprehensive reflection of state-of-the-art of software process simulation modelling, a systematic review is reported in this dissertation as well

La gestion de la connaissance des équipes de développement logiciel

RÉSUMÉ Contexte : Le développement logiciel est un travail d’équipe manipulant un produit essentiellement invisible. En conséquent, le développement logiciel nécessite des échanges de connaissances importants entre développeurs afin que l’équipe effectue une résolution de problème adéquate. Cette résolution de problème résulte en une prise de décision qui aura un impact direct sur la qualité du produit logiciel final. Objectif : Ce travail doctoral a pour objectif de mieux comprendre ces interactions entre développeurs et comment ces interactions peuvent être liées à des problèmes de qualité logicielle. Cette meilleure compréhension du phénomène permet d’améliorer les approches actuelles de développement logiciel afin d’assurer une meilleure qualité du produit final. Méthodologie : Premièrement, des revues de littérature ont été effectuées afin de mieux comprendre l’état actuel de la recherche en gestion de connaissance dans le génie logiciel. Deuxièmement, des analyses de code source et des discussions avec les développeurs ont été faites afin de mieux cerner les causes de problèmes classiques de qualité logicielle. Finalement, des observations faites dans l’industrie ont permis de comprendre la prise de décision collective, et comment cette prise de décision impacte la qualité logicielle. Résultats : Les observations effectuées ont démontré que la qualité logicielle n’est pas qu’un problème d’éducation ; l’essentiel des problèmes de qualité ont été introduits par les développeurs en toute connaissance de cause afin de répondre à d’autres impératifs plus urgents au moment de la prise de décision. Améliorer la qualité des logiciels demande de revoir la manière dont les projets de développement logiciel sont gérés afin d’assurer que les décisions prises sur le terrain n’auront pas de conséquences négatives trop coûteuses à long terme. Conclusions : Il est recommandé que les organisations se dote d’un nouveau palier décisionnel faisant la jointure entre besoins techniques (i.e. qualité logicielle) et administratifs (i.e. ressources disponibles). Ce nouveau palier décisionnel se situerait au niveau de la base de code (« codebase »), soit entre le palier organisationnel et le palier de gestion de projet. Une base de code étant modifiée de manière concurrente par plusieurs projets en parallèle, il devient nécessaire d’avoir un meilleur contrôle sur les modifications effectuées sur celle-ci. Ce nouveau palier serait le gardien des connaissances en lien avec la base de code, selon le principe « you build it, you run it » favorisé dans certaines organisations. Ce nouveau palier serait responsable d’assurer que la base de code reste d’une qualité suffisamment bonne pour supporter les activités de l’organisation dans l’avenir.----------ABSTRACT Context: Software development is a process requiring teamwork on an essentially invisible product. Therefore, software development requires important knowledge exchanges between developers in order to ensure a proper problem resolution. This problem resolution affects the decision making process, which will have a direct impact on the software quality of the final product. Objective: This thesis work aims to better understand these interactions between developers and how they can be linked to software quality problems. With a better understanding of the relation, it will be possible to improve the current software development management practices in order to ensure a better quality of the final software product. Method: First, literature reviews were made with the objective to understand the current state of the research in knowledge management in software engineering. Second, source code analyzes and discussions with the developers were executed in order to better understand the causes of typical software quality issues. Finally, observations were made in an industrial context in order to observe collective decision making in the field, and to understand how these decisions impacts software quality. Results: The bservations made demonstrated that software quality is not only an educational problem; most of the quality problems found were introduced voluntarily by the developers in order to answer a more urgent requirement at the time. Improving software quality therefore requires a review of how software development projects are managed in order to ensure that the decision made in the field do not have overly costly consequences in the long term. Conclusions: It is recommended that organization assign a new decision level linking the technical requirements (i.e. software quality) with administrative requirements (i.e. available resources). This new decision level would be situated at the codebase level, between the organizational strategy level and the project management level. A codebase being modified concurrently by multiple projects, it is therefore necessary to have a better control of the modifications made on it. The people at this new decision level would be the knowledge repository related to the codebase, under the “you build it, you run it” principle popular in some organizations. This new decision level would be responsible of ensuring that the codebase remains of a sufficient quality in order to support the future activities of the organization

Systematic construction of goal-oriented COTS taxonomies

El proceso de construir software a partir del ensamblaje e integración de soluciones de software pre-fabricadas, conocidas como componentes COTS (Comercial-Off-The-Shelf) se ha convertido en una necesidad estratégica en una amplia variedad de áreas de aplicación. En general, los componentes COTS son componentes de software que proveen una funcionalidad específica, que están disponibles en el mercado para ser adquiridos e integrados dentro de otros sistemas de software. Los beneficios potenciales de esta tecnología son principalmente la reducción de costes y el acortamiento del tiempo de desarrollo, a la vez que fomenta la calidad. Sin embargo, numerosos retos que van desde problemas técnicos y legales deben ser afrontados para adaptar las actividades tradicionales de ingeniería de software para explotar los beneficios del uso de COTS para el desarrollo de sistemas.Actualmente, existe un incrementalmente enorme mercado de componentes COTS; así, una de las actividades más críticas en el desarrollo de sistemas basados en COTS es la selección de componentes que deben ser integrados en el sistema a desarrollar. La selección está básicamente compuesta de dos procesos principales: La búsqueda de componentes candidatos en el mercado y su posterior evaluación con respecto a los requisitos del sistema. Desafortunadamente, la mayoría de los métodos existentes para seleccionar COTS, se enfocan en el proceso de evaluación, dejando de lado el problema de buscar los componentes en el mercado. La búsqueda de componentes en el mercado no es una tarea trivial, teniendo que afrontar varias características del mercado de COTS, tales como su naturaleza dispersa y siempre creciente, cambio y evolución constante; en este contexto, la obtención de información de calidad acerca de los componentes no es una tarea fácil. Como consecuencia, el proceso de selección de COTS se ve seriamente dañado. Además, las alternativas tradicionales de reuso también carecen de soluciones apropiadas para reusar componentes COTS y el conocimiento adquirido en cada proceso de selección. Esta carencia de propuestas es un problema muy serio que incrementa los riesgos de los proyectos de selección de COTS, además de hacerlos ineficientes y altamente costosos. Esta disertación presenta el método GOThIC (Goal- Oriented Taxonomy and reuse Infrastructure Construction) enfocado a la construcción de infraestructuras de reuso para facilitar la búsqueda y reuso de componentes COTS. El método está basado en el uso de objetivos para construir taxonomías abstractas, bien fundamentadas y estables para lidiar con las características del mercado de COTS. Los nodos de las taxonomías son caracterizados por objetivos, sus relaciones son declaradas como dependencias y varios artefactos son construidos y gestionados para promover la reusabilidad y lidiar con la evolución constante.El método GOThIC ha sido elaborado a través de un proceso iterativo de investigación-acción para identificar los retos reales relacionados con el proceso de búsqueda de COTS. Posteriormente, las soluciones posibles fueron evaluadas e implementadas en varios casos de estudio en el ámbito industrial y académico en diversos dominios. Los resultados más relevantes fueron registrados y articulados en el método GOThIC. La evaluación industrial preliminar del método se ha llevado a cabo en algunas compañías en Noruega.The process of building software systems by assembling and integrating pre-packaged solutions in the form of Commercial-Off-The-Shelf (COTS) software components has become a strategic need in a wide variety of application areas. In general, COTS components are software components that provide a specific functionality, available in the market to be purchased, interfaced and integrated into other software systems. The potential benefits of this technology are mainly its reduced costs and shorter development time, while maintaining the quality. Nevertheless, many challenges ranging form technical to legal issues must be faced for adapting the traditional software engineering activities in order to exploit these benefits.Nowadays there is an increasingly huge marketplace of COTS components; therefore, one of the most critical activities in COTS-based development is the selection of the components to be integrated into the system under development. Selection is basically composed of two main processes, namely: searching of candidates from the marketplace and their evaluation with respect to the system requirements. Unfortunately, most of the different existing methods for COTS selection focus their efforts on evaluation, letting aside the problem of searching components in the marketplace. Searching candidate COTS is not an easy task, having to cope with some challenging marketplace characteristics related to its widespread, evolvable and growing nature; and the lack of available and well-suited information to obtain a quality-assured search. Indeed, traditional reuse approaches also lack of appropriate solutions to reuse COTS components and the knowledge gained in each selection process. This lack of proposals is a serious drawback that makes the whole selection process highly risky, and often expensive and inefficient. This dissertation introduces the GOThIC (Goal- Oriented Taxonomy and reuse Infrastructure Construction) method aimed at building a domain reuse infrastructure for facilitating COTS components searching and reuse. It is based on goal-oriented approaches for building abstract, well-founded and stable taxonomies capable of dealing with the COTS marketplace characteristics. Thus, the nodes of these taxonomies are characterized by means of goals, their relationships declared as dependencies among them and several artifacts are constructed and managed for reusability and evolution purposes. The GOThIC method has been elaborated following an iterative process based on action research premises to identify the actual challenges related to COTS components searching. Then, possible solutions were envisaged and implemented by several industrial and academic case studies in different domains. Successful results were recorded to articulate the synergic GOThIC method solution, followed by its preliminary industrial evaluation in some Norwegian companies

PROCESS CONFORMANCE TESTING: A METHODOLOGY TO IDENTIFY AND UNDERSTAND PROCESS VIOLATIONS IN ENACTMENT OF SOFTWARE PROCESSES

Today's software development is driven by software processes and practices that when followed increase the chances of building high quality software products. Not following these guidelines results in increased risk that the goal for the software's quality characteristics cannot be reached. Current process analysis approaches are limited in identifying and understanding process deviations and ultimately fail in comprehending why a process does not work in a given environment and what steps of the process have to be changed and tailored. In this work I will present a methodology for formulating, identifying and investigating process violations in the execution of software processes. The methodology, which can be thought of as "Process Conformance Testing", consists of a four step iterative model, compromising templates and tools. A strong focus is set on identifying violations in a cost efficient and unobtrusive manner by utilizing automatically collected data gathered through commonly used software development tools, such as version control systems. To evaluate the usefulness and correctness of the model a series of four studies have been conducted in both classroom and professional environments. A total of eight different software processes have been investigated and tested. The results of the studies show that the steps and iterative character of the methodology are useful for formulating and tailoring violation detection strategies and investigating violations in classroom study environments and professional environments. All the investigated processes were violated in some way, which emphasizes the importance of conformance measurement. This is especially important when running an empirical study to evaluate the effectiveness of a software process, as the experimenters want to make sure they are evaluating the specified process and not a variation of it. Violation detection strategies were tailored based upon analysis of the history of violations and feedback from then enactors and mangers yielding greater precision of identification of non-conformities. The overhead cost of the approach is shown to be feasible with a 3.4% (professional environment) and 12.1% (classroom environment) overhead. One interesting side result is that process enactors did not always follow the process for good reason, e.g. the process was not tailored for the environment, it was not specified at the right level of granularity, or was too difficult to follow. Two specific examples in this thesis are XP Pair Switching and Test Driven Development. In XP Pair Switching, the practice was violated because the frequency of switching was too high. The definition of Test Driven Development is simple and clear but requires a fair amount of discipline to follow, especially by novice programmers

Domain- and Quality-aware Requirements Engineering for Law-compliant Systems

Titel in deutscher Übersetzung: Domänen- und qualitätsgetriebene Anforderungserhebung für gesetzeskonforme Systeme Der bekannte Leitsatz in der Anforderungserhebung und -analyse besagt, dass es schwierig ist, das richtige System zu bauen, wenn man nicht weiß, was das 'Richtige' eigentlich ist. Es existieren überzeugende Belege, dass dieser Leitsatz die Notwendigkeit der Anforderungserhebung und -analyse exakt definiert und beschreibt. Zum Beispiel ergaben Studien, dass das Beheben von Defekten in einer Software, die bereits produktiv genutzt wird, bis zu 80 mal so teuer ist wie das frühzeitige Beheben der korrespondierenden Defekte in den Anforderungen. Generell hat es sich gezeigt, dass das Durchführen einer angemessenen Anforderungserhebung und -analyse ein wichtiger Erfolgsfaktor für Softwareentwicklungsprojekte ist. Während der Progression von den initialen Wünschen der beteiligten Interessensvertretern für ein zu entwickelndes System zu einer Spezifikation für eben dieses Systems müssen Anforderungsanalysten einen komplexen Entscheidungsprozess durchlaufen, der die initialen Wünsche in die Spezifikation überführt. Tatsächlich wird das Treffen von Entscheidungen als integraler Bestandteil der Anforderungsanalyse gesehen. In dieser Arbeit werden wir versuchen zu verstehen welche Aktivitäten und Information von Nöten sind, um eine fundierte Auswahl von Anforderungen vorzunehmen, welche Herausforderungen damit verbunden sind, wie eine ideale Lösung zur Anforderungswahl aussehen könnte und in welchen Bereichen der aktuelle Stand der Technik in Bezug auf diese ideale Lösung lückenhaft ist. Innerhalb dieser Arbeit werden wir die Informationen, die notwendig für eine fundierte Anforderungsauswahl sind, identifizieren, einen Prozess präsentieren, um diese notwendigen Informationen zu sammeln, die Herausforderungen herausstellen, die durch diesen Prozess und die damit verbundenen Aktivitäten adressiert werden und eine Auswahl von Methoden diskutieren, mit deren Hilfe man die Aktivitäten des Prozesses umsetzen kann. Die gesammelten Informationen werden dann für eine automatisierte Anforderungsauswahl verwendet. Für die Auswahl kommt ein Optimierungsmodell, das Teil des Beitrags dieser Arbeit ist, zum Einsatz. Da wir während der Erstellung dieser Arbeit zwei große Lücken im Stand der Technik bezüglich unseres Prozesses und der damit verbundenen Aktivitäten identifiziert haben, präsentieren wir darüber hinaus zwei neuartige Methoden für die Kontexterhebung und die Erhebung von rechtlichen Anforderungen, um diese Lücken zu schließen. Diese Methoden sind Teil des Hauptbeitrags dieser Arbeit. Unsere Lösung für der Erhebung des Kontext für ein zu entwickelndes System ermöglicht das Etablieren eines domänenspezifischen Kontextes unter Zuhilfenahme von Mustern für verschiedene Domänen. Diese Kontextmuster erlauben eine strukturierte Erhebung und Dokumentation aller relevanten Interessensvertreter und technischen Entitäten für ein zu entwickelndes System. Sowohl die Dokumentation in Form von grafischen Musterinstanzen und textuellen Vorlageninstanzen als auch die Methode zum Sammeln der notwendigen Informationen sind expliziter Bestandteil jedes Kontextmusters. Zusätzlich stellen wir auch Hilfsmittel für die Erstellung neuer Kontextmuster und das Erweitern der in dieser Arbeit präsentierten Kontextmustersprache zur Verfügung. Unsere Lösung für die Erhebung von rechtlichen Anforderungen basiert auch auf Mustern und stellt eine Methode bereit, welche es einem erlaubt, die relevanten Gesetze für ein zu erstellendes System, welches in Form der funktionalen Anforderungen bereits beschrieben sein muss, zu identifizieren und welche die bestehenden funktionalen Anforderungen mit den rechtlichen Anforderungen verknüpft. Diese Methode beruht auf der Zusammenarbeit zwischen Anforderungsanalysten und Rechtsexperten und schließt die Verständnislücke zwischen ihren verschiedenartigen Welten. Wir veranschaulichen unseren Prozess unter der Zuhilfenahme eines durchgehenden Beispiels aus dem Bereich der service-orientierten Architekturen. Zusätzlich präsentieren wir sowohl die Ergebnisse der Anwendung unseres Prozesses (bzw. Teilen davon) auf zwei reale Fälle aus den Bereichen von Smart Grids und Wahlsystemen, als auch alle anderen Ergebnisse der wissenschaftlichen Methoden, die wir genutzt haben, um unsere Lösung zu fundieren und validieren.The long known credo of requirements engineering states that it is challenging to build the right system if you do not know what right is. There is strong evidence that this credo exactly defines and describes the necessity of requirements engineering. Fixing a defect when it is already fielded is reported to be up to eighty times more expensive than fixing the corresponding requirements defects early on. In general, conducting sufficient requirements engineering has shown to be a crucial success factor for software development projects. Throughout the progression from initial stakeholders' wishes regarding the system-to-be to a specification for the system-to-be requirements engineers have to undergo a complex decision process for forming the actual plan connecting stakeholder wishes and the final specification. Indeed, decision making is considered to be an inherent part of requirements engineering. In this thesis, we try to understand which activities and information are needed for selecting requirements, which the challenges are, how an ideal solution for selecting requirements would look like, and where the current state of the art is deficient regarding the ideal solution. Within this thesis we identify the information necessary for an informed requirements selection, present a process in which one collects all the necessary information, highlight the challenges to be addressed by this process and its activities, and a selection of methods to conduct the activities of the process. All the collected information is then used for an automated requirements selection using an optimization model which is also part of the contribution of this thesis. As we identified two major gaps in the state of the art considering the proposed process and its activities, we also present two novel methods for context elicitation and for legal compliance requirements elicitation to fill the gaps as part of the main contribution. Our solution for context elicitation enables a domain-specific context establishment based on patterns for different domains. The context patterns allow a structured elicitation and documentation of relevant stakeholders and technical entities for a system-to-be. Both, the documentation in means of graphical pattern instances and textual template instances as well as the method for collecting the necessary information are explicitly given in each context pattern. Additionally, we also provide the means which are necessary to derive new context patterns and extend our context patterns language which is part of this thesis. Our solution for legal compliance requirements elicitation is a pattern-based and guided method which lets one identify the relevant laws for a system-to-be, which is described in means of functional requirements, and which intertwines the functional requirements with the according legal requirements. This method relies on the collaboration of requirements engineers and legal experts, and bridges the gap between their distinct worlds. Our process is exemplified using a running example in the domain of service oriented architectures. Additionally, the results of applying (parts of) the process to real life cases from the smart grid domain and voting system domain are presented, as well as all other results from the scientific means we took to ground and validate the proposed solutions

Applying latent semantic analysis to computer assisted assessment in the Computer Science domain: a framework, a tool, and an evaluation

This dissertation argues that automated assessment systems can be useful for both students and educators provided that the results correspond well with human markers. Thus, evaluating such a system is crucial. I present an evaluation framework and show how and why it can be useful for both producers and consumers of automated assessment systems. The framework is a refinement of a research taxonomy that came out of the effort to analyse the literature review of systems based on Latent Semantic Analysis (LSA), a statistical natural language processing technique that has been used for automated assessment of essays. The evaluation framework can help developers publish their results in a format that is comprehensive, relatively compact, and useful to other researchers. The thesis claims that, in order to see a complete picture of an automated assessment system, certain pieces must be emphasised. It presents the framework as a jigsaw puzzle whose pieces join together to form the whole picture. The dissertation uses the framework to compare the accuracy of human markers and EMMA, the LSA-based assessment system I wrote as part of this dissertation. EMMA marks short, free text answers in the domain of computer science. I conducted a study of five human markers and then used the results as a benchmark against which to evaluate EMMA. An integral part of the evaluation was the success metric. The standard inter-rater reliability statistic was not useful; I located a new statistic and applied it to the domain of computer assisted assessment for the first time, as far as I know. Although EMMA exceeds human markers on a few questions, overall it does not achieve the same level of agreement with humans as humans do with each other. The last chapter maps out a plan for further research to improve EMMA