Building sound COTS products taxonomies

The acquisition of process of COTS products requires the identification of the suitable products. The evaluation of COTS products varies from type to type, e.g. a DBMS has to be evaluated differently from a web browser. The techniques used to integrate the COTS products are strongly dependent on the type of products. The same applies to the architectural patterns and style used to build COTS-based systems. So it is important to be able to classify COTS products into homogeneous classes. Taxonomies are a tool to perform such a classification.Peer ReviewedPostprint (published version

Uncertainty explicit assessment of off-the-shelf software: A Bayesian approach

Assessment of software COTS components is an essential part of component-based software development. Poorly chosen components may lead to solutions of low quality and that are difficult to maintain. The assessment may be based on incomplete knowledge about the COTS component itself and other aspects (e.g. vendor’s credentials, etc.), which may affect the decision of selecting COTS component(s). We argue in favor of assessment methods in which uncertainty is explicitly represented (‘uncertainty explicit’ methods) using probability distributions. We provide details of a Bayesian model, which can be used to capture the uncertainties in the simultaneous assessment of two attributes, thus, also capturing the dependencies that might exist between them. We also provide empirical data from the use of this method for the assessment of off-the-shelf database servers which illustrate the advantages of ‘uncertainty explicit’ methods over conventional methods of COTS component assessment which assume that at the end of the assessment the values of the attributes become known with certainty

Open Source Software Acquisition: Beyond the Business Case

Lately we have seen a growing interest from both public and private organisations to adopt Open Source Software (OSS), not only for a few, specific applications but also on a more general level throughout the organisation. As a consequence, the organisations’ decisions on adoption of OSS are becoming increasingly more important and complex. We present three perspectives organisations can employ in their decisions: seeing OSS acquisition as a business case, as COTS acquisition, and as architectural change within a governance framework. We present case studies of decisions on OSS adoption, and categorise the decision criteria we have found. Our results indicate that for large-scale adoption of OSS, focus will be on architectural considerations: enterprise-wide architectures will at first be a barrier, but in the long term OSS’s support of open standards can be a major enabler for OSS adoption. In contrast, in smaller organisations and in small-scale adoption of OSS, the cheap price of OSS is a major enabler, as it provides a good opportunity for experiments and short-term economic benefits. For small organisations these experiments can lead to development of a common IT-architecture, and in larger organisations OSS can be adopted in niche-areas, without significantly violating an existing IT-architecture

A framework for cots software evaluation and selection for COTS mismatches handling and non-functional requirements

The decision to purchase Commercial Off-The-Shelf (COTS) software needs systematic guidelines so that the appropriate COTS software can be selected in order to provide a viable and effective solution to the organizations. However, the existing COTS software evaluation and selection frameworks focus more on functional aspects and do not give adequate attention to accommodate the mismatch between user requirements and COTS software specification, and also integration with non functional requirements of COTS software. Studies have identified that these two criteria are important in COTS software evaluation and selection. Therefore, this study aims to develop a new framework of COTS software evaluation and selection that focuses on handling COTS software mismatches and integrating the nonfunctional requirements. The study is conducted using mixed-mode methodology which involves survey and interview. The study is conducted in four main phases: a survey and interview of 63 organizations to identify COTS software evaluation criteria, development of COTS software evaluation and selection framework using Evaluation Theory, development of a new decision making technique by integrating Analytical Hierarchy Process and Gap Analysis to handle COTS software mismatches, and validation of the practicality and reliability of the proposed COTS software Evaluation and Selection Framework (COTS-ESF) using experts’ review, case studies and yardstick validation. This study has developed the COTS-ESF which consists of five categories of evaluation criteria: Quality, Domain, Architecture, Operational Environment and Vendor Reputation. It also provides a decision making technique and a complete process for performing the evaluation and selection of COTS software. The result of this study shows that the evaluated aspects of the framework are feasible and demonstrate their potential and practicality to be applied in the real environment. The contribution of this study straddles both the research and practical perspectives of software evaluation by improving decision making and providing a systematic guidelines for handling issue in purchasing viable COTS software

Requirements patterns for COTS systems

The reuse of knowledge obtained during the elicitation of requirements for different COTS projects is a subject that still needs more research to be done. In this work we propose the use of Patterns of Requirements for the first steps of the COTS selection processes and the software life cycle.Peer ReviewedPostprint (published version

Beyond the Business Case

Lately we have seen a growing interest from both public and private organisations to adopt Open Source Software (OSS), not only for a few, specific applications but also on a more general level throughout the organisation. As a consequence, the organisations’ decisions on adoption of OSS are becoming increasingly more important and complex. We present three perspectives organisations can employ in their decisions: seeing OSS acquisition as a business case, as COTS acquisition, and as architectural change within a governance framework. We present case studies of decisions on OSS adoption, and categorise the decision criteria we have found. Our results indicate that for large-scale adoption of OSS, focus will be on architectural considerations: enterprise-wide architectures will at first be a barrier, but in the long term OSS’s support of open standards can be a major enabler for OSS adoption. In contrast, in smaller organisations and in small-scale adoption of OSS, the cheap price of OSS is a major enabler, as it provides a good opportunity for experiments and short-term economic benefits. For small organisations these experiments can lead to development of a common IT-architecture, and in larger organisations OSS can be adopted in niche-areas, without significantly violating an existing IT-architecture. Keywords: open source, COTS, IT architecture, governanc

Learning Non-robustness using Simulation-based Testing: a Network Traffic-shaping Case Study

An input to a system reveals a non-robust behaviour when, by making a small change in the input, the output of the system changes from acceptable (passing) to unacceptable (failing) or vice versa. Identifying inputs that lead to non-robust behaviours is important for many types of systems, e.g., cyber-physical and network systems, whose inputs are prone to perturbations. In this paper, we propose an approach that combines simulation-based testing with regression tree models to generate value ranges for inputs in response to which a system is likely to exhibit non-robust behaviours. We apply our approach to a network traffic-shaping system (NTSS) -- a novel case study from the network domain. In this case study, developed and conducted in collaboration with a network solutions provider, RabbitRun Technologies, input ranges that lead to non-robustness are of interest as a way to identify and mitigate network quality-of-service issues. We demonstrate that our approach accurately characterizes non-robust test inputs of NTSS by achieving a precision of 84% and a recall of 100%, significantly outperforming a standard baseline. In addition, we show that there is no statistically significant difference between the results obtained from our simulated testbed and a hardware testbed with identical configurations. Finally we describe lessons learned from our industrial collaboration, offering insights about how simulation helps discover unknown and undocumented behaviours as well as a new perspective on using non-robustness as a measure for system re-configuration.Comment: This paper is accepted at the 16th IEEE International Conference on Software Testing, Verification and Validation (ICST 2023

Model-based integration testing technique using formal finite state behavioral models for component-based software

Many issues and challenges could be identified when considering integration testing of Component-Based Software Systems (CBSS). Consequently, several research have appeared in the literature, aimed at facilitating the integration testing of CBSS. Unfortunately, they suffer from a number of drawbacks and limitations such as difficulty of understanding and describing the behavior of integrated components, lack of effective formalism for test information, difficulty of analyzing and validating the integrated components, and exposing the components implementation by providing semi-formal models. Hence, these problems have made it in effective to test today’s modern complex CBSS. To address these problems, a model-based approach such as Model-Based Testing (MBT) tends to be a suitable mechanism and could be a potential solution to be applied in the context of integration testing of CBSS. Accordingly, this thesis presents a model-based integration testing technique for CBSS. Firstly, a method to extract the formal finite state behavioral models of integrated software components using Mealy machine models was developed. The extracted formal models were used to detect faulty interactions (integration bugs) or compositional problems between integrated components in the system. Based on the experimental results, the proposed method had significant impact in reducing the number of output queries required to extract the formal models of integrated software components and its performance was 50% better compared to the existing methods. Secondly, based on the extracted formal models, an effective model-based integration testing technique (MITT) for CBSS was developed. Finally, the effectiveness of the MITT was demonstrated by employing it in the air gourmet and elevator case studies, using three evaluation parameters. The experimental results showed that the MITT was effective and outperformed Shahbaz technique on the air gourmet and elevator case studies. In terms of learned components for air gourmet and elevator case studies respectively, the MITT results were better by 98.14% and 100%, output queries based on performance were 42.13% and 25.01%, and error detection capabilities were 70.62% and 75% for each of the case study

An Analysis Framework for the Evaluation of Content Management Systems

The content published on the Web never stops growing. It is becoming essential to efficiently manage it. Organizations are increasingly searching for more efficient and feasible ways in which to manage the content of their websites. Content Management Systems offer a solution by facilitating the creation of websites and providing content management. Many organizations are now using Management Content Systems. This increasing interest is apparent in the growing market of such systems and in the updates of existing systems. For an organization, the choice of a Content Management System is not easy, since this is a dynamic market shared among many different publishers. The offer evaluation process requires specific tools. In order to help these organizations in their choice, we have attempted to provide an analysis framework for these applications. This analysis framework designed to help organizations choose their application contributes to the evaluation of these applications. This article puts forward a Content Management software analysis framework and explains its application to a sample of 23 products. The size of this sample allows us to asses the strategic grouping of Content Management Systems. Effectively, the analysis results in the identification of two strategic groups whose main differences lay in the software characteristics and their target markets

A survey on software testability

Context: Software testability is the degree to which a software system or a unit under test supports its own testing. To predict and improve software testability, a large number of techniques and metrics have been proposed by both practitioners and researchers in the last several decades. Reviewing and getting an overview of the entire state-of-the-art and state-of-the-practice in this area is often challenging for a practitioner or a new researcher. Objective: Our objective is to summarize the body of knowledge in this area and to benefit the readers (both practitioners and researchers) in preparing, measuring and improving software testability. Method: To address the above need, the authors conducted a survey in the form of a systematic literature mapping (classification) to find out what we as a community know about this topic. After compiling an initial pool of 303 papers, and applying a set of inclusion/exclusion criteria, our final pool included 208 papers. Results: The area of software testability has been comprehensively studied by researchers and practitioners. Approaches for measurement of testability and improvement of testability are the most-frequently addressed in the papers. The two most often mentioned factors affecting testability are observability and controllability. Common ways to improve testability are testability transformation, improving observability, adding assertions, and improving controllability. Conclusion: This paper serves for both researchers and practitioners as an "index" to the vast body of knowledge in the area of testability. The results could help practitioners measure and improve software testability in their projects