Verification and validation in software product line engineering

Verification and Validation (V&V) is currently performed during application development for many systems, especially safety-critical and mission-critical systems. However, the V&V process has been limited to single system development. This dissertation describes the extension of V&V from an individual application system to a product line of systems that are developed within an architecture-based software engineering environment.;In traditional V&V, the system provides the context under which the software will be evaluated, and V&V activities occur during all phases of the system development lifecycle. The transition to a product line approach to development removes the individual system as the context for evaluation, and introduces activities that are not directly related to a specific system. This dissertation presents an approach to V&V of software product lines that uses the domain model and the domain architecture as the context for evaluation, and enables V&V to be performed throughout the modified lifecycle introduced by domain engineering.;This dissertation presents three advances that assist in the adaptation of V&V from single application systems to a product line of systems. The first is a framework for performing V&V that includes the activities of traditional application-level V&V, and extends these activities into domain engineering and into the transition between domain engineering and application engineering. The second is a detailed method to extend the crucial V&V activity of criticality analysis from single system development to a product line of systems. The third advance is an approach to enable formal reasoning, which is needed for high assurance systems, on systems that are based on commercial-off-the-shelf (COTS) products

Software Engineering for Big Data Systems

Software engineering is the application of a systematic approach to designing, operating and maintaining software systems and the study of all the activities involved in achieving the same. The software engineering discipline and research into software systems flourished with the advent of computers and the technological revolution ushered in by the World Wide Web and the Internet. Software systems have grown dramatically to the point of becoming ubiquitous. They have a significant impact on the global economy and on how we interact and communicate with each other and with computers using software in our daily lives. However, there have been major changes in the type of software systems developed over the years. In the past decade owing to breakthrough advancements in cloud and mobile computing technologies, unprecedented volumes of hitherto inaccessible data, referred to as big data, has become available to technology companies and business organizations farsighted and discerning enough to use it to create new products, and services generating astounding profits. The advent of big data and software systems utilizing big data has presented a new sphere of growth for the software engineering discipline. Researchers, entrepreneurs and major corporations are all looking into big data systems to extract the maximum value from data available to them. Software engineering for big data systems is an emergent field that is starting to witness a lot of important research activity. This thesis investigates the application of software engineering knowledge areas and standard practices, established over the years by the software engineering research community, into developing big data systems by: - surveying the existing software engineering literature on applying software engineering principles into developing and supporting big data systems; - identifying the fields of application for big data systems; - investigating the software engineering knowledge areas that have seen research related to big data systems; - revealing the gaps in the knowledge areas that require more focus for big data systems development; and - determining the open research challenges in each software engineering knowledge area that need to be met. The analysis and results obtained from this thesis reveal that recent advances made in distributed computing, non-relational databases, and machine learning applications have lured the software engineering research and business communities primarily into focusing on system design and architecture of big data systems. Despite the instrumental role played by big data systems in the success of several businesses organizations and technology companies by transforming them into market leaders, developing and maintaining stable, robust, and scalable big data systems is still a distant milestone. This can be attributed to the paucity of much deserved research attention into more fundamental and equally important software engineering activities like requirements engineering, testing, and creating good quality assurance practices for big data systems

Systematic Model-based Design Assurance and Property-based Fault Injection for Safety Critical Digital Systems

With advances in sensing, wireless communications, computing, control, and automation technologies, we are witnessing the rapid uptake of Cyber-Physical Systems across many applications including connected vehicles, healthcare, energy, manufacturing, smart homes etc. Many of these applications are safety-critical in nature and they depend on the correct and safe execution of software and hardware that are intrinsically subject to faults. These faults can be design faults (Software Faults, Specification faults, etc.) or physically occurring faults (hardware failures, Single-event-upsets, etc.). Both types of faults must be addressed during the design and development of these critical systems. Several safety-critical industries have widely adopted Model-Based Engineering paradigms to manage the design assurance processes of these complex CPSs. This thesis studies the application of IEC 61508 compliant model-based design assurance methodology on a representative safety-critical digital architecture targeted for the Nuclear power generation facilities. The study presents detailed experiences and results to demonstrate the benefits of Model testing in finding design flaws and its relevance to subsequent verification steps in the workflow. Additionally, to study the impact of physical faults on the digital architecture we develop a novel property-based fault injection method that overcomes few deficiencies of traditional fault injection methods. The model-based fault injection approach presented here guarantees high efficiency and near-exhaustive input/state/fault space coverage, by utilizing formal model checking principles to identify fault activation conditions and prove the fault tolerance features. The fault injection framework facilitates automated integration of fault saboteurs throughout the model to enable exhaustive fault location coverage in the model

Mission Assurance for Autonomous Underwater Vehicles

The ubiquity of autonomous vehicles (AVs) is all but inevitable, and AVs have made fantastic leaps in their capabilities, partly thanks to advances in artificial intelligence and machine learning (AI/ML). With these great capabilities should come great assurance that AVs will behave safely and achieve their operational goals, or mission, despite foreseen and unforeseen circumstances. AV software is highly complex, increasing the likelihood of faults. AI/ML decision making is poorly understood. And, all computer-based systems are vulnerable to malicious software and other cybersecurity threats. Eliminating or mitigating any one of these is an open research problem. AVs must handle all three, without the benefit of a human operator. This dissertation investigates several aspects of AV mission assurance, and offers solutions for test and evaluation starting early in the development cycle, a use case with which to experiment, and a methodology for iteratively improving assurance as more is learned about a mission and its specific risks. This dissertation focuses on autonomous underwater vehicles (AUVs). Each chapter explores particular aspects of AUV mission assurance and presents approaches to address them. We discuss the risks specific to AUV safety and mission assurance. We introduce the Digital Environment for Simulated Cyber Resilience Engineering, Test and Experimentation (DESCRETE) testbed that enables cost-effective AUV simulation, particularly with respect to system-level faults and attacks. We present the mission-assured AUV (MAAUV) use case, which we used to gather data on DESCRETE to improve the testbed and better understand mission assurance. We propose an iterative mission-assurance refinement analysis (IMARA) methodology for understanding system-failure impacts to mission. Applying IMARA to the MAAUV, we provide a guide for AUV and mission designers to best use limited assurance improvement and mitigation resources. Combining all these provides a comprehensive set of tools to improve AUV assurance

Multilevel Runtime Verification for Safety and Security Critical Cyber Physical Systems from a Model Based Engineering Perspective

Advanced embedded system technology is one of the key driving forces behind the rapid growth of Cyber-Physical System (CPS) applications. CPS consists of multiple coordinating and cooperating components, which are often software-intensive and interact with each other to achieve unprecedented tasks. Such highly integrated CPSs have complex interaction failures, attack surfaces, and attack vectors that we have to protect and secure against. This dissertation advances the state-of-the-art by developing a multilevel runtime monitoring approach for safety and security critical CPSs where there are monitors at each level of processing and integration. Given that computation and data processing vulnerabilities may exist at multiple levels in an embedded CPS, it follows that solutions present at the levels where the faults or vulnerabilities originate are beneficial in timely detection of anomalies. Further, increasing functional and architectural complexity of critical CPSs have significant safety and security operational implications. These challenges are leading to a need for new methods where there is a continuum between design time assurance and runtime or operational assurance. Towards this end, this dissertation explores Model Based Engineering methods by which design assurance can be carried forward to the runtime domain, creating a shared responsibility for reducing the overall risk associated with the system at operation. Therefore, a synergistic combination of Verification & Validation at design time and runtime monitoring at multiple levels is beneficial in assuring safety and security of critical CPS. Furthermore, we realize our multilevel runtime monitor framework on hardware using a stream-based runtime verification language

An Assurance Framework for Independent Co-assurance of Safety and Security

Integrated safety and security assurance for complex systems is difficult for many technical and socio-technical reasons such as mismatched processes, inadequate information, differing use of language and philosophies, etc.. Many co-assurance techniques rely on disregarding some of these challenges in order to present a unified methodology. Even with this simplification, no methodology has been widely adopted primarily because this approach is unrealistic when met with the complexity of real-world system development. This paper presents an alternate approach by providing a Safety-Security Assurance Framework (SSAF) based on a core set of assurance principles. This is done so that safety and security can be co-assured independently, as opposed to unified co-assurance which has been shown to have significant drawbacks. This also allows for separate processes and expertise from practitioners in each domain. With this structure, the focus is shifted from simplified unification to integration through exchanging the correct information at the right time using synchronisation activities