Systematic Testing of Embedded Automotive Software - The Classification-Tree Method for Embedded Systems (CTM/ES)

The software embedded in automotive control systems increasingly determines the functionality and properties of present-day motor vehicles. The development and test process of the systems and the software embedded becomes the limiting factor. While these challenges, on the development side, are met by employing model-based specification, design, and implementation techniques [KCF+04], satisfactory solutions on the testing side are slow in arriving. With regard to the systematic selection (test design) and the description of test scenarios especially, there is a lot of room for improvement. Thus, a main goal is to effectively minimize these deficits by creating an efficient procedure for the selection and description of test scenarios for embedded automotive software and its integration in the model-based development process. The realization of this idea involves the combination of a classical software testing procedure with a technology, prevalent in the automotive industry, which is used for the description of time-dependent stimuli signals. The result of this combination is the classification-tree method for embedded systems, CTM/ES [Con04]. The classification-tree method for embedded systems complements model-based development by employing a novel approach to the systematic selection and description of the test scenarios for the software embedded in the control systems. CTM/ES allows for the graphic representation of time-variable test scenarios on different levels of abstraction: A problem-oriented, compact representation, adequate for a human tester and containing a high potential for reusability, is gradually being transformed into a solution-oriented technical representation which is suited for the test objects\u27 stimulation. The CTM/ES notation facilitates a consistent representation of test scenarios which may result from different test design techniques. The test design technique which this method is primarily based on, is a data-oriented partitioning of the input domain in equivalence classes. Secondary test design techniques are, for instance, the testing of specific values (or value courses) or requirement-based testing. A domain-specific application pragmatics in the form of agendas supports the methodical execution of individual test activities and the interaction of different test design techniques. The methodology description leads up to an effective test strategy for model-based testing, combining the classification-tree method for embedded systems with structural testing on the model level, and accommodating the different forms of representation of the test object during model-based development. Systems which have been developed in a model-based way can be tested systematically and efficiently by means of the CTM/ES and the tools based thereon, such as the classification-tree editor for embedded systems CTE/ES [CTE/ES], as well as the model-based test environment MTest [LBE+04, MTest]

Development of a framework for automated systematic testing of safety-critical embedded systems

“This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. In most cases, these works may not be reposted without the explicit permission of the copyright holder." “Copyright IEEE. Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works must be obtained from the IEEE.”In this paper we introduce the development of a framework for testing safety-critical embedded systems based on the concepts of model-based testing. In model-based testing the test cases are derived from a model of the system under test. In our approach the model is an automaton model that is automatically extracted from the C-source code of the system under test. Beside random test data generation the test case generation uses formal methods, in detail model checking techniques. To find appropriate test cases we use the requirements defined in the system specification. To cover further execution paths we developed an additional, to our best knowledge, novel method based on special structural coverage criteria. We present preliminary results on the model extraction using a concrete industrial case study from the automotive domain

Formal verification of automotive embedded UML designs

Software applications are increasingly dominating safety critical domains. Safety critical domains are domains where the failure of any application could impact human lives. Software application safety has been overlooked for quite some time but more focus and attention is currently directed to this area due to the exponential growth of software embedded applications. Software systems have continuously faced challenges in managing complexity associated with functional growth, flexibility of systems so that they can be easily modified, scalability of solutions across several product lines, quality and reliability of systems, and finally the ability to detect defects early in design phases. AUTOSAR was established to develop open standards to address these challenges. ISO-26262, automotive functional safety standard, aims to ensure functional safety of automotive systems by providing requirements and processes to govern software lifecycle to ensure safety. Each functional system needs to be classified in terms of safety goals, risks and Automotive Safety Integrity Level (ASIL: A, B, C and D) with ASIL D denoting the most stringent safety level. As risk of the system increases, ASIL level increases and the standard mandates more stringent methods to ensure safety. ISO-26262 mandates that ASILs C and D classified systems utilize walkthrough, semi-formal verification, inspection, control flow analysis, data flow analysis, static code analysis and semantic code analysis techniques to verify software unit design and implementation. Ensuring software specification compliance via formal methods has remained an academic endeavor for quite some time. Several factors discourage formal methods adoption in the industry. One major factor is the complexity of using formal methods. Software specification compliance in automotive remains in the bulk heavily dependent on traceability matrix, human based reviews, and testing activities conducted on either actual production software level or simulation level. ISO26262 automotive safety standard recommends, although not strongly, using formal notations in automotive systems that exhibit high risk in case of failure yet the industry still heavily relies on semi-formal notations such as UML. The use of semi-formal notations makes specification compliance still heavily dependent on manual processes and testing efforts. In this research, we propose a framework where UML finite state machines are compiled into formal notations, specification requirements are mapped into formal model theorems and SAT/SMT solvers are utilized to validate implementation compliance to specification. The framework will allow semi-formal verification of AUTOSAR UML designs via an automated formal framework backbone. This semi-formal verification framework will allow automotive software to comply with ISO-26262 ASIL C and D unit design and implementation formal verification guideline. Semi-formal UML finite state machines are automatically compiled into formal notations based on Symbolic Analysis Laboratory formal notation. Requirements are captured in the UML design and compiled automatically into theorems. Model Checkers are run against the compiled formal model and theorems to detect counterexamples that violate the requirements in the UML model. Semi-formal verification of the design allows us to uncover issues that were previously detected in testing and production stages. The methodology is applied on several automotive systems to show how the framework automates the verification of UML based designs, the de-facto standard for automotive systems design, based on an implicit formal methodology while hiding the cons that discouraged the industry from using it. Additionally, the framework automates ISO-26262 system design verification guideline which would otherwise be verified via human error prone approaches

An automotive vehicle dynamics prototyping platform based on a remote control model car

The use of a modified remote control (RC) model car as a vehicle dynamics testing and development platform is detailed. Vehicle dynamics testing is an important aspect of automotive engineering and it plays a key role during the design and tuning of active safety control systems. Considering the fact that such tests are conducted at great expense, scaled model cars can potentially be used to help with the process to reduce the costs. With this view, we instrument and develop a standard electric RC model car into a vehicle dynamics testing platform. We then implement 2 representative active safety control applications based on this platform, namely an antilock brake system using open-loop pulse brake control and a roll-over prevention system utilizing lateral acceleration feedback. Both applications are presented with sensor measurements and the effectiveness of the suggested control algorithms are demonstrated. © TÜBİTAK

An analysis of the factors associated with adoption of electronic supply chain management (e-SCM) procurement systems within the U.S. automotive industry

The U.S. automotive industry is a vitally important industry to our nation; over 7,000,000 individuals directly rely on this industry for their livelihood (Hill, K. (2014). Just How High‐Tech is the Automotive Industry? Ann Arbor, MI: Center for Automotive Research (CAR)). This study investigated the use of electronic supply chain management (e-SCM) system usage within this industry and its influence on work-related procurement outcomes. An electronic questionnaire was used to gather perceptions consistent with the constructs of the technology acceptance model, or “TAM” (Davis, F. D. (1985). A technology acceptance model for empirically testing new end-user information systems: theory and results. Published Doctoral Dissertation, Massachusetts Institute of Technology, Cambridge, MA.) It examined e-SCM systems used in the automotive industry to test whether factors such as ease of system use (EOU) and perception of organizational usefulness (PU) are factors in regarding a decision to use e-SCM systems within the procurement function of their own organizations. The results of the survey analysis showed that the perceptions of organizational usefulness, as well as the ease of system use, are each variables affecting the final adoption of such systems within individual organizations. Further findings showed that these variables, along with other factors, including individual experience, educational level, and gender, were important in predicting work outcomes within this industry, in terms of overall results and predicting organizational decision outcomes. As such, this research offers an understanding of the factors that are critical to achieving continued innovation and corresponding industry success for organizations and individuals alike, transacting business at all supply chain levels within the U.S. automotive industry. Further research efforts should focus on the examination of the usage of this same model and corresponding survey instrument toward other manufacturing-based U.S. industries (such as aerospace/defense or pharmaceutical/medical) as well as other countries’ automotive industries, such as Germany and Japan, for generalizability perspectives

Intelligent fault detection and classification based on hybrid deep learning methods for Hardware-in-the-Loop test of automotive software systems

Hardware-in-the-Loop (HIL) has been recommended by ISO 26262 as an essential test bench for determining the safety and reliability characteristics of automotive software systems (ASSs). However, due to the complexity and the huge amount of data recorded by the HIL platform during the testing process, the conventional data analysis methods used for detecting and classifying faults based on the human expert are not realizable. Therefore, the development of effective means based on the historical data set is required to analyze the records of the testing process in an efficient manner. Even though data-driven fault diagnosis is superior to other approaches, selecting the appropriate technique from the wide range of Deep Learning (DL) techniques is challenging. Moreover, the training data containing the automotive faults are rare and considered highly confidential by the automotive industry. Using hybrid DL techniques, this study proposes a novel intelligent fault detection and classification (FDC) model to be utilized during the V-cycle development process, i.e., the system integration testing phase. To this end, an HIL-based real-time fault injection framework is used to generate faulty data without altering the original system model. In addition, a combination of the Convolutional Neural Network (CNN) and Long Short-Term Memory (LSTM) is employed to build the model structure. In this study, eight types of sensor faults are considered to cover the most common potential faults in the signals of ASSs. As a case study, a gasoline engine system model is used to demonstrate the capabilities and advantages of the proposed method and to verify the performance of the model. The results prove that the proposed method shows better detection and classification performance compared to other standalone DL methods. Specifically, the overall detection accuracies of the proposed structure in terms of precision, recall and F1-score are 98.86%, 98.90% and 98.88%, respectively. For classification, the experimental results also demonstrate the superiority under unseen test data with an average accuracy of 98.8%

Simulation of a visco-elastic damper based on the model of the vehicle shock absorber

This paper presents results of simulation of visco-elastic dampers based on the model of the vehicle shock absorber. The aim of this research is to determine the parameters of dynamic models of special dampers applicable in advanced vehicle suspension systems. One of the key issues of modeling is to determine the acceptable degree of the simplification. Model reduction is based on the determination of the damping coefficient suitable for testing vibration control systems. The diagnostic model of the shock absorber takes into account features such as nonlinearity and asymmetry of damping characteristics. It is also important to take into account the apparent elasticity of the hydraulic damper. Damping and stiffness characteristics of automotive shock absorbers are identified by the proposed algorithm. The obtained results proove that the resulting damper stiffness is an important factor which cannot be neglected at higher frequencies of cyclic load

GRU-based denoising autoencoder for detection and clustering of unknown single and concurrent faults during system integration testing of automotive software systems

Recently, remarkable successes have been achieved in the quality assurance of automotive software systems (ASSs) through the utilization of real-time hardware-in-the-loop (HIL) simulation. Based on the HIL platform, safe, flexible and reliable realistic simulation during the system development process can be enabled. However, notwithstanding the test automation capability, large amounts of recordings data are generated as a result of HIL test executions. Expert knowledge-based approaches to analyze the generated recordings, with the aim of detecting and identifying the faults, are costly in terms of time, effort and difficulty. Therefore, in this study, a novel deep learning-based methodology is proposed so that the faults of automotive sensor signals can be efficiently and automatically detected and identified without human intervention. Concretely, a hybrid GRU-based denoising autoencoder (GRU-based DAE) model with the k-means algorithm is developed for the fault-detection and clustering problem in sequential data. By doing so, based on the real-time historical data, not only individual faults but also unknown simultaneous faults under noisy conditions can be accurately detected and clustered. The applicability and advantages of the proposed method for the HIL testing process are demonstrated by two automotive case studies. To be specific, a high-fidelity gasoline engine and vehicle dynamic system along with an entire vehicle model are considered to verify the performance of the proposed model. The superiority of the proposed architecture compared to other autoencoder variants is presented in the results in terms of reconstruction error under several noise levels. The validation results indicate that the proposed model can perform high detection and clustering accuracy of unknown faults compared to stand-alone techniques

Complete Model-Based Testing Applied to the Railway Domain

Testing is the most important verification technique to assert the correctness of an embedded system. Model-based testing (MBT) is a popular approach that generates test cases from models automatically. For the verification of safety-critical systems, complete MBT strategies are most promising. Complete testing strategies can guarantee that all errors of a certain kind are revealed by the generated test suite, given that the system-under-test fulfils several hypotheses. This work presents a complete testing strategy which is based on equivalence class abstraction. Using this approach, reactive systems, with a potentially infinite input domain but finitely many internal states, can be abstracted to finite-state machines. This allows for the generation of finite test suites providing completeness. However, for a system-under-test, it is hard to prove the validity of the hypotheses which justify the completeness of the applied testing strategy. Therefore, we experimentally evaluate the fault-detection capabilities of our equivalence class testing strategy in this work. We use a novel mutation-analysis strategy which introduces artificial errors to a SystemC model to mimic typical HW/SW integration errors. We provide experimental results that show the adequacy of our approach considering case studies from the railway domain (i.e., a speed-monitoring function and an interlocking-system controller) and from the automotive domain (i.e., an airbag controller). Furthermore, we present extensions to the equivalence class testing strategy. We show that a combination with randomisation and boundary-value selection is able to significantly increase the probability to detect HW/SW integration errors

A Comprehensive Safety Engineering Approach for Software-Intensive Systems Based on STPA

Formal verification and testing are complementary approaches which are used in the development process to verify the functional correctness of software. However, the correctness of software cannot ensure the safe operation of safety-critical software systems. The software must be verified against its safety requirements which are identified by safety analysis, to ensure that potential hazardous causes cannot occur. The complexity of software makes defining appropriate software safety requirements with traditional safety analysis techniques difficult. STPA (Systems-Theoretic Processes Analysis) is a unique safety analysis approach that has been developed to identify system hazards, including the software-related hazards. This paper presents a comprehensive safety engineering approach based on STPA, including software testing and model checking approaches for the purpose of developing safe software. The proposed approach can be embedded within a defined software engineering process or applied to existing software systems, allow software and safety engineers integrate the analysis of software risks with their verification. The application of the proposed approach is illustrated with an automotive software controller