Multipath/RFI/modulation study for DRSS-RFI problem: Voice coding and intelligibility testing for a satellite-based air traffic control system

Analog and digital voice coding techniques for application to an L-band satellite-basedair traffic control (ATC) system for over ocean deployment are examined. In addition to performance, the techniques are compared on the basis of cost, size, weight, power consumption, availability, reliability, and multiplexing features. Candidate systems are chosen on the bases of minimum required RF bandwidth and received carrier-to-noise density ratios. A detailed survey of automated and nonautomated intelligibility testing methods and devices is presented and comparisons given. Subjective evaluation of speech system by preference tests is considered. Conclusion and recommendations are developed regarding the selection of the voice system. Likewise, conclusions and recommendations are developed for the appropriate use of intelligibility tests, speech quality measurements, and preference tests with the framework of the proposed ATC system

Test-driven development of embedded control systems: application in an automotive collision prevention system

With test-driven development (TDD) new code is not written until an automated test has failed, and duplications of functions, tests, or simply code fragments are always removed. TDD can lead to a better design and a higher quality of the developed system, but to date it has mainly been applied to the development of traditional software systems such as payroll applications. This thesis describes the novel application of TDD to the development of embedded control systems using an automotive safety system for preventing collisions as an example. The basic prerequisite for test-driven development is the availability of an automated testing framework as tests are executed very often. Such testing frameworks have been developed for nearly all programming languages, but not for the graphical, signal driven language Simulink. Simulink is commonly used in the automotive industry and can be considered as state-of-the-art for the design and development of embedded control systems in the automotive, aerospace and other industries. The thesis therefore introduces a novel automated testing framework for Simulink. This framework forms the basis for the test-driven development process by integrating the analysis, design and testing of embedded control systems into this process. The thesis then shows the application of TDD to a collision prevention system. The system architecture is derived from the requirements of the system and four software components are identified, which represent problems of particular areas for the realisation of control systems, i.e. logical combinations, experimental problems, mathematical algorithms, and control theory. For each of these problems, a concept to systematically derive test cases from the requirements is presented. Moreover two conventional approaches to design the controller are introduced and compared in terms of their stability and performance. The effectiveness of the collision prevention system is assessed in trials on a driving simulator. These trials show that the system leads to a significant reduction of the accident rate for rear-end collisions. In addition, experiments with prototype vehicles on test tracks and field tests are presented to verify the system’s functional requirements within a system testing approach. Finally, the new test-driven development process for embedded control systems is evaluated in comparison to traditional development processes

Automated testing with Wireless Communication in the digitalised industry : A case study of Mirka Oy

Advanced automation technologies are changing the dynamics of the process and manufacturing industries. Product development processes are becoming smarter with the application of intelligent solutions and automated testing. The industry 4.0 concept of centralized control for industrial devices results in a rapid increase in the demand for the industrial Internet of Things (IoT) and cordless machines. Wireless communication protocols are integral to the functioning of such devices. This thesis work is performed with Mirka Oy during the development process of a smart industrial cordless tool. Various available short-range wireless communication protocols are studied to find out the best possible solution to match the product requirements. Besides, an automated testing platform is developed to verify and validate the functional description of the devices. All the stages, starting from the types of embedded system testing, device test requirements, test case designing leading to a comprehensive testing platform are explained. Results generated by the automated platform are analysed, which shows that all the test execution is successful. The successful implantation of this automated testing platform would significantly increase the efficiency of the development and testing process. Moreover, this dissertation highlights further development in terms of the application of the Artificial Intelligence (AI) and Machine learning (ML) technique for smarter testing processes and increase the overall performance of the testing framework

A Structured Testing Framework for ADAS Software Development

A major task in the design of automated vehicles is the need to quickly and thoroughly validate a development teams algorithms. There currently exists no explicitly defined common standard for developers working on Advanced Driver Assisted Systems to adopt during their software testing process. Instead different teams customize their testing process specifically to their software systems current needs. Literature indicates that these processes can be comprehensive but convoluted, and not flexible to change as test requirements and the system itself does. This thesis introduces a test framework at the unit, integration, and system test levels with the objective of addressing these challenges through a complete test framework centered around rapid execution and modular test design. At the unit test level a recommendation guide is put forth that is largely aimed at new developers with concrete actionable items that can be integrated into a teams process. For integration and system level testing, a software solution for ROS based development referred to as University of Waterloo Structured Testing Framework (UW-STF) is described in regards to both the benefits it provides as well as its low level implementation details. This includes how to tie the framework into using data generated from the popular simulator CARLA for end-to-end testing of a system. Lastly the test framework is applied to the codebase of UWAFT for their development efforts related to connected and automated vehicles. The framework was shown to increase readability/clarity at the unit test level, facilitate robust automated testing at the integration level and provide transparency on the teams current algorithms performance at the system test level (average F1-score of 0.77 and average OSPA of 2.42). When compared to the standard ROS integration test framework, UW-STF executed the same test suite with 60%+ reduction in lines of code and meaningful differences in CPU and memory requirements

Automated System Performance Testing at MongoDB

Distributed Systems Infrastructure (DSI) is MongoDB's framework for running fully automated system performance tests in our Continuous Integration (CI) environment. To run in CI it needs to automate everything end-to-end: provisioning and deploying multi-node clusters, executing tests, tuning the system for repeatable results, and collecting and analyzing the results. Today DSI is MongoDB's most used and most useful performance testing tool. It runs almost 200 different benchmarks in daily CI, and we also use it for manual performance investigations. As we can alert the responsible engineer in a timely fashion, all but one of the major regressions were fixed before the 4.2.0 release. We are also able to catch net new improvements, of which DSI caught 17. We open sourced DSI in March 2020.Comment: Author Preprint. Appearing in DBTest.io 202

Safety-critical scenarios and virtual testing procedures for automated cars at road intersections

This thesis addresses the problem of road intersection safety with regard to a mixed population of automated vehicles and non-automated road users. The work derives and evaluates safety-critical scenarios at road junctions, which can pose a particular safety problem involving automated cars. A simulation and evaluation framework for car-to-car accidents is presented and demonstrated, which allows examining the safety performance of automated driving systems within those scenarios. Given the recent advancements in automated driving functions, one of the main challenges is safe and efficient operation in complex traffic situations such as road junctions. There is a need for comprehensive testing, either in virtual testing environments or on real-world test tracks. Since it is unrealistic to cover all possible combinations of traffic situations and environment conditions, the challenge is to find the key driving situations to be evaluated at junctions. Against this background, a novel method to derive critical pre-crash scenarios from historical car accident data is presented. It employs k-medoids to cluster historical junction crash data into distinct partitions and then applies the association rules algorithm to each cluster to specify the driving scenarios in more detail. The dataset used consists of 1,056 junction crashes in the UK, which were exported from the in-depth On-the-Spot database. The study resulted in thirteen crash clusters for T-junctions, and six crash clusters for crossroads. Association rules revealed common crash characteristics, which were the basis for the scenario descriptions. As a follow-up to the scenario generation, the thesis further presents a novel, modular framework to transfer the derived collision scenarios to a sub-microscopic traffic simulation environment. The software CarMaker is used with MATLAB/Simulink to simulate realistic models of vehicles, sensors and road environments and is combined with an advanced Monte Carlo method to obtain a representative set of parameter combinations. The analysis of different safety performance indicators computed from the simulation outputs reveals collision and near-miss probabilities for selected scenarios. The usefulness and applicability of the simulation and evaluation framework is demonstrated for a selected junction scenario, where the safety performance of different in-vehicle collision avoidance systems is studied. The results show that the number of collisions and conflicts were reduced to a tenth when adding a crossing and turning assistant to a basic forward collision avoidance system. Due to its modular architecture, the presented framework can be adapted to the individual needs of future users and may be enhanced with customised simulation models. Ultimately, the thesis leads to more efficient workflows when virtually testing automated driving at intersections, as a complement to field operational tests on public roads

Benchmarking Diagnostic Algorithms on an Electrical Power System Testbed

Diagnostic algorithms (DAs) are key to enabling automated health management. These algorithms are designed to detect and isolate anomalies of either a component or the whole system based on observations received from sensors. In recent years a wide range of algorithms, both model-based and data-driven, have been developed to increase autonomy and improve system reliability and affordability. However, the lack of support to perform systematic benchmarking of these algorithms continues to create barriers for effective development and deployment of diagnostic technologies. In this paper, we present our efforts to benchmark a set of DAs on a common platform using a framework that was developed to evaluate and compare various performance metrics for diagnostic technologies. The diagnosed system is an electrical power system, namely the Advanced Diagnostics and Prognostics Testbed (ADAPT) developed and located at the NASA Ames Research Center. The paper presents the fundamentals of the benchmarking framework, the ADAPT system, description of faults and data sets, the metrics used for evaluation, and an in-depth analysis of benchmarking results obtained from testing ten diagnostic algorithms on the ADAPT electrical power system testbed