369,350 research outputs found

    Test Maintenance for Machine Learning Systems: A Case Study in the Automotive Industry

    Get PDF
    Machine Learning (ML) systems have seen widespread use for automated decision making. Testing is essential to ensure the quality of these systems, especially safety-critical autonomous systems in the automotive domain. ML systems introduce new challenges with the potential to affect test maintenance, the process of updating test cases to match the evolving system. We conducted an exploratory case study in the automotive domain to identify factors that affect test maintenance for ML systems, as well as to make recommendations to improve the maintenance process. Based on interview and artifact analysis, we identified 14 factors affecting maintenance, including five especially relevant for ML systems—with the most important relating to non-determinism and large input spaces. We also proposed ten recommendations for improving test maintenance, including four targeting ML systems—in particular, emphasizing the use of test oracles tolerant to acceptable non-determinism. The study’s findings expand our knowledge of test maintenance for an emerging class of systems, benefiting the practitioners testing these systems

    Conceptual biology, hypothesis discovery, and text mining: Swanson's legacy

    Get PDF
    Innovative biomedical librarians and information specialists who want to expand their roles as expert searchers need to know about profound changes in biology and parallel trends in text mining. In recent years, conceptual biology has emerged as a complement to empirical biology. This is partly in response to the availability of massive digital resources such as the network of databases for molecular biologists at the National Center for Biotechnology Information. Developments in text mining and hypothesis discovery systems based on the early work of Swanson, a mathematician and information scientist, are coincident with the emergence of conceptual biology. Very little has been written to introduce biomedical digital librarians to these new trends. In this paper, background for data and text mining, as well as for knowledge discovery in databases (KDD) and in text (KDT) is presented, then a brief review of Swanson's ideas, followed by a discussion of recent approaches to hypothesis discovery and testing. 'Testing' in the context of text mining involves partially automated methods for finding evidence in the literature to support hypothetical relationships. Concluding remarks follow regarding (a) the limits of current strategies for evaluation of hypothesis discovery systems and (b) the role of literature-based discovery in concert with empirical research. Report of an informatics-driven literature review for biomarkers of systemic lupus erythematosus is mentioned. Swanson's vision of the hidden value in the literature of science and, by extension, in biomedical digital databases, is still remarkably generative for information scientists, biologists, and physicians. © 2006Bekhuis; licensee BioMed Central Ltd

    A Rapidly Deployable Classification System using Visual Data for the Application of Precision Weed Management

    Full text link
    In this work we demonstrate a rapidly deployable weed classification system that uses visual data to enable autonomous precision weeding without making prior assumptions about which weed species are present in a given field. Previous work in this area relies on having prior knowledge of the weed species present in the field. This assumption cannot always hold true for every field, and thus limits the use of weed classification systems based on this assumption. In this work, we obviate this assumption and introduce a rapidly deployable approach able to operate on any field without any weed species assumptions prior to deployment. We present a three stage pipeline for the implementation of our weed classification system consisting of initial field surveillance, offline processing and selective labelling, and automated precision weeding. The key characteristic of our approach is the combination of plant clustering and selective labelling which is what enables our system to operate without prior weed species knowledge. Testing using field data we are able to label 12.3 times fewer images than traditional full labelling whilst reducing classification accuracy by only 14%.Comment: 36 pages, 14 figures, published Computers and Electronics in Agriculture Vol. 14

    Design, Application and Evaluation of a Multi Agent System in the Logistics Domain

    Full text link
    The increasing demand for flexibility of automated production systems also affects the automated material flow systems (aMFS) they contain and demands reconfigurable systems. However, the centralized control concept usually applied in aMFS hinders an easy adaptation, as the entire control software has to be re-tested, when manually changing sub-parts of the control. As adaption and subsequent testing are a time-consuming task, concepts for splitting the control from one centralized to multiple, decentralized control nodes are required. Therefore, this paper presents a holistic agent-based control concept for aMFS, whereby the system is divided into so-called automated material flow modules (aMFM), each being controlled by a dedicated module agent. The concept allows the reconfiguration of aMFS, consisting of heterogeneous, stationary aMFM, during runtime. Furthermore, it includes aspects such as uniform agent knowledge bases through metamodel-based development, a communication ontology considering different information types and properties, strategic route optimization in decentralized control architecture and a visualization concept to make decisions of the module agents comprehensible to operators and maintenance staff. The evaluation of the concept is performed by means of material flow simulations as well as a prototypical implementation on a lab-sized demonstrator.Comment: 13 pages, https://ieeexplore.ieee.org/abstract/document/9042827

    Assessing safety functionalities in the design and validation of driving automation

    Get PDF
    This paper aims to contribute to the comprehensive and systematic safety assessment of Automated Driving Systems (ADSs) by identifying unknown hazardous areas of operation. The current methodologies employed in this domain typically involve estimating the distributions of situational variables based on human-centered field test, crash databases, or expert knowledge of critical values. However, due to the lack of a-priori knowledge regarding the influential factors, their critical ranges, and their distributions, these approaches may not be entirely suitable for the assessment of emerging automated driving technologies. To deal with this challenging problem, here we propose a testing methodology incorporating realistic yet unobserved driving conditions, distinguished by numerous situational variables, so to encompass unknown unsafe conditions comprehensively. Our methodology utilizes stochastic simulation and uncertainty modeling techniques to account for the variability of realistic driving conditions and their impact on ADSs' performances. By doing so, we aim to identify unsafe operational regions and triggering conditions that can lead to hazardous behaviors, thus improving the development and safety of automated driving functions. For our purposes, the Latin Hypercube Sampling technique and the recently proposed PAWN density-based sensitivity analysis method are employed. We apply this methodology for the first time in the specific field of ADSs design and validation, using an exemplificative use case. We discuss and compare the results obtained from our approach with those obtained from a traditional approach

    Guarantee of quality of electronic testirvaniya in the system of the additional vocational education

    Full text link
    Today becomes urgent the creation of the information texnologies, based on the systems approach to the estimation of test tasks. For this it is expedient to develop test programs for the checking of the standard of knowledge of listeners. Undoubtedly, the automated evaluation of knowledge occupies here fundamental importance. We developed innovation approach to the form of the idea of the results of electronic testing. The special importance takes the form of the idea of the result of testing, which gives advantage during the guarantee of quality of educational services. It is shown that the test technologies increase the quality of instruction in the system of additional vocational educationСегодня становится актуальным создание информационных технологий, основанных на системном подходе к оценке тестовых заданий. Для этого целесообразно разработать тест-программы для контроля уровня знаний слушателей. Несомненно, автоматизированная оценка знаний занимает здесь первостепенное значение. Мы разработали инновационный подход к форме представления результатов электронного тестирования. Особое значение имеет вид представления результата тестирования, которое дает преимущество при обеспечении качества образовательных услуг. Показано, что тестовые технологии повышают качество обучения в системе дополнительного профессионального образовани

    MODELS OF INTEGRATION OF INFORMATION SYSTEMS IN HIGHER EDUCATION INSTITUTIONS

    Get PDF
    At present a lot of automated systems are developing and implementing to support the educational and research processes in the universities. Often these systems duplicate some functions, databases, and also there are problems of compatibility of these systems. The most common educational systems are systems for creating electronic libraries, access to scientific and educational information, a program for detecting plagiarism, testing knowledge, etc. In this article, models and solutions for the integration of such educational automated systems as the information library system (ILS) and the anti-plagiarism system are examined. Integration of systems is based on the compatibility of databases, if more precisely in the metadata of different information models. At the same time, Cloud technologies are used - data processing technology, in which computer resources are provided to the user of the integrated system as an online service. ILS creates e-library of graduation papers and dissertations on the main server. During the creation of the electronic catalog, the communication format MARC21 is used. The database development is distributed for each department. The subsystem of anti-plagiarism analyzes the full-text database for the similarity of texts (dissertations, diploma works and others). Also it identifies the percentage of coincidence, creates the table of statistical information on the coincidence of tests for each author and division, indicating similar fields. The integrated system was developed and tested at the Tashkent University of Information Technologies to work in the corporate mode of various departments (faculties, departments, TUIT branches)

    Testing automated driving systems to calibrate drivers’ trust

    Get PDF
    Automated Driving Systems (ADSs) offer many potential benefits like improved safety, reduced traffic congestion and lower emissions. However, such benefits can only be realised if drivers trust and make use of such systems. The two challenges explored in this thesis are: 1) How to increase trust in ADSs? 2) How to identify the test scenarios to establish the true capabilities and limitations of ADSs? Firstly, drivers’ trust needs to be calibrated to the “appropriate” level to prevent misuse (due to over trust) or disuse (due to under trust) of the system. In this research, a method to calibrate drivers’ trust to the appropriate level has been created. This method involves providing knowledge of the capabilities and limitations of the ADSs to the driver. However, there is a need to establish the capabilities and limitations of the ADSs which form the knowledge to be imparted to the driver. Therefore, the next research contribution lies in the development of a novel method to establish the knowledge of capabilities and limitations of ADSs (used to calibrate trust) in a reliable manner. This knowledge can be created by testing ADSs. However, in literature, an unanswered research question remains: How to identify test scenarios which highlight the limitations of ADSs? In order to identify such test scenarios, a novel hazard based testing approach to establish the capabilities and limitations of ADSs is presented by extending STPA (a hazard identification method) to create test scenarios. To ensure reliability of the hazard classification (and of the knowledge), the author created a novel objective approach for risk classification by creating a rule-set for risk ratings. The contribution of this research lies in developing a method to increase trust in ADSs by creating reliable knowledge using hazard based testing approach which identifies how an ADS can fail

    Simulating Operational Concepts for Autonomous Robotic Space Exploration Systems: A Framework for Early Design Validation

    Get PDF
    During mission design, the concept of operations (ConOps) describes how the system operates during various life cycle phases to meet stakeholder expectations. ConOps is sometimes declined in a simple evaluation of the power consumption or data generation per mode. Different operational timelines are typically developed based on expert knowledge. This approach is robust when designing an automated system or a system with a low level of autonomy. However, when studying highly autonomous systems, designers may be interested in understanding how the system would react in an operational scenario when provided with knowledge about its actions and operational environment. These considerations can help verify and validate the proposed ConOps architecture, highlight shortcomings in both physical and functional design, and help better formulate detailed requirements. Hence, this study aims to provide a framework for the simulation and validation of operational scenarios for autonomous robotic space exploration systems during the preliminary design phases. This study extends current efforts in autonomy technology for planetary systems by focusing on testing their operability and assessing their performances in different scenarios early in the design process. The framework uses Model-Based Systems Engineering (MBSE) as the knowledge base for the studied system and its operations. It then leverages a Markov Decision Process (MDP) to simulate a set of system operations in a relevant scenario. It then outputs a feasible plan with the associated variation of a set of considered resources as step functions. This method was applied to simulate the operations of a small rover exploring an unknown environment to observe and sample a set of targets
    corecore