Collected notes from the Benchmarks and Metrics Workshop

In recent years there has been a proliferation of proposals in the artificial intelligence (AI) literature for integrated agent architectures. Each architecture offers an approach to the general problem of constructing an integrated agent. Unfortunately, the ways in which one architecture might be considered better than another are not always clear. There has been a growing realization that many of the positive and negative aspects of an architecture become apparent only when experimental evaluation is performed and that to progress as a discipline, we must develop rigorous experimental methods. In addition to the intrinsic intellectual interest of experimentation, rigorous performance evaluation of systems is also a crucial practical concern to our research sponsors. DARPA, NASA, and AFOSR (among others) are actively searching for better ways of experimentally evaluating alternative approaches to building intelligent agents. One tool for experimental evaluation involves testing systems on benchmark tasks in order to assess their relative performance. As part of a joint DARPA and NASA funded project, NASA-Ames and Teleos Research are carrying out a research effort to establish a set of benchmark tasks and evaluation metrics by which the performance of agent architectures may be determined. As part of this project, we held a workshop on Benchmarks and Metrics at the NASA Ames Research Center on June 25, 1990. The objective of the workshop was to foster early discussion on this important topic. We did not achieve a consensus, nor did we expect to. Collected here is some of the information that was exchanged at the workshop. Given here is an outline of the workshop, a list of the participants, notes taken on the white-board during open discussions, position papers/notes from some participants, and copies of slides used in the presentations

SCCharts: The Mindstorms Report

SCCharts are a visual language proposed in 2012 for specifying safety-critical reactive systems. This is the second SCCharts report towards the usability of the SCCharts visual language and its KIELER SCCharts implementation. KIELER is an open-source project which researches the pragmatics of model-based languages and related fields. Nine case-studies that were conducted between 2015 and 2019 evaluate the pros and cons in the context of small-scale Lego Mindstorms models and similar projects. Par-ticipants of the studies included undergraduate and graduate students from our local and also external facilities, as well as academics from the synchronous community. In the surveys, both the SCCharts language and the SCCharts tools are compared to other modeling and classical programming languages and tools

Development of an Integrated Process, Modeling and Simulation Platform for Performance-Based Design of Low-Energy and High IEQ Buildings

The objective of this study was to develop a Virtual Design Studio (VDS) : a software platform for integrated, coordinated and optimized design of green building systems with low energy consumption, high indoor environmental quality (IEQ), and high level of sustainability. The VDS is intended to assist collaborating architects, engineers and project management team members throughout from the early phases to the detailed building design stages. It can be used to plan design tasks and workflow, and evaluate the potential impacts of various green building strategies on the building performance by using the state of the art simulation tools as well as industrial/professional standards and guidelines for green building system design. Based on the review and analysis of existing professional practices in building system design, particularly those used in U.S., Germany and UK, a generic process for performance-based building design, construction and operation was proposed. It included Assess, Define, Design, Apply, and Monitoring (ADDAM) stages. The current VDS focused on the first three stages. The VDS considers the building design as a multi-dimensional process involving multiple design teams, design factors, and design stages. The intersection among these three dimensions defines a specific design task in terms of who , what and when . It also considers building design as a multi-objective process that aims to enhance the five aspects of performance for green building systems: site sustainability, materials and resource efficiency, water utilization efficiency, energy efficiency and impacts to the atmospheric environment, and IEQ. The current VDS development has been limited to the energy efficiency and IEQ performance with particular focus on thermal, air quality and lighting environmental quality because of their strong interaction with the energy performance of buildings. The VDS software framework contains four major functions: 1) Design coordination: It enables users to define tasks using the Input-Process-Output flow approach, which specifies the anticipated activities (i.e., the process), required input and output information, and anticipated interactions with other tasks. It also allows task scheduling to define the work flow, and sharing of the design data and information via internet. 2) Modeling and simulation: It enables users to perform building simulations to predict the energy consumption and IEQ conditions at any of the design stages by using EnergyPlus and a combined heat, air, moisture and pollutant simulation (CHAMPS) model. A method for co-simulation was developed to allow the use of both models at the same time step for the combined energy and indoor air quality analysis. 3) Results visualization: It enables users to display a 3-D geometric design of the building by reading BIM (building information model) file generated by design software such as SketchUp, and the predicted results of heat, air, moisture, pollutant and light distributions in the building. 4) Performance evaluation: It enables the users to compare the performance of a proposed building design against a reference building that is defined for the same type of buildings under the same climate condition, and predict the percent of improvements over the minimum requirements specified in ASHRAE Standard 55-2010, 62.1-2010 and 90.1-2010. An approach was developed to estimate the potential impact of a design factor on the whole building performance, and hence can assist the user to identify areas that have most pay back for investment. The VDS software was developed by using C++ with the conventional Model, View and Control (MVC) software architecture. The software has been verified by using a simple 3-zone case building. The application of the VDS concepts and framework for building design and performance analysis has been illustrated by using a medium size five story office building that received the LEED Platinum Certification from USGBC

Development Process for Multi-Disciplinary Embedded Control Systems

This report contains the progress report for the qualification exam for Industrial PhD student Sune Wolff. Initial work on describing a development process for multi-disciplinary systems using collaborative modelling and co-simulation is described

Evaluation of Development Process and Methodology for Co-Models

An embedded control system often requires a tight association between computational and physical system components. In such cases, embedded system development is difficult, as it requires the collaboration among stakeholders with different backgrounds (software engineers, mechanical engineers, managers etc.). With the constant increase in design complexity, caused by advances in implementation technologies, new ways of approaching embedded system development are needed.This thesis presents an evaluation of a tool-oriented development process and methodology, supporting embedded system development. The philosophy of the development process and methodology, is that design complexity can be managed through collaborative work and multi-disciplinary modeling. To obtain input for the evaluation work, the development process is applied during a case study, involving the development of a route following robot and a model of this. To demonstrate the value of this model, it is simulated to predict route completion times for the physical robot.The evaluation work identifies possibilities and challenges of the development process and methodology, with respect to traditional physicalprototyping. This will support developers in choosing the most optimal way of approaching development. In addition to this, suggestions for extensions to the methodology are provided. These intend to increasethe value the development process and methodology may bring thedevelopment work

A Proposed Approach to Mechatronics Design and Implementation Education-Oriented Methodology

Mechatronics engineer is expected to design engineering systems with synergy and integration toward constrains like higher performance, speed, precision, efficiency, lower costs and functionality. The key element in success of a mechatronics engineering education-program, and correspondingly, Mechatronics engineering graduates, is directly related to a well-structured mechatronic system design course and the applied structural design methodology. Guidelines for structural design methodology and tools for the development process of mechatronic products, that can be applied in educational process is highly required. This paper proposes mechatronics systems design education-oriented methodology, which aims to integrate multidisciplinary knowledge, in various stages through the design process and development of mechatronics product. The proposed mechatronics design methodology is described, discussed and applied with the help of example student final year graduation project; design and implementation of mechatronics mobile robotic guidance system in the from of smart wheelchair- Mechatronics Motawif, to help and support people with disabilities and special needs to perform specific predetermined tasks, particularly, performing Al Omrah and motion around holy Kaba, Makka. Keywords: Mechatronics, Design methodology, Parallel design, Synergistic integration, Modeling/ Simulation, Prototyping, Mobile robot, Motawif

A Holistic Approach to Functional Safety for Networked Cyber-Physical Systems

Functional safety is a significant concern in today's networked cyber-physical systems such as connected machines, autonomous vehicles, and intelligent environments. Simulation is a well-known methodology for the assessment of functional safety. Simulation models of networked cyber-physical systems are very heterogeneous relying on digital hardware, analog hardware, and network domains. Current functional safety assessment is mainly focused on digital hardware failures while minor attention is devoted to analog hardware and not at all to the interconnecting network. In this work we believe that in networked cyber-physical systems, the dependability must be verified not only for the nodes in isolation but also by taking into account their interaction through the communication channel. For this reason, this work proposes a holistic methodology for simulation-based safety assessment in which safety mechanisms are tested in a simulation environment reproducing the high-level behavior of digital hardware, analog hardware, and network communication. The methodology relies on three main automatic processes: 1) abstraction of analog models to transform them into system-level descriptions, 2) synthesis of network infrastructures to combine multiple cyber-physical systems, and 3) multi-domain fault injection in digital, analog, and network. Ultimately, the flow produces a homogeneous optimized description written in C++ for fast and reliable simulation which can have many applications. The focus of this thesis is performing extensive fault simulation and evaluating different functional safety metrics, \eg, fault and diagnostic coverage of all the safety mechanisms