MLPerf Inference Benchmark

Machine-learning (ML) hardware and software system demand is burgeoning. Driven by ML applications, the number of different ML inference systems has exploded. Over 100 organizations are building ML inference chips, and the systems that incorporate existing models span at least three orders of magnitude in power consumption and five orders of magnitude in performance; they range from embedded devices to data-center solutions. Fueling the hardware are a dozen or more software frameworks and libraries. The myriad combinations of ML hardware and ML software make assessing ML-system performance in an architecture-neutral, representative, and reproducible manner challenging. There is a clear need for industry-wide standard ML benchmarking and evaluation criteria. MLPerf Inference answers that call. In this paper, we present our benchmarking method for evaluating ML inference systems. Driven by more than 30 organizations as well as more than 200 ML engineers and practitioners, MLPerf prescribes a set of rules and best practices to ensure comparability across systems with wildly differing architectures. The first call for submissions garnered more than 600 reproducible inference-performance measurements from 14 organizations, representing over 30 systems that showcase a wide range of capabilities. The submissions attest to the benchmark's flexibility and adaptability.Comment: ISCA 202

10421 Abstracts Collection -- Model-Based Testing in Practice

From 17.10. to 22.10.2010, the Dagstuhl Seminar 10421 ``Model-Based Testing in Practice \u27\u27 was held in Schloss Dagstuhl~--~Leibniz Center for Informatics. During the seminar, several participants presented their current research, and ongoing work and open problems were discussed. Abstracts of the presentations given during the seminar as well as abstracts of seminar results and ideas are put together in this paper. The first section describes the seminar topics and goals in general. Links to extended abstracts or full papers are provided, if available

A Structural Analysis of On-Line Fault Detection Mechanisms in Network-On-Chip Architectures

Network-on-Chip (NoC) communication architectures are widely used as on-chip interconnect in multi-core systems. These systems are increasingly used in safety-critical applications, so it is essential to quickly detect faults within the NoC during system operation. The current approach to detect a fault in an NoC system is to apply periodic test using built-in self-test (BIST) circuitry during system idle periods. This approach has the advantage that it is a structural test, so can quickly achieve high fault coverage. A second advantage is that the BIST infrastructure can be used during manufacturing test. The disadvantage is the need for the idle time to apply the test, and the time to save/restore the functional state that is overwritten during the test. An additional disadvantage is that the system is at risk of an undetected fault between self-tests. In this research we propose to test the NoC system while it is in functional operation, which is an on-line test. We will use functional invariants to detect errors in functional operation, which can then trigger diagnosis and fault recovery or system reconfiguration. The advantage of this approach is that it minimizes fault detection latency, and avoids the need for a system idle period or for save/restore state operations. The disadvantage is that it is much more difficult to achieve high fault coverage since our approach detects functional errors based on existing functional network traffic, rather than self-test stimulus. In order to evaluate our functional test approach, we have designed a gate-level NoC implementation, which can be the target for gate-level fault injection and simulation using realistic network traffic. We inject stuck-at faults and single event transients into the gate-level logic during simulation of synthetic NoC traffic. We found that the functional invariants proposed in prior work miss detection of many faults. Most of these escapes are detected by end-to-end cyclical redundancy checks. However, we found it necessary to create additional functional checkers to detect the remaining faults

Detecting non-secure memory deallocation with CBMC

2021 Fall.Includes bibliographical references.Scrubbing sensitive data before releasing memory is a widely recommended but often ignored programming practice for developing secure software. Consequently, sensitive data such as cryptographic keys, passwords, and personal data, can remain in memory indefinitely, thereby increasing the risk of exposure to hackers who can retrieve the data using memory dumps or exploit vulnerabilities such as Heartbleed and Etherleak. We propose an approach for detecting a specific memory safety bug called Improper Clearing of Heap Memory Before Release, referred to as Common Weakness Enumeration 244. The CWE-244 bug in a program allows the leakage of confidential information when a variable is not wiped before heap memory is freed. Our approach uses the CBMC model checker to detect this weakness and is based on instrumenting the program using (1) global variable declarations that track and monitor the state of the program variables relevant for CWE-244, and (2) assertions that help CBMC to detect unscrubbed memory. We develop a tool, SecMD-Checker, implementing our instrumentation based algorithm, and we provide experimental validation on the Juliet Test Suite that the tool is able to detect all the CWE-244 instances present in the test suite. The proposed approach has the potential to work with other model checkers and can be extended for detecting other weaknesses that require variable tracking and monitoring, such as CWE-226, CWE-319, and CWE-1239

자동차 사양 변경을 실시간 반영하는 데이터 기반 디자인 접근 방법

학위논문 (박사) -- 서울대학교 대학원 : 융합과학기술대학원 융합과학부(지능형융합시스템전공), 2020. 8. 곽노준.The automotive industry is entering a new phase in response to changes in the external environment through the expansion of eco-friendly electric/hydrogen vehicles and the simplification of modules during the manufacturing process. However, in the existing automotive industry, conflicts between structured production guidelines and various stake-holders, who are aligned with periodic production plans, can be problematic. For example, if there is a sudden need to change either production parts or situation-specific designs, it is often difficult for designers to reflect those requirements within the preexisting guidelines. Automotive design includes comprehensive processes that represent the philosophy and ideology of a vehicle, and seeks to derive maximum value from the vehicle specifications. In this study, a system that displays information on parts/module components necessary for real-time design was proposed. Designers will be able to use this system in automotive design processes, based on data from various sources. By applying the system, three channels of information provision were established. These channels will aid in the replacement of specific component parts if an unexpected external problem occurs during the design process, and will help in understanding and using the components in advance. The first approach is to visualize real-time data aggregation in automobile factories using Google Analytics, and to reflect these in self-growing characters to be provided to designers. Through this, it is possible to check production and quality status data in real time without the use of complicated labor resources such as command centers. The second approach is to configure the data flow to be able to recognize and analyze the surrounding situation. This is done by applying the vehicles camera to the CCTV in the inventory and distribution center, as well as the direction inside the vehicle. Therefore, it is possible to identify and record the parts resources and real-time delivery status from the internal camera function without hesitation from existing stakeholders. The final approach is to supply real-time databases of vehicle parts at the site of an accident for on-site repair, using a public API and sensor-based IoT. This allows the designer to obtain information on the behavior of parts to be replaced after accidents involving light contact, so that it can be reflected in the design of the vehicle. The advantage of using these three information channels is that designers can accurately understand and reflect the modules and components that are brought in during the automotive design process. In order to easily compose the interface for the purpose of providing information, the information coming from the three channels is displayed in their respective, case-specific color in the CAD software that designers use in the automobile development process. Its eye tracking usability evaluation makes it easy for business designers to use as well. The improved evaluation process including usability test is also included in this study. The impact of the research is both dashboard application and CAD system as well as data systems from case studies are currently reflected to the design ecosystem of the motors group.자동차 산업은 친환경 전기/수소 자동차의 확대와 제조 공정에서의 모듈 단순화를 통해서 외부 환경의 변화에 따른 새로운 국면을 맞이하고 있다. 하지만 기존의 자동차 산업에서 구조화된 생산 가이드라인과 기간 단위 생산 계획에 맞춰진 여러 이해관계자들과의 갈등은 변화에 대응하는 방안이 관성과 부딪히는 문제로 나타날 수 있다. 예를 들어, 갑작스럽게 생산에 필요한 부품을 변경해야 하거나 특정 상황에 적용되는 디자인을 변경할 경우, 주어진 가이드라인에 따라 디자이너가 직접 의견을 반영하기 어려운 경우가 많다. 자동차 디자인은 차종의 철학과 이념을 나타내고 해당 차량제원으로 최대의 가치를 끌어내고자 하는 종합적인 과정이다. 본 연구에서는 여러 원천의 데이터를 기반으로 자동차 디자인 과정에서 활용할 수 있도록 디자인에 필요한 부품/모듈 구성요소들에 대한 정보를 실시간으로 표시해주는 시스템을 고안하였다. 이를 적용하여 자동차 디자인 과정에서 예상 못한 외부 문제가 발생했을 때 선택할 구성 부품을 대체하거나 사전에 해당 부품을 이해하고 디자인에 활용할 수 있도록 세 가지 정보 제공 채널을 구성하였다. 첫 번째는 자동차 공장 내 실시간 데이터 집계를 Google Analytics를 활용하여 시각화하고, 이를 공장 자체의 자가 성장 캐릭터에 반영하여 디자이너에게 제공하는 방식이다. 이를 통해 종합상황실 등의 복잡한 인력 체계 없이도 생산 및 품질 현황 데이터를 실시간으로 확인 가능하도록 하였다. 두 번째는 차량용 주차보조 센서 카메라를 차량 부착 뿐만 아니라 인벤토리와 물류센터의 CCTV에도 적용하여 주변상황을 인식하고 분석할 수 있도록 구성하였다. 차량의 조립 생산 단계에서 부품 단위의 이동, 운송, 출하를 거쳐 완성차의 주행 단계에 이르기까지 데이터 흐름을 파악하는 것이 디자인 부문에 필요한 정보를 제공할 수 있는 방법으로 활용되었다. 이를 통해 기존 이해관계자들의 큰 반발 없이 내부의 카메라 기능으로부터 부품 리소스와 운송 상태를 실시간 파악 및 기록 가능하도록 하였다. 마지막으로 공공 API와 센서 기반의 사물인터넷을 활용해서 도로 위 차량 사고가 발생한 위치에서의 현장 수리를 위한 차량 부품 즉시 수급 및 데이터베이스화 방법도 개발 되었다. 이는 디자이너로 하여금 가벼운 접촉 사고에서의 부품 교체 행태에 대한 정보를 얻게 하여 차량의 디자인에 반영 가능하도록 하였다. 시나리오를 바탕으로 이 세 가지 정보 제공 채널을 활용할 경우, 자동차 디자인 과정에서 불러들여오는 부품 및 모듈의 구성 요소들을 디자이너가 정확히 알고 반영할 수 있다는 장점이 부각되었다. 정보 제공의 인터페이스를 쉽게 구성하기 위해서, 실제로 디자이너들이 자동차 개발 과정에서 디자인 프로세스 상에서 활용하는 CAD software에 세 가지 채널들로부터 들어오는 정보를 사례별 컬러로 표시하고, 이를 시선추적 사용성 평가를 통해 현업 디자이너들이 사용하기 쉽게 개선한 과정도 본 연구에 포함시켜 설명하였다.1 Introduction 1 1.1 Research Background 1 1.2 Objective and Scope 2 1.3 Environmental Changes 3 1.4 Research Method 3 1.4.1 Causal Inference with Graphical Model 3 1.4.2 Design Thinking Methodology with Co-Evolution 4 1.4.3 Required Resources 4 1.5 Research Flow 4 2 Data-driven Design 7 2.1 Big Data and Data Management 6 2.1.1 Artificial Intelligence and Data Economy 6 2.1.2 API (Application Programming Interface) 7 2.1.3 AI driven Data Management for Designer 7 2.2 Datatype from Automotive Industry 8 2.2.1 Data-driven Management in Automotive Industry 8 2.2.2 Automotive Parts Case Studies 8 2.2.3 Parameter for Generative Design 9 2.3 Examples of Data-driven Design 9 2.3.1 Responsive-reactive 9 2.3.2 Dynamic Document Design 9 2.3.3 Insignts from Data-driven Design 10 3 Benchmark of Data-driven Automotive Design 12 3.1 Method of Global Benchmarking 11 3.2 Automotive Design 11 3.2.1 HMI Design and UI/UX 11 3.2.2 Hardware Design 12 3.2.3 Software Design 12 3.2.4 Convergence Design Process Model 13 3.3 Component Design Management 14 4 Vehicle Specification Design in Mobility Industry 16 4.1 Definition of Vehicle Specification 16 4.2 Field Study 17 4.3 Hypothesis 18 5 Three Preliminary Practical Case Studies for Vehicle Specification to Datadriven 21 5.1 Production Level 31 5.1.1 Background and Input 31 5.1.2 Data Process from Inventory to Designer 41 5.1.3 Output to Designer 51 5.2 Delivery Level 61 5.2.1 Background and Input 61 5.2.2 Data Process from Inventory to Designer 71 5.2.3 Output to Designer 81 5.3 Consumer Level 91 5.3.1 Background and Input 91 5.3.2 Data Process from Inventory to Designer 101 5.3.3 Output to Designer 111 6 Two Applications for Vehicle Designer 86 6.1 Real-time Dashboard DB for Decision Making 123 6.1.1 Searchable Infographic as a Designer's Tool 123 6.1.2 Scope and Method 123 6.1.3 Implementation 123 6.1.4 Result 124 6.1.5 Evaluation 124 6.1.6 Summary 124 6.2 Application to CAD for vehicle designer 124 6.2.1 CAD as a Designer's Tool 124 6.2.2 Scope and Method 125 6.2.3 Implementation and the Display of the CAD Software 125 6.2.4 Result 125 6.2.5 Evaluation: Usability Test with Eyetracking 126 6.2.6 Summary 128 7 Conclusion 96 7.1 Summary of Case Studies and Application Release 129 7.2 Impact of the Research 130 7.3 Further Study 131Docto

Tuotemallien tarkistuksen metriikan kehitys ja automaatio

A lot of interest and research has been focused on product quality and it is recognized as a crucial aspect of engineering. The quality of product models can also be seen as essential in engineering workflow especially in systems based on downstream data. Model quality effects not only the models accuracy and modifiability but also the agility of the whole engineering systems. Careful and thorough verification plays an important part in effecting product model quality. Verifying product models and designs manually can be laborious and time-consuming process. By automating parts of the verification process, benefits can be seen in the time frame and end results of the verification. The goal of the thesis is to develop metrics and automation for product model verification. Development of metrics is executed by researching literature for model quality metrics and construct a set of metrics for the company. Furthermore, the possibilities of product model verification automation are studied and a working automated model verification tool shall be created based on the metrics. The tool is intended be used in the current modeling environment. The outcomes of this thesis are a list of product quality dimensions with their corresponding metrics and a customized PTC ModelCHECK check that can automatically identify issues in product models. Quality dimensions were identified based on company needs and literature research. ModelCHECK platform was chosen for verification tool development as the software is readily available for the company which means it is a cost-effective way of utilizing automated product model verification in current design environment.Tuotteiden laatuun on jo pidemmän aikaa kiinnitetty paljon huomiota insinööriprosesseissa ja tutkimuksessa. Myös tuotemallien laatu voidaan nähdä insinöörityön kannalta elintärkeässä asemassa, erityisesti systeemeissä jotka perustuvat alaspäin virtaavaan tietoon. Mallien laatu vaikuttaa muun muassa sen tarkkuuteen ja muokattavuuteen sekä koko mallinnus- ja suunnittelujärjestelmän ketteryyteen. Huolellinen ja läpikotainen tarkistus on tärkeä osa tuotemallien laadun kehittämistä. Mallien manuaalinen tarkastaminen voi olla työlästä ja aikaavievää. Käyttämällä automaatiota tarkistuksen apuna, voidaan saavuttaa etuja tarkistuksen nopeudessa ja lopputuloksessa. Tämän diplomityön tavoitteena on kehittää tuotemallien tarkastuksen metriikkaa ja automaatiota. Metriikan kehitys perustuu kirjallisuustutkimukseen sekä muun muassa haastatteluissa kartoitettuihin yrityksen tarpeisiin. Tavoitteena on luoda tuotemalleille metriikkaa, joita vasten niiden ominaisuuksia voidaan arvioida. Myös tarkistuksen automaatiota tutkitaan ja tavoitteena on luoda automaattinen työkalu, jota voidaan käyttää yrityksen tämän hetkisessä suunnittelujärjestelmässä. Tutkimuksen lopputuloksena syntyi lista tuotemallien laadun ulottuvuuksista niihin liitetyillä metriikoilla ja metriikan mukainen PTC ModelCHECK tarkistuspohja 3D-malleille, joka löytyy automaattisesti virheitä malleista. ModelCHECK valittiin työkaluksi, koska se on valmiiksi saatavilla yrityksen nykyisessä mallinnusjärjestelmässä, joilloin automatisointi on erittäin kustannustehokasta

Analyzing Robustness of UML State Machines

UML State Machines constitute an integral part of software behavior specification within the Unified Modeling Language (UML). The development of realistic software applications often results in complex and distributed models. Hence, potential errors can be very subtle and hard to locate for the developer. In this paper, we present a set of robustness rules that seek to avoid common types of errors by ruling out certain modelling constructs. Furthermore, adherence to these rules can improve model readability and maintainability. The robustness rules constitute a general Statechart style guide for different dialects, such as UML State Machines, Statemate, and Esterel Studio. Based on this style guide, an automated checking framework has been implemented as a plug-in for the prototypical Statechart modeling tool KIEL. Simple structural checks can be formulated in a compact, abstract manner in the Object Constraint Language (OCL). The framework can also incorporate checks that go beyond the expressiveness of OCL by implementing them in Java directly, which can also serve as a gateway to formal verification tools; we have exploited this to incorporate a theorem prover for more advanced checks. As a case study, we adopted the UML well-formedness rules; this confirmed that individual rules can easily be incorporated into the framework