3,562 research outputs found
Report from GI-Dagstuhl Seminar 16394: Software Performance Engineering in the DevOps World
This report documents the program and the outcomes of GI-Dagstuhl Seminar
16394 "Software Performance Engineering in the DevOps World".
The seminar addressed the problem of performance-aware DevOps. Both, DevOps
and performance engineering have been growing trends over the past one to two
years, in no small part due to the rise in importance of identifying
performance anomalies in the operations (Ops) of cloud and big data systems and
feeding these back to the development (Dev). However, so far, the research
community has treated software engineering, performance engineering, and cloud
computing mostly as individual research areas. We aimed to identify
cross-community collaboration, and to set the path for long-lasting
collaborations towards performance-aware DevOps.
The main goal of the seminar was to bring together young researchers (PhD
students in a later stage of their PhD, as well as PostDocs or Junior
Professors) in the areas of (i) software engineering, (ii) performance
engineering, and (iii) cloud computing and big data to present their current
research projects, to exchange experience and expertise, to discuss research
challenges, and to develop ideas for future collaborations
Runtime Verification Based on Executable Models: On-the-Fly Matching of Timed Traces
Runtime verification is checking whether a system execution satisfies or
violates a given correctness property. A procedure that automatically, and
typically on the fly, verifies conformance of the system's behavior to the
specified property is called a monitor. Nowadays, a variety of formalisms are
used to express properties on observed behavior of computer systems, and a lot
of methods have been proposed to construct monitors. However, it is a frequent
situation when advanced formalisms and methods are not needed, because an
executable model of the system is available. The original purpose and structure
of the model are out of importance; rather what is required is that the system
and its model have similar sets of interfaces. In this case, monitoring is
carried out as follows. Two "black boxes", the system and its reference model,
are executed in parallel and stimulated with the same input sequences; the
monitor dynamically captures their output traces and tries to match them. The
main problem is that a model is usually more abstract than the real system,
both in terms of functionality and timing. Therefore, trace-to-trace matching
is not straightforward and allows the system to produce events in different
order or even miss some of them. The paper studies on-the-fly conformance
relations for timed systems (i.e., systems whose inputs and outputs are
distributed along the time axis). It also suggests a practice-oriented
methodology for creating and configuring monitors for timed systems based on
executable models. The methodology has been successfully applied to a number of
industrial projects of simulation-based hardware verification.Comment: In Proceedings MBT 2013, arXiv:1303.037
COST Action IC 1402 ArVI: Runtime Verification Beyond Monitoring -- Activity Report of Working Group 1
This report presents the activities of the first working group of the COST
Action ArVI, Runtime Verification beyond Monitoring. The report aims to provide
an overview of some of the major core aspects involved in Runtime Verification.
Runtime Verification is the field of research dedicated to the analysis of
system executions. It is often seen as a discipline that studies how a system
run satisfies or violates correctness properties. The report exposes a taxonomy
of Runtime Verification (RV) presenting the terminology involved with the main
concepts of the field. The report also develops the concept of instrumentation,
the various ways to instrument systems, and the fundamental role of
instrumentation in designing an RV framework. We also discuss how RV interplays
with other verification techniques such as model-checking, deductive
verification, model learning, testing, and runtime assertion checking. Finally,
we propose challenges in monitoring quantitative and statistical data beyond
detecting property violation
Checking-in on Network Functions
When programming network functions, changes within a packet tend to have
consequences---side effects which must be accounted for by network programmers
or administrators via arbitrary logic and an innate understanding of
dependencies. Examples of this include updating checksums when a packet's
contents has been modified or adjusting a payload length field of a IPv6 header
if another header is added or updated within a packet. While static-typing
captures interface specifications and how packet contents should behave, it
does not enforce precise invariants around runtime dependencies like the
examples above. Instead, during the design phase of network functions,
programmers should be given an easier way to specify checks up front, all
without having to account for and keep track of these consequences at each and
every step during the development cycle. In keeping with this view, we present
a unique approach for adding and generating both static checks and dynamic
contracts for specifying and checking packet processing operations. We develop
our technique within an existing framework called NetBricks and demonstrate how
our approach simplifies and checks common dependent packet and header
processing logic that other systems take for granted, all without adding much
overhead during development.Comment: ANRW 2019 ~ https://irtf.org/anrw/2019/program.htm
IST Austria Technical Report
We argue that the time is ripe to investigate differential monitoring, in which the specification of a program's behavior is implicitly given by a second program implementing the same informal specification. Similar ideas have been proposed before, and are currently implemented in restricted form for testing and specialized run-time analyses, aspects of which we combine. We discuss the challenges of implementing differential monitoring as a general-purpose, black-box run-time monitoring framework, and present promising results of a preliminary implementation, showing low monitoring overheads for diverse programs
LNCS
We argue that the time is ripe to investigate differential monitoring, in which the specification of a program's behavior is implicitly given by a second program implementing the same informal specification. Similar ideas have been proposed before, and are currently implemented in restricted form for testing and specialized run-time analyses, aspects of which we combine. We discuss the challenges of implementing differential monitoring as a general-purpose, black-box run-time monitoring framework, and present promising results of a preliminary implementation, showing low monitoring overheads for diverse programs
Recommended from our members
Automated Testing and Debugging for Big Data Analytics
The prevalence of big data analytics in almost every large-scale software system has generated a substantial push to build data-intensive scalable computing (DISC) frameworks such as Google MapReduce and Apache Spark that can fully harness the power of existing data centers. However, frameworks once used by domain experts are now being leveraged by data scientists, business analysts, and researchers. This shift in user demographics calls for immediate advancements in the development, debugging, and testing practices of big data applications, which are falling behind compared to the DISC framework design and implementation. In practice, big data applications often fail as users are unable to test all behaviors emerging from interleaving dataflow operators, user-defined functions, and framework's code. "Testing based on a random sample" rarely guarantees the reliability and "trial and error" and "print" debugging methods are expensive and time-consuming. Thus, the current practice of developing a big data application must be improved and the tools built to enhance the developer's productivity must adapt to the distinct characteristics of data-intensive scalable computing. By synthesizing ideas from software engineering and database systems, our hypothesis is that we can design effective and scalable testing and debugging algorithms for big data analytics without compromising the performance and efficiency of the underlying DISC framework. To design such techniques, we investigate how we can build interactive and responsive debugging primitives that significantly reduce the debugging time, yet do not pose much performance overhead on big data applications. Furthermore, we investigate how we can leverage data provenance techniques from databases and fault-isolation algorithms from software engineering to pinpoint the minimal subset of failure-inducing inputs efficiently. To improve the reliability of big data analytics, we investigate how we can abstract the semantics of dataflow operators and use them in tandem with the semantics of user-defined functions to generate a minimum set of synthetic test inputs capable of revealing more defects than the entire input dataset.To examine the first hypothesis, we introduce interactive, real-time debugging primitives for big data analytics through innovative and scalable debugging features such as simulated breakpoint, dynamic watchpoint, and crash culprit identification. Second, we design a new automated fault localization approach that combines insights from both the software engineering and database literature to bring delta debugging closer to a reality in the big data applications by leveraging data provenance and by constructing systems optimizations for debugging provenance queries. Lastly, we devise a new symbolic-execution based white-box testing algorithm for big data applications that abstracts the implementation of dataflow operators using logical specifications instead of modeling their implementations and combines them with the semantics of any arbitrary user-defined function. We instantiate the idea of an interactive debugging algorithm as BigDebug, the idea of an automated debugging algorithm as BigSift, and the idea of symbolic execution-based testing as BigTest. Our investigation shows that the interactive debugging primitives can scale to terabytes---our record-level tracing incurs less than 25% overhead on average and provides up to 100% time saving compared to the baseline replay debugger. Second, we observe that by combining data provenance with delta debugging, we can identify the minimum faulty input in just under 30% of the original job execution time. Lastly, we verify that by abstracting dataflow operators using logical specifications, we can efficiently generate the most concise test data suitable for local testing while revealing twice as many faults as prior approaches. Our investigations collectively demonstrate that developer productivity can be significantly improved through effective and scalable testing and debugging techniques for big data analytics, without impacting the DISC framework's performance. This dissertation affirms the feasibility of automated debugging and testing techniques for big data analytics---techniques that were previously considered infeasible for large-scale data processing
- …