1,300 research outputs found
Automatic Performance Testing using Input-Sensitive Profiling
During performance testing, software engineers commonly perform application profiling to analyze an application\u27s traces with different inputs to understand performance behaviors, such as time and space consumption. However, a non-trivial application commonly has a large number of inputs, and it is mostly manual to identify the specific inputs leading to performance bottlenecks. Thus, it is challenge is to automate profiling and find these specific inputs. To solve these problems, we propose novel approaches, FOREPOST, GA-Prof and PerfImpact, which automatically profile applications for finding the specific combinations of inputs triggering performance bottlenecks, and further analyze the corresponding traces to identify problematic methods. Specially, our approaches work in two different types of real-world scenarios of performance testing: i) a single-version scenario, in which performance bottlenecks are detected in a single software release, and ii) a two-version scenario, in which code changes responsible for performance regressions are detected by considering two consecutive software releases
On Improving (Non)Functional Testing
Software testing is commonly classified into two categories, nonfunctional testing and functional testing. The goal of nonfunctional testing is to test nonfunctional requirements, such as performance and reliability. Performance testing is one of the most important types of nonfunctional testing, one goal of which is to detect the phenomena that an Application Under Testing (AUT) exhibits unexpectedly worse performance (e.g., lower throughput) with some input data. During performance testing, a critical challenge is to understand the AUT’s behaviors with large numbers of combinations of input data and find the particular subset of inputs leading to performance bottlenecks. However, enumerating those particular inputs and identifying those bottlenecks are always laborious and intellectually intensive. In addition, for an evolving software system, some code changes may accidentally degrade performance between two software versions, it is even more challenging to find problematic changes (out of a large number of committed changes) may lead to performance regressions under certain test inputs. This dissertation presents a set of approaches to automatically find specific combinations of input data for exposing performance bottlenecks and further analyze execution traces to identify performance bottlenecks. In addition, this dissertation also provides an approach that automatically estimates the impact of code changes on performance degradation between two released software versions to identify the problematic ones likely leading to performance regressions. Functional testing is used to test the functional correctness of AUTs. Developers commonly write test suites for AUTs to test different functionalities and locate functional faults. During functional testing, developers rely on some strategies to order test cases to achieve certain objectives, such as exposing faults faster, which is known as Test Case Prioritization (TCP). TCP techniques are commonly classified into two categories, dynamic and static techniques. A set of empirical studies has been conducted to examine and understand different TCP techniques, but there is a clear gap in existing studies. No study has compared static techniques against dynamic techniques and comprehensively examined the impact of test granularity, program size, fault characteristics, and the similarities in terms of fault detection on TCP techniques. Thus, this dissertation presents an empirical study to thoroughly compare static and dynamic TCP techniques in terms of effectiveness, efficiency, and similarity of uncovered faults at different granularities on a large set of real-world programs, and further analyze the potential impact of program size and fault characteristics on TCP evaluation. Moreover, in the prior work, TCP techniques have been typically evaluated against synthetic software defects, called mutants. For this reason, it is currently unclear whether TCP performance on mutants would be representative of the performance achieved on real faults. to answer this fundamental question, this dissertation presents the first empirical study that investigates TCP performance when applied to both real-world faults and mutation faults for understanding the representativeness of mutants
Single-Path Mobile AutoML: Efficient ConvNet Design and NAS Hyperparameter Optimization
Can we reduce the search cost of Neural Architecture Search (NAS) from days
down to only few hours? NAS methods automate the design of Convolutional
Networks (ConvNets) under hardware constraints and they have emerged as key
components of AutoML frameworks. However, the NAS problem remains challenging
due to the combinatorially large design space and the significant search time
(at least 200 GPU-hours). In this work, we alleviate the NAS search cost down
to less than 3 hours, while achieving state-of-the-art image classification
results under mobile latency constraints. We propose a novel differentiable NAS
formulation, namely Single-Path NAS, that uses one single-path
over-parameterized ConvNet to encode all architectural decisions based on
shared convolutional kernel parameters, hence drastically decreasing the search
overhead. Single-Path NAS achieves state-of-the-art top-1 ImageNet accuracy
(75.62%), hence outperforming existing mobile NAS methods in similar latency
settings (~80ms). In particular, we enhance the accuracy-runtime trade-off in
differentiable NAS by treating the Squeeze-and-Excitation path as a fully
searchable operation with our novel single-path encoding. Our method has an
overall cost of only 8 epochs (24 TPU-hours), which is up to 5,000x faster
compared to prior work. Moreover, we study how different NAS formulation
choices affect the performance of the designed ConvNets. Furthermore, we
exploit the efficiency of our method to answer an interesting question: instead
of empirically tuning the hyperparameters of the NAS solver (as in prior work),
can we automatically find the hyperparameter values that yield the desired
accuracy-runtime trade-off? We open-source our entire codebase at:
https://github.com/dstamoulis/single-path-nas.Comment: Detailed extension (journal) of the Single-Path NAS ECMLPKDD'19 paper
(arXiv:1904.02877
ScalAna: Automating Scaling Loss Detection with Graph Analysis
Scaling a parallel program to modern supercomputers is challenging due to
inter-process communication, Amdahl's law, and resource contention. Performance
analysis tools for finding such scaling bottlenecks either base on profiling or
tracing. Profiling incurs low overheads but does not capture detailed
dependencies needed for root-cause analysis. Tracing collects all information
at prohibitive overheads. In this work, we design ScalAna that uses static
analysis techniques to achieve the best of both worlds - it enables the
analyzability of traces at a cost similar to profiling. ScalAna first leverages
static compiler techniques to build a Program Structure Graph, which records
the main computation and communication patterns as well as the program's
control structures. At runtime, we adopt lightweight techniques to collect
performance data according to the graph structure and generate a Program
Performance Graph. With this graph, we propose a novel approach, called
backtracking root cause detection, which can automatically and efficiently
detect the root cause of scaling loss. We evaluate ScalAna with real
applications. Results show that our approach can effectively locate the root
cause of scaling loss for real applications and incurs 1.73% overhead on
average for up to 2,048 processes. We achieve up to 11.11% performance
improvement by fixing the root causes detected by ScalAna on 2,048 processes.Comment: conferenc
- …