5,024 research outputs found
Transfer Learning for Improving Model Predictions in Highly Configurable Software
Modern software systems are built to be used in dynamic environments using
configuration capabilities to adapt to changes and external uncertainties. In a
self-adaptation context, we are often interested in reasoning about the
performance of the systems under different configurations. Usually, we learn a
black-box model based on real measurements to predict the performance of the
system given a specific configuration. However, as modern systems become more
complex, there are many configuration parameters that may interact and we end
up learning an exponentially large configuration space. Naturally, this does
not scale when relying on real measurements in the actual changing environment.
We propose a different solution: Instead of taking the measurements from the
real system, we learn the model using samples from other sources, such as
simulators that approximate performance of the real system at low cost. We
define a cost model that transform the traditional view of model learning into
a multi-objective problem that not only takes into account model accuracy but
also measurements effort as well. We evaluate our cost-aware transfer learning
solution using real-world configurable software including (i) a robotic system,
(ii) 3 different stream processing applications, and (iii) a NoSQL database
system. The experimental results demonstrate that our approach can achieve (a)
a high prediction accuracy, as well as (b) a high model reliability.Comment: To be published in the proceedings of the 12th International
Symposium on Software Engineering for Adaptive and Self-Managing Systems
(SEAMS'17
Task Runtime Prediction in Scientific Workflows Using an Online Incremental Learning Approach
Many algorithms in workflow scheduling and resource provisioning rely on the
performance estimation of tasks to produce a scheduling plan. A profiler that
is capable of modeling the execution of tasks and predicting their runtime
accurately, therefore, becomes an essential part of any Workflow Management
System (WMS). With the emergence of multi-tenant Workflow as a Service (WaaS)
platforms that use clouds for deploying scientific workflows, task runtime
prediction becomes more challenging because it requires the processing of a
significant amount of data in a near real-time scenario while dealing with the
performance variability of cloud resources. Hence, relying on methods such as
profiling tasks' execution data using basic statistical description (e.g.,
mean, standard deviation) or batch offline regression techniques to estimate
the runtime may not be suitable for such environments. In this paper, we
propose an online incremental learning approach to predict the runtime of tasks
in scientific workflows in clouds. To improve the performance of the
predictions, we harness fine-grained resources monitoring data in the form of
time-series records of CPU utilization, memory usage, and I/O activities that
are reflecting the unique characteristics of a task's execution. We compare our
solution to a state-of-the-art approach that exploits the resources monitoring
data based on regression machine learning technique. From our experiments, the
proposed strategy improves the performance, in terms of the error, up to
29.89%, compared to the state-of-the-art solutions.Comment: Accepted for presentation at main conference track of 11th IEEE/ACM
International Conference on Utility and Cloud Computin
Uncertainty-Aware Performance Prediction for Highly Configurable Software Systems via Bayesian Neural Networks
Configurable software systems are employed in many important application
domains. Understanding the performance of the systems under all configurations
is critical to prevent potential performance issues caused by misconfiguration.
However, as the number of configurations can be prohibitively large, it is not
possible to measure the system performance under all configurations. Thus, a
common approach is to build a prediction model from a limited measurement data
to predict the performance of all configurations as scalar values. However, it
has been pointed out that there are different sources of uncertainty coming
from the data collection or the modeling process, which can make the scalar
predictions not certainly accurate. To address this problem, we propose a
Bayesian deep learning based method, namely BDLPerf, that can incorporate
uncertainty into the prediction model. BDLPerf can provide both scalar
predictions for configurations' performance and the corresponding confidence
intervals of these scalar predictions. We also develop a novel uncertainty
calibration technique to ensure the reliability of the confidence intervals
generated by a Bayesian prediction model. Finally, we suggest an efficient
hyperparameter tuning technique so as to train the prediction model within a
reasonable amount of time whilst achieving high accuracy. Our experimental
results on 10 real-world systems show that BDLPerf achieves higher accuracy
than existing approaches, in both scalar performance prediction and confidence
interval estimation
- …