An Empirical Study on the Discrepancy between Performance Testing Results from Virtual and Physical Environments

Large software systems often undergo performance tests to ensure their capability to handle expected loads. These performance tests often consume large amounts of computing resources and time in order to exercise the system extensively and build confidence on results. Making it worse, the ever evolving field environments require frequent updates to the performance testing environment. In practice, virtual machines (VMs) are widely exploited to provide flexible and less costly environments for performance tests. However, the use of VMs may introduce confounding overhead (e.g., a higher than expected memory utilization with unstable I/O traffic) to the testing environment and lead to unrealistic performance testing results. Yet, little research has studied the impact on test results of using VMs in performance testing activities. In this thesis, we evaluate the discrepancy between the performance testing results from virtual and physical environments. We perform a case study on two open source systems -- namely Dell DVD Store (DS2) and CloudStore. We conduct the same performance tests in both virtual and physical environments and compare the performance testing results based on the three aspects that are typically examined for performance testing results: 1) single performance metric (e.g. CPU usage from virtual environment vs. CPU usage from physical environment), 2) the relationship between two performance metrics (e.g. correlation between CPU usage and I/O traffic) and 3) statistical performance models that are built to predict system performance. Our results show that 1) A single metric from virtual and physical environments do not follow the same distribution, hence practitioners cannot simply use a scaling factor to compare the performance between environments, 2) correlations among performance metrics in virtual environments are different from those in physical environments and 3) statistical models built based on the performance metrics from virtual environments are different from the models built from physical environments suggesting that practitioners cannot use the performance testing results across virtual and physical environments. In order to assist the practitioners leverage performance testing results in both environments, we investigate ways to reduce the discrepancy. We find that such discrepancy may be reduced by normalizing performance metrics based on deviance. Overall, we suggest that practitioners should not use the performance testing results from virtual environment with the simple assumption of a straightforward performance overhead. Instead, practitioners and future research should investigate leveraging normalization techniques to reduce the discrepancy before examining performance testing results from virtual and physical environments

Parametric Estimation of Load for Air Force Datacenters

The Office of Management and Budget (OMB) has tasked Federal agencies to develop a Data Center Consolidation Plan. Effective planning requires a repeatable method to effectively and efficiently size Air Force Base-level data centers. Review of commercial literature on data center design found emphasis in power efficiency, thermal modeling and cooling, and network speed and availability. The topic of sizing data center processing capacity seems undeveloped. This thesis provides a better, pedigreed solution to the data center sizing problem. By analogy, Erlang\u27s formulae for the probability of blocking and queuing should be applicable to cumulative CPU utilization in a data center. Using survey data collected by 38th Engineering Squadron, a simulation is built and correlation between the observed survey measurements and simulation measurements, and the Erlang, Gamma, and Gaussian-Normal distributions is found. For a sample dataset of 70 servers over 14 hours of observation and a supposed .99999 requirement for traffic to be passed or otherwise unimpeded, Erlang distribution predicts 10 CPU cores are required, Gamma distribution predicts 10 CPU cores are required, Gaussian-Normal distribution predicts 9 CPU cores are required, Erlang B formulae predicts 14 CPU cores are required, and Erlang C formulae predicts 15 CPU cores are required

Automatic Comparison of Load Tests to Support the Performance Analysis of Large Enterprise Systems

Abstract — Load testing is crucial to uncover functional and performance bugs in large-scale systems. Load tests generate vast amounts of performance data, which needs to be compared and analyzed in limited time across tests. This helps performance analysts to understand the resource usage of an application and to find out if an application is meeting its performance goals. The biggest challenge for performance analysts is to identify the few important performance counters in the highly redundant performance data. In this paper, we employed a statistical technique, Principal Component Analysis (PCA) to reduce the large volume of performance counter data, to a smaller, more meaningful and manageable set. Furthermore, our methodology automates the process of comparing the important counters across load tests to identify performance gains/losses. A case study on load test data of a large enterprise application shows that our methodology can effectively guide performance analysts to identify and compare top performance counters across tests in limited time. Keywords-Load Test; Performance Counters; PCA I