An Empirical Study on the Discrepancy between Performance Testing Results from Virtual and Physical Environments

Large software systems often undergo performance tests to ensure their capability to handle expected loads. These performance tests often consume large amounts of computing resources and time in order to exercise the system extensively and build confidence on results. Making it worse, the ever evolving field environments require frequent updates to the performance testing environment. In practice, virtual machines (VMs) are widely exploited to provide flexible and less costly environments for performance tests. However, the use of VMs may introduce confounding overhead (e.g., a higher than expected memory utilization with unstable I/O traffic) to the testing environment and lead to unrealistic performance testing results. Yet, little research has studied the impact on test results of using VMs in performance testing activities. In this thesis, we evaluate the discrepancy between the performance testing results from virtual and physical environments. We perform a case study on two open source systems -- namely Dell DVD Store (DS2) and CloudStore. We conduct the same performance tests in both virtual and physical environments and compare the performance testing results based on the three aspects that are typically examined for performance testing results: 1) single performance metric (e.g. CPU usage from virtual environment vs. CPU usage from physical environment), 2) the relationship between two performance metrics (e.g. correlation between CPU usage and I/O traffic) and 3) statistical performance models that are built to predict system performance. Our results show that 1) A single metric from virtual and physical environments do not follow the same distribution, hence practitioners cannot simply use a scaling factor to compare the performance between environments, 2) correlations among performance metrics in virtual environments are different from those in physical environments and 3) statistical models built based on the performance metrics from virtual environments are different from the models built from physical environments suggesting that practitioners cannot use the performance testing results across virtual and physical environments. In order to assist the practitioners leverage performance testing results in both environments, we investigate ways to reduce the discrepancy. We find that such discrepancy may be reduced by normalizing performance metrics based on deviance. Overall, we suggest that practitioners should not use the performance testing results from virtual environment with the simple assumption of a straightforward performance overhead. Instead, practitioners and future research should investigate leveraging normalization techniques to reduce the discrepancy before examining performance testing results from virtual and physical environments

Experimental study and modelling of Networked Virtual Environment server traffic

The paradigm of virtual world environment arises as an useful tool in diverse fields such as e-Health or education, where they provide a new way of communication and interaction with end users. Networking capabilities play an important role in these systems, which motivates the study and understanding of the gaming network traffic. The present work focuses on Open Wonderland, a system that provides the basis for the development of Networked Virtual Environments with educational or health purposes. The goal of this paper is defining a testing environment and modelling the behaviour of the outgoing network traffic at the server side.Ministerio de Industria, Comercio y Turismo AAL-010000-2012-10Ministerio de Ciencia e Innovación TEC2009-10639-C04-0

Virtuality in human supervisory control: Assessing the effects of psychological and social remoteness

Virtuality would seem to offer certain advantages for human supervisory control. First, it could provide a physical analogue of the 'real world' environment. Second, it does not require control room engineers to be in the same place as each other. In order to investigate these issues, a low-fidelity simulation of an energy distribution network was developed. The main aims of the research were to assess some of the psychological concerns associated with virtual environments. First, it may result in the social isolation of the people, and it may have dramatic effects upon the nature of the work. Second, a direct physical correspondence with the 'real world' may not best support human supervisory control activities. Experimental teams were asked to control an energy distribution network. Measures of team performance, group identity and core job characteristics were taken. In general terms, the results showed that teams working in the same location performed better than team who were remote from one another

The effects of changing projection geometry on perception of 3D objects on and around tabletops

Funding: Natural Sciences and Engineering Research Council of Canada Networks of Centres of Excellence of Canada.Displaying 3D objects on horizontal displays can cause problems in the way that the virtual scene is presented on the 2D surface; inappropriate choices in how 3D is represented can lead to distorted images and incorrect object interpretations. We present four experiments that test 3D perception. We varied projection geometry in three ways: type of projection (perspective/parallel), separation between the observer’s point of view and the projection’s center (discrepancy), and the presence of motion parallax (with/without parallax). Projection geometry had strong effects different for each task. Reducing discrepancy is desirable for orientation judgments, but not for object recognition or internal angle judgments. Using a fixed center of projection above the table reduces error and improves accuracy in most tasks. The results have far-reaching implications for the design of 3D views on tables, in particular for multi-user applications where projections that appear correct for one person will not be perceived correctly by another.PostprintPeer reviewe

Network traffic characterisation, analysis, modelling and simulation for networked virtual environments

Networked virtual environment (NVE) refers to a distributed software system where a simulation, also known as virtual world, is shared over a data network between several users that can interact with each other and the simulation in real-time. NVE systems are omnipresent in the present globally interconnected world, from entertainment industry, where they are one of the foundations for many video games, to pervasive games that focus on e-learning, e-training or social studies. From this relevance derives the interest in better understanding the nature and internal dynamics of the network tra c that vertebrates these systems, useful in elds such as network infrastructure optimisation or the study of Quality of Service and Quality of Experience related to NVE-based services. The goal of the present work is to deepen into this understanding of NVE network tra c by helping to build network tra c models that accurately describe it and can be used as foundations for tools to assist in some of the research elds enumerated before. First contribution of the present work is a formal characterisation for NVE systems, which provides a tool to determine which systems can be considered as NVE. Based on this characterisation it has been possible to identify numerous systems, such as several video games, that qualify as NVE and have an important associated literature focused on network tra c analysis. The next contribution has been the study of this existing literature from a NVE perspective and the proposal of an analysis pipeline, a structured collection of processes and techniques to de ne microscale network models for NVE tra c. This analysis pipeline has been tested and validated against a study case focused on Open Wonderland (OWL), a framework to build NVE systems of di erent purpose. The analysis pipeline helped to de ned network models from experimental OWL tra c and assessed on their accuracy from a statistical perspective. The last contribution has been the design and implementation of simulation tools based on the above OWL models and the network simulation framework ns-3. The purpose of these simulations was to con rm the validity of the OWL models and the analysis pipeline, as well as providing potential tools to support studies related to NVE network tra c. As a result of this nal contribution, it has been proposed to exploit the parallelisation potential of these simulations through High Throughput Computing techniques and tools, aimed to coordinate massively parallel computing workloads over distributed resources

CUshop: A Simulated Shopping Environment Fostering Consumer-Centric Packaging Design & Testing

Consumer product packaging provides product damage protection, extends product shelf life, and communicates product usage instructions to the consumer. Its collective contribution to the waste stream is notorious, but its role in product salability is much less understood. Consumers now make the majority of product purchase decisions while present at the shelf, and since they do it very quickly (within 5-8 seconds), and do not appear to adhere to strong brand loyalty as was once more common, packaging (and more specifically, its aesthetics and contrast with its competitors) plays a dominant role in the decision-making process. It is difficult, however, to measure and predict the effectiveness of package design via empirical consumer response testing, and even more challenging to seamlessly integrate consumer response measures into the package design process. The key to meaningful measurement of consumer behavior in the package design process is immersion of the consumer in a convincing environment that elicits natural shopping behavior. While an actual retail store offers the most realistic environment, controlling experimental conditions in this setting is problematic. An artificial simulation of such an environment is desirable for reasons of efficiency, cost, and flexibility. CUshop, a unique laboratory mixing physical store elements with those akin to virtual reality simulation, is introduced. The laboratory has been created with the goal of priming participants into a shopping context, or shopping frame of mind, prompting realistic consumer behavior that can be measured and studied via objective forms of measurement (e.g., eye tracking)