virtFlow: guest independent execution flow analysis across virtualized environments

An agent-less technique to understand virtual machines (VMs) behavior and their changes during the VM life-cycle is essential for many performance analysis and debugging tasks in the cloud environment. Because of privacy and security issues, ease of deployment and execution overhead, the method preferably limits its data collection to the physical host level, without internal access to the VMs. We propose a host-based, precise method to recover execution flow of virtualized environments, regardless of the level of virtualization. Given a VM, the Any-Level VM Detection Algorithm (ADA) and Nested VM State Detection (NSD) Algorithm compute its execution path along with the state of virtual CPUs (vCPUs) from the host kernel trace. The state of vCPUs is displayed in an interactive trace viewer (TraceCompass) for further inspection. Then, a new approach for profiling threads and processes inside the VMs is proposed. Our proposed VM trace analysis algorithms have been open-sourced for further enhancements and to the benefit of other developers. Our new techniques are being evaluated with workloads generated by different benchmarking tools. These approaches are based on host hypervisor tracing, which brings a lower overhead (around 1%) as compared to other approaches

Virtual Machine Flow Analysis Using Host Kernel Tracing

L’infonuagique a beaucoup gagné en popularité car elle permet d’offrir des services à coût réduit, avec le modèle économique Pay-to-Use, un stockage illimité avec les systèmes de stockage distribué, et une grande puissance de calcul grâce à l’accès direct au matériel. La technologie de virtualisation permet de partager un serveur physique entre plusieurs environnements virtualisés isolés, en déployant une couche logicielle (Hyperviseur) au-dessus du matériel. En conséquence, les environnements isolés peuvent fonctionner avec des systèmes d’exploitation et des applications différentes, sans interférence mutuelle. La croissance du nombre d’utilisateurs des services infonuagiques et la démocratisation de la technologie de virtualisation présentent un nouveau défi pour les fournisseurs de services infonuagiques. Fournir une bonne qualité de service et une haute disponibilité est une exigence principale pour l’infonuagique. La raison de la dégradation des performances d’une machine virtuelle peut être nombreuses. a Activité intense d’une application à l’intérieur de la machine virtuelle. b Conflits avec d’autres applications à l’intérieur de la machine même virtuelle. c Conflits avec d’autres machines virtuelles qui roulent sur la même machine physique. d Échecs de la plateforme infonuagique. Les deux premiers cas peuvent être gérés par le propriétaire de la machine virtuelle et les autres cas doivent être résolus par le fournisseur de l’infrastructure infonuagique. Ces infrastructures sont généralement très complexes et peuvent contenir différentes couches de virtualisation. Il est donc nécessaire d’avoir un outil d’analyse à faible surcoût pour détecter ces types de problèmes. Dans cette thèse, nous présentons une méthode précise permettant de récupérer le flux d’exécution des environnements virtualisés à partir de la machine hôte, quel que soit le niveau de la virtualisation. Pour éviter des problèmes de sécurité, faciliter le déploiement et minimiser le surcoût, notre méthode limite la collecte de données au niveau de l’hyperviseur. Pour analyser le comportement des machines virtuelles, nous utilisons un outil de traçage léger appelé Linux Trace Toolkit Next Generation (LTTng) [1]. LTTng est capable d’effectuer un traçage à haut débit et à faible surcoût, grâce aux mécanismes de synchronisation sans verrous utilisés pour mettre à jour le contenu des tampons de traçage.----------ABSTRACT: Cloud computing has gained popularity as it offers services at lower cost, with Pay-per-Use model, unlimited storage, with distributed storage, and flexible computational power, with direct hardware access. Virtualization technology allows to share a physical server, between several isolated virtualized environments, by deploying an hypervisor layer on top of hardware. As a result, each isolated environment can run with its OS and application without mutual interference. With the growth of cloud usage, and the use of virtualization, performance understanding and debugging are becoming a serious challenge for Cloud providers. Offering a better QoS and high availability are expected to be salient features of cloud computing. Nonetheless, possible reasons behind performance degradation in VMs are numerous. a) Heavy load of an application inside the VM. b) Contention with other applications inside the VM. c) Contention with other co-located VMs. d) Cloud platform failures. The first two cases can be managed by the VM owner, while the other cases need to be solved by the infrastructure provider. One key requirement for such a complex environment, with different virtualization layers, is a precise low overhead analysis tool. In this thesis, we present a host-based, precise method to recover the execution flow of virtualized environments, regardless of the level of nested virtualization. To avoid security issues, ease deployment and reduce execution overhead, our method limits its data collection to the hypervisor level. In order to analyse the behavior of each VM, we use a lightweight tracing tool called the Linux Trace Toolkit Next Generation (LTTng) [1]. LTTng is optimised for high throughput tracing with low overhead, thanks to its lock-free synchronization mechanisms used to update the trace buffer content

VM processes state detection by hypervisor tracing

The diagnosis of performance issues in cloud environments is a challenging problem, due to the different levels of virtualization, the diversity of applications and their interactions on the same physical host. Moreover, because of privacy, security, ease of deployment and execution overhead, an agent-less method, which limits its data collection to the physical host level, is often the only acceptable solution. In this paper, a precise host-based method, to recover wait state for the processes inside a given Virtual Machine (VM), is proposed. The virtual Process State Detection (vPSD) algorithm computes the state of processes through host kernel tracing. The state of a virtual Process (vProcess) is displayed in an interactive trace viewer (Trace Compass) for further inspection. Our proposed VM trace analysis algorithm has been open-sourced for further enhancements and for the benefit of other developers. Experimental evaluations were conducted using a mix of workload types (CPU, Disk, and Network), with different applications like Hadoop, MySQL, and Apache. vPSD, being based on host hypervisor tracing, brings a lower overhead (around 0.03%) as compared to other approaches

Virtual CPU state detection and execution flow analysis by host tracing

Cloud computing offers to the end user the ability of accessing a pool of resources with the Pay as Use (PaU) model. By leveraging this technology, users can benefit from hardware virtualization for on-demand resource acquisition and rapid elasticity. However, there is no effective tool to analyze virtual hardware performance, especially when isolation between these virtual resources is not adequate. The existing tools need to access and trace the whole activity of the VM and host. However, in most cases, tracing the virtual machine (VM) is not possible because of security issues and the added overhead. Therefore, there is a need for a tool to troubleshoot unexpected behavior of VMs without internal access for tracing or debugging. In this paper, we propose a new method to study the state of CPUs inside VMs without internal access. Our tool can detect unexpected delays and their root causes. We developed a virtual CPU (vCPU) state analyser to detect the state of vCPUs along with the reason for being in that state. This approach relies on host tracing, thus adding less overhead to VMs as compared to existing approaches. Then we propose a new approach for profiling threads inside the VMs by host tracing. We implemented different views for the TraceCompass trace viewer to let the administrator visually track different threads and their states inside the VMs. Our tool can detect different problems such as overcommitment of resources

Wait analysis of virtual machines using host kernel tracing

An agent-less method to understand virtual machines (VMs) behavior its evolution during the VM life-cycle is an essential task for IaaS provider. It allows the IaaS provider to better scale the VMs resources by properly allocating the physical resources. On the other hand, because of privacy, security, ease of deployment and execution overhead issues, the method presented limits its data collection to the physical host level, without internal access to the VMs. We propose a host-based, precise method to recover wait states for the virtual CPUs (vCPUs) of a given VM. The Wait Analysis Algorithm (W2A) computes the state of vCPUs through the host kernel trace. The state of vCPUs is displayed in an interactive trace viewer (TraceCompass) for further inspection. Our proposed VM trace analysis algorithm has been open-sourced for further enhancements and to the benefit of other developers. Our new technique is being evaluated with representative workloads, generated by different benchmarking tools. These approaches are based on host hypervisor tracing, which brings a lower overhead (around 0.03%) as compared to other approaches

Fine-grained nested virtual machine performance analysis through first level hypervisor tracing

Nowadays, nested VMs are often being used to address compatibility issues, security concerns, software scaling and continuous integration scenarios. With the increased adoption of nested VMs, there is a need for newer techniques to troubleshoot any unexpected behavior. Because of privacy and security issues, ease of deployment and execution overhead, these investigation techniques should preferably limit their data collection in most cases to the physical host level, without internal access to the VMs. This paper introduces the Nested Virtual Machine Detection Algorithm (NDA) - a host hypervisor based analysis method which can investigate the performance of nested VMs. NDA can uncover the CPU overhead entailed by the host hypervisor and guest hypervisors, and compare it to the CPU usage of Nested VMs. We further developed several graphical views, for the TraceCompass trace visualization tool, to display the virtual CPUs of VMs and their corresponding nested VMs, along with their states. These approaches are based on host hypervisor tracing, which brings a lower overhead (around 1%) as compared to other approaches. Based on our analysis and the implemented graphical views, our techniques can quickly detect different problems and their root causes, such as unexpected delays inside nested VMs

Low overhead hardware-assisted virtual machine analysis and profiling

Cloud infrastructure providers need reliable performance analysis tools for their nodes. Moreover, the analysis of Virtual Machines (VMs) is a major requirement in quantifying cloud performance. However, root cause analysis, in case of unexpected crashes or anomalous behavior in VMs, remains a major challenge. Modern tracing tools such as LTTng allow fine grained analysis - albeit at a minimal execution overhead,and being OS dependent. In this paper, we propose HAVAna,a hardware-assisted VM analysis algorithm that gathers and analyzes pure hardware trace data, without any dependence on the underlying OS or performance analysis infrastructure. Our approach is totally non-intrusive and does not require any performance statistics, trace or log gathering from the VM. We used the recently introduced Intel PT ISA extensions on modern Intel Skylake processors to demonstrate its efficiency and observed that, in our experimental scenarios, it leads to a tiny overhead of up to 1%, as compared to 3.6-28.7% for similar VM trace analysis done with software-only schemes such as LTTng. Our proposed VM trace analysis algorithm has also been opensourced for further enhancements and to the benefit of other developers. Furthermore, we developed interactive Resource and Process Control Flow visualization tools to analyze the hardware trace data and present a real-life usecase in the paper that allowed us to see unexpected resource consumption by VMs

Virtual Radar: A Novel and Advanced Tool for Monitoring Virtualized Networks

The IP Multimedia Subsystem (IMS) is a core element of Next Generation Networks (NGNs), which aim at offering users feature-rich and QoS-enabled high-speed multimedia services. With the widespread usage of cloud computing, network function virtualization has emerged as a way to cloudify and virtualize networking resources. However, the complexity of cloud-based virtualized networks requires accurate and efficient monitoring of resources, in order to detect problems and oversee the network\u27s operation. Few monitoring approaches and tools have been proposed in the literature. However, none of those solutions is lightweight, resource- efficient, or tailored to the needs of complex virtualized networks. In this work, we address this limitation by proposing a sophisticated and resource-efficient monitoring approach for virtualized networks - our tool is dubbed Virtual Radar. We use the virtualized IMS as a case study for testing and demonstrating the capabilities of our system. Compared to the existing monitoring tools, Virtual Radar gives a comprehensive view of a large number of high-level and low-level networking parameters and does so while imposing minimal monitoring overhead, offering a high monitoring resolution and high scalability - thus making it a valuable tool for virtual networks\u27 monitoring