Performance metrics for consolidated servers

In spite of the widespread adoption of virtualization and consol- idation, there exists no consensus with respect to how to bench- mark consolidated servers that run multiple guest VMs on the same physical hardware. For example, VMware proposes VMmark which basically computes the geometric mean of normalized throughput values across the VMs; Intel uses vConsolidate which reports a weighted arithmetic average of normalized throughput values. These benchmarking methodologies focus on total system through- put (i.e., across all VMs in the system), and do not take into account per-VM performance. We argue that a benchmarking methodology for consolidated servers should quantify both total system through- put and per-VM performance in order to provide a meaningful and precise performance characterization. We therefore present two performance metrics, Total Normalized Throughput (TNT) to characterize total system performance, and Average Normalized Reduced Throughput (ANRT) to characterize per-VM performance. We compare TNT and ANRT against VMmark using published performance numbers, and report several cases for which the VM- mark score is misleading. This is, VMmark says one platform yields better performance than another, however, TNT and ANRT show that both platforms represent different trade-offs in total system throughput versus per-VM performance. Or, even worse, in a cou- ple cases we observe that VMmark yields opposite conclusions than TNT and ANRT, i.e., VMmark says one system performs better than another one which is contradicted by TNT/ANRT performance characterization

Public Performance Metrics: Driving Physician Motivation and Performance

Introduction: As providers transition from “fee-for-service” to “pay-for-performance” models, focus has shifted to improving performance. This trend extends to the emergency department (ED) where visits continue to increase across the United States. Our objective was to determine whether displaying public performance metrics of physician triage data could drive intangible motivators and improve triage performance in the ED.Methods: This is a single institution, time-series performance study on a physician-in-triage system. Individual physician baseline metrics—number of patients triaged and dispositioned per shift—were obtained and prominently displayed with identifiable labels during each quarterly physician group meeting. Physicians were informed that metrics would be collected and displayed quarterly and that there would be no bonuses, punishments, or required training; physicians were essentially free to do as they wished. It was made explicit that the goal was to increase the number triaged, and while the number dispositioned would also be displayed, it would not be a focus, thereby acting as this study’s control. At the end of one year, we analyzed metrics.Results: The group’s average number of patients triaged per shift were as follows: Q1-29.2; Q2-31.9; Q3-34.4; Q4-36.5 (Q1 vs Q4, p < 0.00001). The average numbers of patients dispositioned per shift were Q1-16.4; Q2-17.8; Q3-16.9; Q4-15.3 (Q1 vs Q4, p = 0.14). The top 25% of Q1 performers increased their average numbers triaged from Q1-36.5 to Q4-40.3 (ie, a statistically insignificant increase of 3.8 patients per shift [p = 0.07]). The bottom 25% of Q1 performers, on the other hand, increased their averages from Q1-22.4 to Q4-34.5 (ie, a statistically significant increase of 12.2 patients per shift [p = 0.0013]).Conclusion: Public performance metrics can drive intangible motivators (eg, purpose, mastery, and peer pressure), which can be an effective, low-cost strategy to improve individual performance, achieve institutional goals, and thrive in the pay-for-performance era

MOL-Eye: A New Metric for the Performance Evaluation of a Molecular Signal

Inspired by the eye diagram in classical radio frequency (RF) based communications, the MOL-Eye diagram is proposed for the performance evaluation of a molecular signal within the context of molecular communication. Utilizing various features of this diagram, three new metrics for the performance evaluation of a molecular signal, namely the maximum eye height, standard deviation of received molecules, and counting SNR (CSNR) are introduced. The applicability of these performance metrics in this domain is verified by comparing the performance of binary concentration shift keying (BCSK) and BCSK with consecutive power adjustment (BCSK-CPA) modulation techniques in a vessel-like environment with laminar flow. The results show that, in addition to classical performance metrics such as bit-error rate and channel capacity, these performance metrics can also be used to show the advantage of an efficient modulation technique over a simpler one

Performance measurement of IT service management: a case study of an Australian university

IT departments are adopting service orientation by implementing IT service management (ITSM) frameworks. Most organisations are hesitant to discuss their ITSM performance measurement practices, tending to focus more on challenges. However there are good practices that are found amidst the challenges. We present a case study that provides an account of the performance measurement practices in the ICT Division of an Australian university. This case study was conducted with the aim of understanding the internal and external factors that influence the selection of ITSM performance metrics. It also explores how and why metrics and frameworks are used to measure the performance of ITSM in organisations. Interviews were conducted to identify the specific ITSM performance metrics used and how they were derived. It was found that a number of factors internal and external to the organisation influenced the selection of the performance metrics. The internal factors include meeting the need for improved governance, alignment of IT strategy with organisation strategy, having a mechanism to provide feedback to IT customers (university staff and students). External factors include benchmarking against others in the same industry and the choice of metrics offered by ITSM software tool adopted

Exploring Symmetry of Binary Classification Performance Metrics

Selecting the proper performance metric constitutes a key issue for most classification problems in the field of machine learning. Although the specialized literature has addressed several topics regarding these metrics, their symmetries have yet to be systematically studied. This research focuses on ten metrics based on a binary confusion matrix and their symmetric behaviour is formally defined under all types of transformations. Through simulated experiments, which cover the full range of datasets and classification results, the symmetric behaviour of these metrics is explored by exposing them to hundreds of simple or combined symmetric transformations. Cross-symmetries among the metrics and statistical symmetries are also explored. The results obtained show that, in all cases, three and only three types of symmetries arise: labelling inversion (between positive and negative classes); scoring inversion (concerning good and bad classifiers); and the combination of these two inversions. Additionally, certain metrics have been shown to be independent of the imbalance in the dataset and two cross-symmetries have been identified. The results regarding their symmetries reveal a deeper insight into the behaviour of various performance metrics and offer an indicator to properly interpret their values and a guide for their selection for certain specific applications.University of Seville (Spain) by Telefónica Chair “Intelligence in Networks

True Performance Metrics in Electrochemical Energy Storage

A dramatic expansion of research in the area of electrochemical energy storage (EES) during the past decade has been driven by the demand for EES in handheld electronic devices, transportation, and storage of renewable energy for the power grid (1–3). However, the outstanding properties reported for new electrode materials may not necessarily be applicable to performance of electrochemical capacitors (ECs). These devices, also called supercapacitors or ultra-capacitors (4), store charge with ions from solution at charged porous electrodes. Unlike batteries, which store large amounts of energy but deliver it slowly, ECs can deliver energy faster (develop high power), but only for a short time. However, recent work has claimed energy densities for ECs approaching (5) or even exceeding that of batteries. We show that even when some metrics seem to support these claims, actual device performance may be rather mediocre. We will focus here on ECs, but these considerations also apply to lithium (Li)—ion batteries