Demo: An Interoperability Development and Performance Diagnosis Environment

Interoperability is key to widespread adoption of sensor network technology, but interoperable systems have traditionally been difficult to develop and test. We demonstrate an interoperable system development and performance diagnosis environment in which different systems, different software, and different hardware can be simulated in a single network configuration. This allows both development, verification, and performance diagnosis of interoperable systems. Estimating the performance is important since even when systems interoperate, the performance can be sub-optimal, as shown in our companion paper that has been conditionally accepted for SenSys 2011

Automating Performance Diagnosis in Networked Systems

Diagnosing performance degradation in distributed systems is a complex and difficult task. Software that performs well in one environment may be unusably slow in another, and determining the root cause is time-consuming and error-prone, even in environments in which all the data may be available. End users have an even more difficult time trying to diagnose system performance, since both software and network problems have the same symptom: a stalled application. The central thesis of this dissertation is that the source of performance stalls in a distributed system can be automatically detected and diagnosed with very limited information: the dependency graph of data flows through the system, and a few counters common to almost all data processing systems. This dissertation presents FlowDiagnoser, an automated approach for diagnosing performance stalls in networked systems. FlowDiagnoser requires as little as two bits of information per module to make a diagnosis: one to indicate whether the module is actively processing data, and one to indicate whether the module is waiting on its dependents. To support this thesis, FlowDiagnoser is implemented in two distinct environments: an individual host's networking stack, and a distributed streams processing system. In controlled experiments using real applications, FlowDiagnoser correctly diagnoses 99% of networking-related stalls due to application, connection-specific, or network-wide performance problems, with a false positive rate under 3%. The prototype system for diagnosing messaging stalls in a commercial streams processing system correctly finds 93% of message-processing stalls, with a false positive rate of 2%

Performance monitoring of MPC based on dynamic principal component analysis

A unified framework based on the dynamic principal component analysis (PCA) is proposed for performance monitoring of constrained multi-variable model predictive control (MPC) systems. In the proposed performance monitoring framework, the dynamic PCA based performance benchmark is adopted for performance assessment, while performance diagnosis is carried out using a unified weighted dynamic PCA similarity measure. Simulation results obtained from the case study of the Shell process demonstrate that the use of the dynamic PCA performance benchmark can detect the performance deterioration more quickly compared with the traditional PCA method, and the proposed unified weighted dynamic PCA similarity measure can correctly locate the root cause for poor performance of MPC controller

A data-based approach for multivariate model predictive control performance monitoring

An intelligent statistical approach is proposed for monitoring the performance of multivariate model predictive control (MPC) controller, which systematically integrates both the assessment and diagnosis procedures. Model predictive error is included into the monitored variable set and a 2-norm based covariance benchmark is presented. By comparing the data of a monitored operational period with the "golden" user-predefined one, this method can properly evaluate the performance of an MPC controller at the monitored operational stage. Characteristic direction information is mined from the operating data and the corresponding classes are built. The eigenvector angle is defined to describe the similarity between the current data set and the established classes, and an angle-based classifier is introduced to identify the root cause of MPC performance degradation when a poor performance is detected. The effectiveness of the proposed methodology is demonstrated in a case study of the Wood–Berry distillation column system

A method for performance diagnosis and evaluation of video trackers

Several measures for evaluating multi-target video trackers exist that generally aim at providing ‘end performance.’ End performance is important particularly for ranking and comparing trackers. However, for a deeper insight into trackers’ performance it would also be desirable to analyze key contributory factors (false positives, false negatives, ID changes) that (implicitly or explicitly) lead to the attainment of a certain end performance. Specifically, this paper proposes a new approach to enable a diagnosis of the performance of multi-target trackers as well as providing a means to determine the end performance to still enable their comparison in a video sequence. Diagnosis involves analyzing probability density functions of false positives, false negatives and ID changes of trackers in a sequence. End performance is obtained in terms of the extracted performance scores related to false positives, false negatives and ID changes. In the experiments, we used four state-of-the-art trackers on challenging real-world public datasets to show the effectiveness of the proposed approach