Detecting Flow Anomalies in Distributed Systems

Deep within the networks of distributed systems, one often finds anomalies that affect their efficiency and performance. These anomalies are difficult to detect because the distributed systems may not have sufficient sensors to monitor the flow of traffic within the interconnected nodes of the networks. Without early detection and making corrections, these anomalies may aggravate over time and could possibly cause disastrous outcomes in the system in the unforeseeable future. Using only coarse-grained information from the two end points of network flows, we propose a network transmission model and a localization algorithm, to detect the location of anomalies and rank them using a proposed metric within distributed systems. We evaluate our approach on passengers' records of an urbanized city's public transportation system and correlate our findings with passengers' postings on social media microblogs. Our experiments show that the metric derived using our localization algorithm gives a better ranking of anomalies as compared to standard deviation measures from statistical models. Our case studies also demonstrate that transportation events reported in social media microblogs matches the locations of our detect anomalies, suggesting that our algorithm performs well in locating the anomalies within distributed systems

Are the dimensions of private information more multiple than expected? Information asymmetries in the market of supplementary private health insurance in England

Our study reexamines standard econometric approaches for the detection of information asymmetries on insurance markets. We claim that evidence based on a standard framework with 2 equations, which uses potential sources of information asymmetries, should stress the importance of heterogeneity in the parameters. We argue that conclusions derived from this methodology can be misleading if the estimated coefficients in such an `unused characteristics' framework are driven by different parts of the population. We show formally that an individual's expected risk from the perspective of insurance, conditioned on certain characteristics (which are not used for calculating the risk premium), can equal the population's expectation in risk { although such characteristics are both related to risk and insurance probability, which is usually interpreted as an indicator of information asymmetries. We provide empirical evidence on the existence of information asymmetries in the market for supplementary private health insurance in the UK. Overall, we found evidence for advantageous selection into the private risk pool; ie people with lower health risk tend to insure more. The main drivers of this phenomenon seem to be characteristics such as income and wealth. Nevertheless, we also found parameter heterogeneity to be relevant, leading to possible misinterpretation if the standard `unused characteristics' approach is applied

Testing methods and techniques - A compilation

Nondestructive test methods and techniques including computer programs and data reductio

Distribution Grid Line Outage Detection with Privacy Data

Change point detection is important for many real-world applications. While sensor readings enable line outage identification, they bring privacy concerns by allowing an adversary to divulge sensitive information such as household occupancy and economic status. In this paper, to preserve privacy, we develop a decentralized randomizing scheme to ensure no direct exposure of each user's raw data. Brought by the randomizing scheme, the trade-off between privacy gain and degradation of change point detection performance is quantified via studying the differential privacy framework and the Kullback-Leibler divergence. Furthermore, we propose a novel statistic to mitigate the impact of randomness, making our detection procedure both privacy-preserving and have optimal performance. The results of comprehensive experiments show that our proposed framework can effectively find the outage with privacy guarantees.Comment: 5 page

Statistical Active Learning Algorithms for Noise Tolerance and Differential Privacy

We describe a framework for designing efficient active learning algorithms that are tolerant to random classification noise and are differentially-private. The framework is based on active learning algorithms that are statistical in the sense that they rely on estimates of expectations of functions of filtered random examples. It builds on the powerful statistical query framework of Kearns (1993). We show that any efficient active statistical learning algorithm can be automatically converted to an efficient active learning algorithm which is tolerant to random classification noise as well as other forms of "uncorrelated" noise. The complexity of the resulting algorithms has information-theoretically optimal quadratic dependence on $1/(1-2\eta)$, where $\eta$ is the noise rate. We show that commonly studied concept classes including thresholds, rectangles, and linear separators can be efficiently actively learned in our framework. These results combined with our generic conversion lead to the first computationally-efficient algorithms for actively learning some of these concept classes in the presence of random classification noise that provide exponential improvement in the dependence on the error $\epsilon$ over their passive counterparts. In addition, we show that our algorithms can be automatically converted to efficient active differentially-private algorithms. This leads to the first differentially-private active learning algorithms with exponential label savings over the passive case.Comment: Extended abstract appears in NIPS 201

A reasonable benchmarking frontier using DEA : an incentive scheme to improve efficiency in public hospitals

There exists research relating management concepts with productivity measurement methods that offers useful solutions for improving management control in the public sector. Within this sphere, we connect agency theory with efficiency analysis and describe how to define an incentives scheme that can be applied in the public sector to monitor the efficiency and productivity of managers. To fulfill the main objective of this research, we propose an iterative process for determining what we define as a ‘reasonable frontier’, a concept that provides the foundation required to establish the incentive scheme for the managers. Our ‘reasonable frontier’ has the following properties: i) it detects the presence of outliers, ii) it proposes a procedure to establish the influence introduced by extreme observations, and iii) it sorts out the problem of data masking. The proposed method is applied to a sample of hospitals taken from the public network of the Spanish health service. The results obtained confirm the applicability of the proposal made. Summing up, we define and apply a useful method, combining aspects of agency theory and efficiency analysis, which is of interest to those public authorities trying to design effective incentive schemes which influence the decision making of the public managers