Towards Operator-less Data Centers Through Data-Driven, Predictive, Proactive Autonomics

Continued reliance on human operators for managing data centers is a major impediment for them from ever reaching extreme dimensions. Large computer systems in general, and data centers in particular, will ultimately be managed using predictive computational and executable models obtained through data-science tools, and at that point, the intervention of humans will be limited to setting high-level goals and policies rather than performing low-level operations. Data-driven autonomics, where management and control are based on holistic predictive models that are built and updated using live data, opens one possible path towards limiting the role of operators in data centers. In this paper, we present a data-science study of a public Google dataset collected in a 12K-node cluster with the goal of building and evaluating predictive models for node failures. Our results support the practicality of a data-driven approach by showing the effectiveness of predictive models based on data found in typical data center logs. We use BigQuery, the big data SQL platform from the Google Cloud suite, to process massive amounts of data and generate a rich feature set characterizing node state over time. We describe how an ensemble classifier can be built out of many Random Forest classifiers each trained on these features, to predict if nodes will fail in a future 24-hour window. Our evaluation reveals that if we limit false positive rates to 5%, we can achieve true positive rates between 27% and 88% with precision varying between 50% and 72%.This level of performance allows us to recover large fraction of jobs' executions (by redirecting them to other nodes when a failure of the present node is predicted) that would otherwise have been wasted due to failures. [...

A monitoring approach for runtime service discovery

Effective runtime service discovery requires identification of services based on different service characteristics such as structural, behavioural, quality, and contextual characteristics. However, current service registries guarantee services described in terms of structural and sometimes quality characteristics and, therefore, it is not always possible to assume that services in them will have all the characteristics required for effective service discovery. In this paper, we describe a monitor-based runtime service discovery framework called MoRSeD. The framework supports service discovery in both push and pull modes of query execution. The push mode of query execution is performed in parallel to the execution of a service-based system, in a proactive way. Both types of queries are specified in a query language called SerDiQueL that allows the representation of structural, behavioral, quality, and contextual conditions of services to be identified. The framework uses a monitor component to verify if behavioral and contextual conditions in the queries can be satisfied by services, based on translations of these conditions into properties represented in event calculus, and verification of the satisfiability of these properties against services. The monitor is also used to support identification that services participating in a service-based system are unavailable, and identification of changes in the behavioral and contextual characteristics of the services. A prototype implementation of the framework has been developed. The framework has been evaluated in terms of comparison of its performance when using and when not using the monitor component

This Time It's Personal: from PIM to the Perfect Digital Assistant

Interacting with digital PIM tools like calendars, to-do lists, address books, bookmarks and so on, is a highly manual, often repetitive and frequently tedious process. Despite increases in memory and processor power over the past two decades of personal computing, not much has changed in the way we engage with such applications. We must still manually decompose frequently performed tasks into multiple smaller, data specific processes if we want to be able to recall or reuse the information in some meaningful way. "Meeting with Yves at 5 in Stata about blah" breaks down into rigid, fixed semantics in separate applications: data to be recorded in calendar fields, address book fields and, as for the blah, something that does not necessarily exist as a PIM application data structure. We argue that a reason Personal Information Management tools may be so manual, and so effectively fragmented, is that they are not personal enough. If our information systems were more personal, that is, if they knew in a manner similar to the way a personal assistant would know us and support us, then our tools would be more helpful: an assistive PIM tool would gather together the necessary material in support of our meeting with Yves. We, therefore, have been investigating the possible paths towards PIM tools as tools that work for us, rather than tools that seemingly make us work for them. To that end, in the following sections we consider how we may develop a framework for PIM tools as "perfect digital assistants" (PDA). Our impetus has been to explore how, by considering the affordances of a Real World personal assistant, we can conceptualize a design framework, and from there a development program for a digital simulacrum of such an assistant that is not for some far off future, but for the much nearer term

Towards Data-Driven Autonomics in Data Centers

Continued reliance on human operators for managing data centers is a major impediment for them from ever reaching extreme dimensions. Large computer systems in general, and data centers in particular, will ultimately be managed using predictive computational and executable models obtained through data-science tools, and at that point, the intervention of humans will be limited to setting high-level goals and policies rather than performing low-level operations. Data-driven autonomics, where management and control are based on holistic predictive models that are built and updated using generated data, opens one possible path towards limiting the role of operators in data centers. In this paper, we present a data-science study of a public Google dataset collected in a 12K-node cluster with the goal of building and evaluating a predictive model for node failures. We use BigQuery, the big data SQL platform from the Google Cloud suite, to process massive amounts of data and generate a rich feature set characterizing machine state over time. We describe how an ensemble classifier can be built out of many Random Forest classifiers each trained on these features, to predict if machines will fail in a future 24-hour window. Our evaluation reveals that if we limit false positive rates to 5%, we can achieve true positive rates between 27% and 88% with precision varying between 50% and 72%. We discuss the practicality of including our predictive model as the central component of a data-driven autonomic manager and operating it on-line with live data streams (rather than off-line on data logs). All of the scripts used for BigQuery and classification analyses are publicly available from the authors' website.Comment: 12 pages, 6 figure

Context for Ubiquitous Data Management

In response to the advance of ubiquitous computing technologies, we believe that for computer systems to be ubiquitous, they must be context-aware. In this paper, we address the impact of context-awareness on ubiquitous data management. To do this, we overview different characteristics of context in order to develop a clear understanding of context, as well as its implications and requirements for context-aware data management. References to recent research activities and applicable techniques are also provided