13,244 research outputs found
Towards Operator-less Data Centers Through Data-Driven, Predictive, Proactive Autonomics
Continued reliance on human operators for managing data centers is a major
impediment for them from ever reaching extreme dimensions. Large computer
systems in general, and data centers in particular, will ultimately be managed
using predictive computational and executable models obtained through
data-science tools, and at that point, the intervention of humans will be
limited to setting high-level goals and policies rather than performing
low-level operations. Data-driven autonomics, where management and control are
based on holistic predictive models that are built and updated using live data,
opens one possible path towards limiting the role of operators in data centers.
In this paper, we present a data-science study of a public Google dataset
collected in a 12K-node cluster with the goal of building and evaluating
predictive models for node failures. Our results support the practicality of a
data-driven approach by showing the effectiveness of predictive models based on
data found in typical data center logs. We use BigQuery, the big data SQL
platform from the Google Cloud suite, to process massive amounts of data and
generate a rich feature set characterizing node state over time. We describe
how an ensemble classifier can be built out of many Random Forest classifiers
each trained on these features, to predict if nodes will fail in a future
24-hour window. Our evaluation reveals that if we limit false positive rates to
5%, we can achieve true positive rates between 27% and 88% with precision
varying between 50% and 72%.This level of performance allows us to recover
large fraction of jobs' executions (by redirecting them to other nodes when a
failure of the present node is predicted) that would otherwise have been wasted
due to failures. [...
The predictability of precipitation episodes during the West African dry season
Precipitation episodes in tropical West Africa (7-15°N, 10°W-10°E) during the dry season from November to March are rare, but can have significant impacts on human activities reaching from greening of pastures to spoiling harvests and health implications. Previous work has shown a link between these unseasonal rainfalls and extratropical disturbances via a decrease of surface pressure over the Sahara/Sahel and a subsequent inflow of moist air from the Gulf of Guinea. This paper supports the previously stated hypothesis that the extratropical influence leads to a high rainfall predictability through a careful analysis of operational 5 day forecasts from the European Centre for Medium-Range Weather Forecasts' (ECMWF) ensemble prediction system (EPS), which are evaluated using Global Precipitation Climatology Project (GPCP) and Tropical Rainfall Measuring Mission (TRMM) precipitation estimates for the 11 dry seasons 1998/99-2008/09. The long-term regional average of ensemble-mean precipitation lies between the two observational datasets, with GPCP being considerably wetter. Temporal correlations between the ensemble mean and observations are 0.8. Standard probabilistic evaluation methods such as reliability and relative operating characteristic (ROC) diagrams indicate remarkably good reliability, sharpness and skill across a range of precipitation thresholds. However, a categorical verification focusing on the most extreme ensemble mean values indicates too many false alarms. Despite the considerable observational uncertainty the results show that the ECMWF EPS is capable of predicting winter rainfall events in tropical West Africa with good accuracy, at least on regional spatial and synoptic time-scales, which should encourage West African weather services to capitalize more on the valuable information provided by ensemble prediction systems during the dry season
Crime Distribution & Victim Behavior During a Crime Wave
The study of how crime affects different income groups faces several difficulties. The first is that crime-avoiding activities vary across income groups. Thus, a lower victimization rate in one group may not reflect a lower burden of crime, but rather a higher investment in avoiding crime. A second difficulty is that, typically, only a small fraction of the population is victimized so that empirical tests often lack the statistical power to detect differences across groups. We take advantage of a dramatic increase in crime rates in Argentina during the late 1990s to document several interesting patterns. First, the increase in victimization experienced by the poor is larger than the increase endured by the rich. The difference appears large: low-income people have experienced increases in victimization rates that are almost 50 percent higher than those suffered by high-income people. Second, for home robberies, where the rich can protect themselves (by hiring private security, for example), we find significantly larger increases in victimization rates amongst the poor. In contrast, for robberies on the street, where the rich can only mimic the poor, we find similar increases in victimization for both income groups. Third, we document direct evidence on pecuniary and non-pecuniary protection activities by both the rich and poor, ranging from the avoidance of dark places to the hiring of private security. Fourth, we show the correlations between changes in protection and mimicking and changes in crime victimization. Fifth, we offer one possible way of using these estimates to explain the incidence of crime across income groups.http://deepblue.lib.umich.edu/bitstream/2027.42/57229/1/wp849 .pd
Towards Data-Driven Autonomics in Data Centers
Continued reliance on human operators for managing data centers is a major
impediment for them from ever reaching extreme dimensions. Large computer
systems in general, and data centers in particular, will ultimately be managed
using predictive computational and executable models obtained through
data-science tools, and at that point, the intervention of humans will be
limited to setting high-level goals and policies rather than performing
low-level operations. Data-driven autonomics, where management and control are
based on holistic predictive models that are built and updated using generated
data, opens one possible path towards limiting the role of operators in data
centers. In this paper, we present a data-science study of a public Google
dataset collected in a 12K-node cluster with the goal of building and
evaluating a predictive model for node failures. We use BigQuery, the big data
SQL platform from the Google Cloud suite, to process massive amounts of data
and generate a rich feature set characterizing machine state over time. We
describe how an ensemble classifier can be built out of many Random Forest
classifiers each trained on these features, to predict if machines will fail in
a future 24-hour window. Our evaluation reveals that if we limit false positive
rates to 5%, we can achieve true positive rates between 27% and 88% with
precision varying between 50% and 72%. We discuss the practicality of including
our predictive model as the central component of a data-driven autonomic
manager and operating it on-line with live data streams (rather than off-line
on data logs). All of the scripts used for BigQuery and classification analyses
are publicly available from the authors' website.Comment: 12 pages, 6 figure
Planetary Candidates Observed by Kepler VI: Planet Sample from Q1-Q16 (47 Months)
\We present the sixth catalog of Kepler candidate planets based on nearly 4
years of high precision photometry. This catalog builds on the legacy of
previous catalogs released by the Kepler project and includes 1493 new Kepler
Objects of Interest (KOIs) of which 554 are planet candidates, and 131 of these
candidates have best fit radii <1.5 R_earth. This brings the total number of
KOIs and planet candidates to 7305 and 4173 respectively. We suspect that many
of these new candidates at the low signal-to-noise limit may be false alarms
created by instrumental noise, and discuss our efforts to identify such
objects. We re-evaluate all previously published KOIs with orbital periods of
>50 days to provide a consistently vetted sample that can be used to improve
planet occurrence rate calculations. We discuss the performance of our planet
detection algorithms, and the consistency of our vetting products. The full
catalog is publicly available at the NASA Exoplanet Archive.Comment: 18 pages, to be published in the Astrophysical Journal Supplement
Serie
- …