Calibrated Prediction Intervals for Neural Network Regressors

Ongoing developments in neural network models are continually advancing the state of the art in terms of system accuracy. However, the predicted labels should not be regarded as the only core output; also important is a well-calibrated estimate of the prediction uncertainty. Such estimates and their calibration are critical in many practical applications. Despite their obvious aforementioned advantage in relation to accuracy, contemporary neural networks can, generally, be regarded as poorly calibrated and as such do not produce reliable output probability estimates. Further, while post-processing calibration solutions can be found in the relevant literature, these tend to be for systems performing classification. In this regard, we herein present two novel methods for acquiring calibrated predictions intervals for neural network regressors: empirical calibration and temperature scaling. In experiments using different regression tasks from the audio and computer vision domains, we find that both our proposed methods are indeed capable of producing calibrated prediction intervals for neural network regressors with any desired confidence level, a finding that is consistent across all datasets and neural network architectures we experimented with. In addition, we derive an additional practical recommendation for producing more accurate calibrated prediction intervals. We release the source code implementing our proposed methods for computing calibrated predicted intervals. The code for computing calibrated predicted intervals is publicly available

Markov and Neural Network Models for Prediction of Structural Deterioration of Stormwater Pipe Assets

Storm-water pipe networks in Australia are designed to convey water from rainfall and surface runoff. They do not transport sewerage. Their structural deterioration is progressive with aging and will eventually cause pipe collapse with consequences of service interruption. Predicting structural condition of pipes provides vital information for asset management to prevent unexpected failures and to extend service life. This study focused on predicting the structural condition of storm-water pipes with two objectives. The first objective is the prediction of structural condition changes of the whole network of storm-water pipes by a Markov model at different times during their service life. This information can be used for planning annual budget and estimating the useful life of pipe assets. The second objective is the prediction of structural condition of any particular pipe by a neural network model. This knowledge is valuable in identifying pipes that are in poor condition for repair actions. A case study with closed circuit television inspection snapshot data was used to demonstrate the applicability of these two models

Machine Prognosis with Full Utilization of Truncated Lifetime Data

Intelligent machine fault prognostics estimates how soon and likely a failure will occur with little human expert judgement. It minimizes production downtime, spares inventory and maintenance labour costs. Prognostic models, especially probabilistic methods, require numerous historical failure instances. In practice however, industrial and military communities would rarely allow their engineering assets to run to failure. It is only known that the machine component survived up to the time of repair or replacement but there is no information as to when the component would have failed if left undisturbed. Data of this sort are called truncated data. This paper proposes a novel model, the Intelligent Product Limit Estimator (iPLE), which utilizes truncated data to perform adaptive long-range prediction of a machine component's remaining lifetime. It takes advantage of statistical models' ability to provide useful representation of survival probabilities, and of neural networks ability to recognise nonlinear relationships between a machine component's future survival condition and a given series of prognostic data features. Progressive bearing degradation data were simulated and used to train and validate the proposed model. The results support our hypothesis that the iPLE can perform better than similar prognostics models that neglect truncated data