Unachievable Region in Precision-Recall Space and Its Effect on Empirical Evaluation

Precision-recall (PR) curves and the areas under them are widely used to summarize machine learning results, especially for data sets exhibiting class skew. They are often used analogously to ROC curves and the area under ROC curves. It is known that PR curves vary as class skew changes. What was not recognized before this paper is that there is a region of PR space that is completely unachievable, and the size of this region depends only on the skew. This paper precisely characterizes the size of that region and discusses its implications for empirical evaluation methodology in machine learning.Comment: ICML2012, fixed citations to use correct tech report numbe

Precision-Recall-Gain Curves:PR Analysis Done Right

Precision-Recall analysis abounds in applications of binary classification where true negatives do not add value and hence should not affect assessment of the classifier’s performance. Perhaps inspired by the many advantages of receiver op-erating characteristic (ROC) curves and the area under such curves for accuracy-based performance assessment, many researchers have taken to report Precision-Recall (PR) curves and associated areas as performance metric. We demonstrate in this paper that this practice is fraught with difficulties, mainly because of in-coherent scale assumptions – e.g., the area under a PR curve takes the arithmetic mean of precision values whereas the Fβ score applies the harmonic mean. We show how to fix this by plotting PR curves in a different coordinate system, and demonstrate that the new Precision-Recall-Gain curves inherit all key advantages of ROC curves. In particular, the area under Precision-Recall-Gain curves con-veys an expected F1 score on a harmonic scale, and the convex hull of a Precision-Recall-Gain curve allows us to calibrate the classifier’s scores so as to determine, for each operating point on the convex hull, the interval of β values for which the point optimises Fβ. We demonstrate experimentally that the area under traditional PR curves can easily favour models with lower expected F1 score than others, and so the use of Precision-Recall-Gain curves will result in better model selection.

Precision-Recall-Gain Curves: PR Analysis Done Right

Abstract Precision-Recall analysis abounds in applications of binary classification where true negatives do not add value and hence should not affect assessment of the classifier's performance. Perhaps inspired by the many advantages of receiver operating characteristic (ROC) curves and the area under such curves for accuracybased performance assessment, many researchers have taken to report PrecisionRecall (PR) curves and associated areas as performance metric. We demonstrate in this paper that this practice is fraught with difficulties, mainly because of incoherent scale assumptions -e.g., the area under a PR curve takes the arithmetic mean of precision values whereas the F β score applies the harmonic mean. We show how to fix this by plotting PR curves in a different coordinate system, and demonstrate that the new Precision-Recall-Gain curves inherit all key advantages of ROC curves. In particular, the area under Precision-Recall-Gain curves conveys an expected F 1 score on a harmonic scale, and the convex hull of a PrecisionRecall-Gain curve allows us to calibrate the classifier's scores so as to determine, for each operating point on the convex hull, the interval of β values for which the point optimises F β . We demonstrate experimentally that the area under traditional PR curves can easily favour models with lower expected F 1 score than others, and so the use of Precision-Recall-Gain curves will result in better model selection

The Dynamics of Mixing and Subtidal Flow in a Maine Estuary

Long-term material transport in estuaries is largely controlled by subtidal flow. Subtidal flows are driven by conditions specific to their geographic location, namely river discharge, tidal forcing, and wind. These conditions are further modified by mixing, curvature of the estuary, Coriolis, and friction. While many practices of measuring these various forcing mechanisms exist, the current literature fails to provide a standard for measuring turbulent mixing in well- mixed estuaries. As a changing climate affects these various forcing and modifying mechanisms via sea level rise and increased precipitation, the corresponding material transport scheme is expected to change accordingly. Subtidal flow and mixing dynamics in the Damariscotta River, a critical hub in Maine’s oyster-aquaculture industry, are investigated to explore how a changing climate may affect local dynamics. Multiple field surveys are performed to adequately characterize all three ‘reaches’ of the Damariscotta River, each characterized by unique bathymetric features, during varying river discharge and tidal conditions. In September 2016 and March - July 2017, a total of eight field surveys were performed during sequential spring and neap tides to cover both wet and dry seasons and a full range of tidal conditions. An acoustic Doppler current profiler measured current velocities and a shear probe microstructure profiler provided turbulent kinetic energy dissipation rates, density, and turbidity measurements at four locations across estuary. Results show that subtidal flow structure changes significantly between reaches, exhibiting a vertically sheared pattern in the lower reach and a mix of vertically- and laterally-sheared patterns in the mid- and upper reaches. These patterns are further investigated through an analysis of the subtidal momentum balance, which allows for the inspection of each forcing mechanism’s individual contribution to the observed dynamics. Lateral and longitudinal advection and frictional effects were found to dominate in the estuary, all of which increased in magnitude up estuary. Based on the momentum balance results, predictions for the dynamic response to sea level rise and increased precipitation can be made. Mixing conditions are also found to vary considerably between reaches with largest mean turbulent kinetic energy dissipation rates observed in the upper reach. These patterns exhibit increased tidal asymmetry up-estuary, indicating the possibility of significant intermittency. Intermittency in turbulence has recently received significantly more attention in the past decade as oceanic and atmospheric researchers become aware of the problems it poses on accurately measuring turbulence. A sensitivity analysis to dataset-size is performed link various scales of intermittency to tidal and hydrographic characteristics and identify how many profiles of turbulent kinetic energy are necessary to precisely represent turbulent mixing in well-mixed estuaries. Internal intermittency is found linked to regions of complex geometry, transitions phases of the tide, and regions of strong lateral and longitudinal straining of velocity shears. Appropriate recommendations of sampling technique are made for use in other like estuaries

A General Framework for Updating Belief Distributions

We propose a framework for general Bayesian inference. We argue that a valid update of a prior belief distribution to a posterior can be made for parameters which are connected to observations through a loss function rather than the traditional likelihood function, which is recovered under the special case of using self information loss. Modern application areas make it is increasingly challenging for Bayesians to attempt to model the true data generating mechanism. Moreover, when the object of interest is low dimensional, such as a mean or median, it is cumbersome to have to achieve this via a complete model for the whole data distribution. More importantly, there are settings where the parameter of interest does not directly index a family of density functions and thus the Bayesian approach to learning about such parameters is currently regarded as problematic. Our proposed framework uses loss-functions to connect information in the data to functionals of interest. The updating of beliefs then follows from a decision theoretic approach involving cumulative loss functions. Importantly, the procedure coincides with Bayesian updating when a true likelihood is known, yet provides coherent subjective inference in much more general settings. Connections to other inference frameworks are highlighted.Comment: This is the pre-peer reviewed version of the article "A General Framework for Updating Belief Distributions", which has been accepted for publication in the Journal of Statistical Society - Series B. This article may be used for non-commercial purposes in accordance with Wiley Terms and Conditions for Self-Archivin