Error curves for evaluating the quality of feature rankings

peer reviewedIn this article, we propose a method for evaluating feature ranking algorithms. A feature ranking algorithm estimates the importance of descriptive features when predicting the target variable, and the proposed method evaluates the correctness of these importance values by computing the error measures of two chains of predictive models. The models in the first chain are built on nested sets of top-ranked features, while the models in the other chain are built on nested sets of bottom ranked features. We investigate which predictive models are appropriate for building these chains, showing empirically that the proposed method gives meaningful results and can detect differences in feature ranking quality. This is first demonstrated on synthetic data, and then on several real-world classification benchmark problems

Global detection and analysis of coastline associated rainfall using an objective pattern recognition technique

Coastally associated rainfall is a common feature especially in tropical and subtropical regions. However, it has been difficult to quantify the contribution of coastal rainfall features to the overall local rainfall. We develop a novel technique to objectively identify precipitation associated with land-sea interaction and apply it to satellite based rainfall estimates. The Maritime Continent, the Bight of Panama, Madagascar and the Mediterranean are found to be regions where land-sea interactions plays a crucial role in the formation of precipitation. In these regions $\approx$ 40% to 60% of the total rainfall can be related to coastline effects. Due to its importance for the climate system, the Maritime Continent is a particular region of interest with high overall amounts of rainfall and large fractions resulting from land-sea interactions throughout the year. To demonstrate the utility of our identification method we investigate the influence of several modes of variability, such as the Madden-Julian-Oscillation and the El Ni\~no Southern Oscillation, on coastal rainfall behavior. The results suggest that during large scale suppressed convective conditions coastal effects tend modulate the rainfall over the Maritime Continent leading to enhanced rainfall over land regions compared to the surrounding oceans. We propose that the novel objective dataset of coastally influenced precipitation can be used in a variety of ways, such as to inform cumulus parametrization or as an additional tool for evaluating the simulation of coastal precipitation within weather and climate models

Uncertainty Propagation and Feature Selection for Loss Estimation in Performance-based Earthquake Engineering

This report presents a new methodology, called moment matching, of propagating the uncertainties in estimating repair costs of a building due to future earthquake excitation, which is required, for example, when assessing a design in performance-based earthquake engineering. Besides excitation uncertainties, other uncertain model variables are considered, including uncertainties in the structural model parameters and in the capacity and repair costs of structural and non-structural components. Using the first few moments of these uncertain variables, moment matching requires only a few well-chosen point estimates to propagate the uncertainties to estimate the first few moments of the repair costs with high accuracy. Furthermore, the use of moment matching to estimate the exceedance probability of the repair costs is also addressed. These examples illustrate that the moment-matching approach is quite general; for example, it can be applied to any decision variable in performance-based earthquake engineering. Two buildings are chosen as illustrative examples to demonstrate the use of moment matching, a hypothetical three-story shear building and a real seven-story hotel building. For these two examples, the assembly-based vulnerability approach is employed when calculating repair costs. It is shown that the moment-matching technique is much more accurate than the well-known First-Order-Second-Moment approach when propagating the first two moments, while the resulting computational cost is of the same order. The repair-cost moments and exceedance probability estimated by the moment-matching technique are also compared with those by Monte Carlo simulation. It is concluded that as long as the order of the moment matching is sufficient, the comparison is satisfactory. Furthermore, the amount of computation for moment matching scales only linearly with the number of uncertain input variables. Last but not least, a procedure for feature selection is presented and illustrated for the second example. The conclusion is that the most important uncertain input variables among the many influencing the uncertainty in future repair costs are, in order of importance, ground-motion spectral acceleration, component capacity, ground-motion details and unit repair costs

Lost in Translation: Piloting a Novel Framework to Assess the Challenges in Translating Scientific Uncertainty From Empirical Findings to WHO Policy Statements.

BACKGROUND:Calls for evidence-informed public health policy, with implicit promises of greater program effectiveness, have intensified recently. The methods to produce such policies are not self-evident, requiring a conciliation of values and norms between policy-makers and evidence producers. In particular, the translation of uncertainty from empirical research findings, particularly issues of statistical variability and generalizability, is a persistent challenge because of the incremental nature of research and the iterative cycle of advancing knowledge and implementation. This paper aims to assess how the concept of uncertainty is considered and acknowledged in World Health Organization (WHO) policy recommendations and guidelines. METHODS:We selected four WHO policy statements published between 2008-2013 regarding maternal and child nutrient supplementation, infant feeding, heat action plans, and malaria control to represent topics with a spectrum of available evidence bases. Each of these four statements was analyzed using a novel framework to assess the treatment of statistical variability and generalizability. RESULTS:WHO currently provides substantial guidance on addressing statistical variability through GRADE (Grading of Recommendations Assessment, Development, and Evaluation) ratings for precision and consistency in their guideline documents. Accordingly, our analysis showed that policy-informing questions were addressed by systematic reviews and representations of statistical variability (eg, with numeric confidence intervals). In contrast, the presentation of contextual or "background" evidence regarding etiology or disease burden showed little consideration for this variability. Moreover, generalizability or "indirectness" was uniformly neglected, with little explicit consideration of study settings or subgroups. CONCLUSION:In this paper, we found that non-uniform treatment of statistical variability and generalizability factors that may contribute to uncertainty regarding recommendations were neglected, including the state of evidence informing background questions (prevalence, mechanisms, or burden or distributions of health problems) and little assessment of generalizability, alternate interventions, and additional outcomes not captured by systematic review. These other factors often form a basis for providing policy recommendations, particularly in the absence of a strong evidence base for intervention effects. Consequently, they should also be subject to stringent and systematic evaluation criteria. We suggest that more effort is needed to systematically acknowledge (1) when evidence is missing, conflicting, or equivocal, (2) what normative considerations were also employed, and (3) how additional evidence may be accrued

Regional integration and trade: controlling for varying degrees of heterogeneity in the gravity model

Using a panel dataset of bilateral export flows from 12 EU countries to 20 OECD trading partners over the 1992–2003 period, this paper examines whether the effect on trade of European regional integration, denoted by an EU dummy, holds across a representative number of specifications for two gravity models, one based on the traditional trade determinants, the other based on newer trade theories (NTT). For both gravity model specifications the coefficient of the EU dummy declines in magnitude and becomes insignificant as an increasing degree of country heterogeneity is admitted into the model. This suggests the fundamental importance of the econometric specification when evaluating trade policy effects