Reproducibility Probability Estimation and RP-Testing for Some Nonparametric Tests

Several reproducibility probability (RP)-estimators for the binomial, sign, Wilcoxon signed rank and Kendall tests are studied. Their behavior in terms of MSE is investigated, as well as their performances for RP-testing. Two classes of estimators are considered: the semi-parametric one, where RP-estimators are derived from the expression of the exact or approximated power function, and the non-parametric one, whose RP-estimators are obtained on the basis of the nonparametric plug-in principle. In order to evaluate the precision of RP-estimators for each test, the MSE is computed, and the best overall estimator turns out to belong to the semi-parametric class. Then, in order to evaluate the RP-testing performances provided by RP estimators for each test, the disagreement between the RP-testing decision rule, i.e., "accept H0 if the RP-estimate is lower than, or equal to, 1/2, and reject H0 otherwise", and the classical one (based on the critical value or on the p-value) is obtained. It is shown that the RP-based testing decision for some semi-parametric RP estimators exactly replicates the classical one. In many situations, the RP-estimator replicating the classical decision rule also provides the best MSE

Bootstrap Methods and Reproducibility Probability

Bootstrap method is one of the resampling methods, it is a powerful and computer-based method. We will use two different bootstrap methods, Efron’s bootstrap and smoothed Efron’s bootstrap. They are resampling methods but in different manner and will be explained later in this paper. The reproducibility probability is the important topic will be discussed here and will be used to compare the two bootstrap methods. It is reflecting the stability of the results of hypothesis tests, it is the probability of obtaining the same decision when repeated hypothesis test