Continuous Monitoring of A/B Tests without Pain: Optional Stopping in Bayesian Testing

A/B testing is one of the most successful applications of statistical theory in modern Internet age. One problem of Null Hypothesis Statistical Testing (NHST), the backbone of A/B testing methodology, is that experimenters are not allowed to continuously monitor the result and make decision in real time. Many people see this restriction as a setback against the trend in the technology toward real time data analytics. Recently, Bayesian Hypothesis Testing, which intuitively is more suitable for real time decision making, attracted growing interest as an alternative to NHST. While corrections of NHST for the continuous monitoring setting are well established in the existing literature and known in A/B testing community, the debate over the issue of whether continuous monitoring is a proper practice in Bayesian testing exists among both academic researchers and general practitioners. In this paper, we formally prove the validity of Bayesian testing with continuous monitoring when proper stopping rules are used, and illustrate the theoretical results with concrete simulation illustrations. We point out common bad practices where stopping rules are not proper and also compare our methodology to NHST corrections. General guidelines for researchers and practitioners are also provided

Novel Application Of Untargeted Metabolomics To Diseases Of Neurosurgical Significance

Metabolomics, an emerging technique to study hundreds of small-molecule metabolites simultaneously, has been seldom applied to diseases of neurosurgical significance. We utilized metabolomics to explore two distinct questions: 1. to identify global metabolic changes and metabolite predictors of long-term outcome in aneurysmal subarachnoid hemorrhage (SAH) patients, 2. to identify differential metabolites profiles of radiation necrosis vs. recurrent tumor of metastatic brain lesions post-Gamma Knife radiosurgery. The first study applied gas chromatography time-of-flight mass spectrometry (GC-TOF) to cerebrospinal fluid samples collected from 15 high-grade aSAH patients (modified Fisher grades 3 and 4). Analysis was performed at two time points; metabolite levels at each time point were correlated with Glasgow Outcome Scale (GOS) of patients at 1 year post-aSAH. Of 97 metabolites identified, 16 metabolites (primarily free amino acids) significantly changed between the two time points; these changes were magnified in modified Fisher grade 4 compared with grade 3. Six metabolites (2-hydroxyglutarate, tryptophan, glycine, proline, isoleucine, and alanine) correlated with GOS at 1 year post-aSAH. These results suggest that specific metabolite changes occur in the brain during the course of aSAH and that quantification of specific CSF metabolites may be used to predict long-term outcomes. This is the first study to implicate 2- hydroxyglutarate, a known marker of tissue hypoxia, in aSAH pathogenesis. The second study applied GC- TOF to histologically-validated specimens (7 each) of pure radiation necrosis and pure recurrent tumor obtained from patient brain biopsies. Of 141 metabolites identified, 17 were found to be statistically significantly different between comparison groups. Of these metabolites, 6 were increased in tumor, and 11 metabolites were increased in radiation necrosis. An unsupervised hierarchical clustering analysis found that tumor had elevated levels of metabolites associated with energy metabolism whereas radiation necrosis had elevated levels of metabolites that were fatty acids and antioxidants/cofactors. This is the first tissue- based metabolomics study of radiation necrosis and tumor. Radiation necrosis and recurrent tumor following Gamma Knife radiosurgery for brain metastases have unique metabolite profiles that may be targeted in the future to develop non-invasive metabolic imaging techniques

Predictability of Stock Returns based on the Partial Least Squares Methodology

Empirical evidence on the predictability of aggregate stock returns has shown that many commonly used predictor variables have little power to predict the market out-of-sample. However, a recent paper by Kelly and Pruitt (2013) find that predictors with strong out-of-sample performance can be constructed, using a partial least squares methodology, from the valuation ratios of portfolios. This paper shows that the statistical significance of this out-of-sample predictability is overstated for two reasons. Firstly, the analysis is conducted on gross returns rather than excess returns, and this raises the apparent predictability of the equity premium due to inclusion predictable movements of interest rates. Secondly, the bootstrap statistics used to assess out-of-sample significance do not account for small-sample bias in the estimated coefficients. This bias is well known to affect tests of in-sample significance (Stambaugh (1986)) and I show it is also important for out-of-sample tests of significance. Accounting for both these effects can radically change the conclusions; for example, the recursive out-of-sample R2 values for the sample period 1965-2010 are insignificant for the prediction of one-year excess returns, and one-month returns, except in the case of the book-to-market ratios of six size-and value-sorted portfolios, which is significant at the 10% level