Sources of Irreproducibility in Machine Learning: A Review

Lately, several benchmark studies have shown that the state of the art in some of the sub-fields of machine learning actually has not progressed despite progress being reported in the literature. The lack of progress is partly caused by the irreproducibility of many model comparison studies. Model comparison studies are conducted that do not control for many known sources of irreproducibility. This leads to results that cannot be verified by third parties. Our objective is to provide an overview of the sources of irreproducibility that are reported in the literature. We review the literature to provide an overview and a taxonomy in addition to a discussion on the identified sources of irreproducibility. Finally, we identify three lines of further inquiry

Spatial methods for event reconstruction in CLEAN

In CLEAN (Cryogenic Low Energy Astrophysics with Noble gases), a proposed neutrino and dark matter detector, background discrimination is possible if one can determine the location of an ionizing radiation event with high accuracy. We simulate ionizing radiation events that produce multiple scintillation photons within a spherical detection volume filled with liquid neon. We estimate the radial location of a particular ionizing radiation event based on the observed count data corresponding to that event. The count data are collected by detectors mounted at the spherical boundary of the detection volume. We neglect absorption, but account for Rayleigh scattering. To account for wavelength-shifting of the scintillation light, we assume that photons are absorbed and re-emitted at the detectors. Here, we develop spatial Maximum Likelihood methods for event reconstruction, and study their performance in computer simulation experiments. We also study a method based on the centroid of the observed count data. We calibrate our estimates based on training data

Optimal Proton Trapping in a Neutron Lifetime Experiment

In a neutron lifetime experiment conducted at the National Institute of Standards and Technology, protons produced by neutron decay events are confined in a Penning trap. In each run of the experiment, there is a trapping stage of duration $\tau$. After the trapping stage, protons are purged from the trap. A proton detector provides incomplete information because it goes dead after detecting the first of any purged protons. Further, there is a dead time $\delta$ between the end of the trapping stage in one run and the beginning of the next trapping stage in the next run. Based on the fraction of runs where a proton is detected, I estimate the trapping rate $\lambda$ by the method of maximum likelihood. I show that the expected value of the maximum likelihood estimate is infinite. To obtain a maximum likelihood estimate with a finite expected value and a well-defined and finite variance, I restrict attention to a subsample of all realizations of the data. This subsample excludes an exceedingly rare realization that yields an infinite-valued estimate of $\lambda$. I present asymptotically valid formulas for the bias, root-mean-square prediction error, and standard deviation of the maximum likelihood estimate of $\lambda$ for this subsample. Based on nominal values of $\lambda$ and the dead time $\delta$, I determine the optimal duration of the trapping stage $\tau$ by minimizing the root-mean-square prediction error of the estimate.Comment: 21 pages, 4 figures This is a revised version of "Optimal Proton Trapping". Based on a review, some aspects of the techical argument were refine

Dysregulated RasGRP1 Responds to Cytokine Receptor Input in T Cell Leukemogenesis

Enhanced signaling by the small guanosine triphosphatase Ras is common in T cell acute lymphoblastic leukemia/lymphoma (T-ALL), but the underlying mechanisms are unclear. We identified the guanine nucleotide exchange factor RasGRP1 (Rasgrp1 in mice) as a Ras activator that contributes to leukemogenesis. We found increased RasGRP1 expression in many pediatric T-ALL patients, which is not observed in rare early T cell precursor T-ALL patients with KRAS and NRAS mutations, such as K-Ras[superscript G12D]. Leukemia screens in wild-type mice, but not in mice expressing the mutant K-Ras[superscript G12D] that encodes a constitutively active Ras, yielded frequent retroviral insertions that led to increased Rasgrp1 expression. Rasgrp1 and oncogenic K-Ras[superscript G12D] promoted T-ALL through distinct mechanisms. In K-Ras[superscript G12D] T-ALLs, enhanced Ras activation had to be uncoupled from cell cycle arrest to promote cell proliferation. In mouse T-ALL cells with increased Rasgrp1 expression, we found that Rasgrp1 contributed to a previously uncharacterized cytokine receptor–activated Ras pathway that stimulated the proliferation of T-ALL cells in vivo, which was accompanied by dynamic patterns of activation of effector kinases downstream of Ras in individual T-ALLs. Reduction of Rasgrp1 abundance reduced cytokine-stimulated Ras signaling and decreased the proliferation of T-ALL in vivo. The position of RasGRP1 downstream of cytokine receptors as well as the different clinical outcomes that we observed as a function of RasGRP1 abundance make RasGRP1 an attractive future stratification marker for T-ALL.National Institutes of Health (U.S.). Pioneer AwardNational Cancer Institute (U.S.). Physical Sciences-Oncology Center (U54CA143874)National Institutes of Health (U.S.). (P01 AI091580

Does modifying competition affect the frequency of technical skills in junior rugby league?

The technical demands of games can be affected by changing the number of players, pitch size and rules. This controlled trial compared the frequency of technical skills between a 'traditional' and newly introduced systematically 'modified' game of primary rugby league. A total of 475 primary rugby league players (Under 7s - 9s) were filmed playing traditional (n=49) and modified (n= 249) formats. Notational analysis examined the frequency of technical skills (e.g., number of passes) within 'traditional' and 'modified' games. At each age category, multivariate analysis of variance indicated the clear superiority of the 'modified' game for the frequency of technical skills (e.g., Under 7s total skill opportunities - 'traditional' = 342.9±47.0; 'modified' = 449.4±93.3, d=1.44, p<0.001). Systematically modifying the competitive game is an effective way to increase skill opportunities for children within rugby league. Future research should examine the outcomes of modifying games in optimizing skill development in youth sport