Trustworthy Experimentation Under Telemetry Loss

Failure to accurately measure the outcomes of an experiment can lead to bias and incorrect conclusions. Online controlled experiments (aka AB tests) are increasingly being used to make decisions to improve websites as well as mobile and desktop applications. We argue that loss of telemetry data (during upload or post-processing) can skew the results of experiments, leading to loss of statistical power and inaccurate or erroneous conclusions. By systematically investigating the causes of telemetry loss, we argue that it is not practical to entirely eliminate it. Consequently, experimentation systems need to be robust to its effects. Furthermore, we note that it is nontrivial to measure the absolute level of telemetry loss in an experimentation system. In this paper, we take a top-down approach towards solving this problem. We motivate the impact of loss qualitatively using experiments in real applications deployed at scale, and formalize the problem by presenting a theoretical breakdown of the bias introduced by loss. Based on this foundation, we present a general framework for quantitatively evaluating the impact of telemetry loss, and present two solutions to measure the absolute levels of loss. This framework is used by well-known applications at Microsoft, with millions of users and billions of sessions. These general principles can be adopted by any application to improve the overall trustworthiness of experimentation and data-driven decision making.Comment: Proceedings of the 27th ACM International Conference on Information and Knowledge Management, October 201

Physics at a Fermilab Proton Driver

This report documents the physics case for building a 2 MW, 8 GeV superconducting linac proton driver at Fermilab.Comment: 52 pages, 15 figure

Preprosti poskusi v interaktivnem fizikalnem laboratoriju: dijakova notranja motivacija in razumevanje

Experiments in different forms can certainly be suitable tools for increasing student interest in physics. However, educators continuously discuss which forms of experimenting (if any) are the most beneficial for these purposes. At the Faculty of Mathematics and Physics, Charles University, Prague, two different forms of physics experiments are offered to upper secondary students: hands-on experimental work in the Interactive Physics Laboratory, and physics demonstration shows where the students watch experiments conducted by a lecturer. Our research focuses primarily on student feedback about their immediate attitudes towards these two projects. Data collection was undertaken using questionnaire research based on the Intrinsic Motivation Inventory. This research was subsequently supplemented with a qualitative study examining the influence of students’ experimental work in the Interactive Physics Laboratory on their understanding of selected physics concepts. The results of the main research show that the two projects do not exhibit significant differences in terms of student interest and perceived usefulness; nevertheless, students felt the need for significantly more effort and experienced pressure during their work in the Interactive Physics Laboratory. One interesting finding, which goes against our original hypothesis, is that grades in physics are quite a strong predictor of students’ assessment of the projects: better grades indicate more positive assessment of both projects as well as less pressure felt during hands-on activities in the laboratory. (DIPF/Orig.

Distributed Deep Learning for Question Answering

This paper is an empirical study of the distributed deep learning for question answering subtasks: answer selection and question classification. Comparison studies of SGD, MSGD, ADADELTA, ADAGRAD, ADAM/ADAMAX, RMSPROP, DOWNPOUR and EASGD/EAMSGD algorithms have been presented. Experimental results show that the distributed framework based on the message passing interface can accelerate the convergence speed at a sublinear scale. This paper demonstrates the importance of distributed training. For example, with 48 workers, a 24x speedup is achievable for the answer selection task and running time is decreased from 138.2 hours to 5.81 hours, which will increase the productivity significantly.Comment: This paper will appear in the Proceeding of The 25th ACM International Conference on Information and Knowledge Management (CIKM 2016), Indianapolis, US