Learning from the Crowd with Pairwise Comparison

Efficient learning of halfspaces is arguably one of the most important problems in machine learning and statistics. With the unprecedented growth of large-scale data sets, it has become ubiquitous to appeal to crowd for data annotation, and the central problem that attracts a surge of recent interests is how one can provably learn the underlying halfspace from the highly noisy crowd feedback. On the other hand, a large body of recent works have been dedicated to the problem of learning with not only labels, but also pairwise comparisons, since in many cases it is easier to compare than to label. In this paper we study the problem of learning halfspaces from the crowd under the realizable PAC learning setting, and we assume that the crowd workers can provide (noisy) labels or pairwise comparison tags upon request. We show that with a powerful boosting framework, together with our novel design of a filtering process, the overhead (to be defined) of the crowd acts as a constant, whereas the natural extension of standard approaches to crowd setting leads to an overhead growing with the size of the data sets

Extending the multi-arm multi-stage trial design

The multi-arm multi-stage (MAMS) adaptive clinical trial design has been successfully implemented in several randomised phase III trials. Intermediate short-term outcomes identify and stop recruitment to research arms demonstrating insu cient bene t compared to the control arm at interim stages, before the nal analysis on the primary outcome. The design has been shown to reduce the time and resources required to identify an e ective treatment compared to traditional two-arm designs. This PhD extends the applications of the MAMS design to a broader range of research questions, with the aim of increasing uptake of the design. Stopping recruitment early has been introduced to arms demonstrating overwhelming e cacy on the primary outcome, whilst also stopping for lack-of-bene t on the intermediate outcome for the time-to-event setting. The methods could reduce the patients and resources required should any e cacious arm be identi ed early. Guidelines have been developed on how to design a trial of this nature, and it is shown how to modify the design to control the familywise error rate and power at a pre-speci ed level. It may be necessary to restrict the number of arms in each stage of a MAMS design due to budget constraints or limitations on the number of patients available. This thesis explores how pre-speci ed treatment selection could be implemented, where a subset of arms is chosen at each interim analysis, reducing the maximum sample size. Since selection can potentially lead to bias in treatment e ect estimates, this research also addresses estimation concerns in the proposed design by quantifying the extent of potential bias. Programs for designing MAMS trials have been updated in Stata to accommodate the new methods, to encourage easy adoption of the designs. Finally, practical recommendations have been developed for implementing the proposed ideas, and demonstrates the applications of each of the methods using real trials

Semi-verified PAC Learning from the Crowd with Pairwise Comparisons

We study the problem of crowdsourced PAC learning of threshold functions with pairwise comparisons. This is a challenging problem and only recently have query-efficient algorithms been established in the scenario where the majority of the crowd are perfect. In this work, we investigate the significantly more challenging case that the majority are incorrect, which in general renders learning impossible. We show that under the semi-verified model of Charikar~et~al.~(2017), where we have (limited) access to a trusted oracle who always returns the correct annotation, it is possible to PAC learn the underlying hypothesis class while drastically mitigating the labeling cost via the more easily obtained comparison queries. Orthogonal to recent developments in semi-verified or list-decodable learning that crucially rely on data distributional assumptions, our PAC guarantee holds by exploring the wisdom of the crowd.Comment: v2 incorporates a simpler Filter algorithm, thus the technical assumption (in v1) is no longer needed. v2 also reorganizes and emphasizes new algorithm component

Timely and reliable evaluation of the effects of interventions: a framework for adaptive meta-analysis (FAME)

Most systematic reviews are retrospective and use aggregate data AD) from publications, meaning they can be unreliable, lag behind therapeutic developments and fail to influence ongoing or new trials. Commonly, the potential influence of unpublished or ongoing trials is overlooked when interpreting results, or determining the value of updating the meta-analysis or need to collect individual participant data (IPD). Therefore, we developed a Framework for Adaptive Metaanalysis (FAME) to determine prospectively the earliest opportunity for reliable AD meta-analysis. We illustrate FAME using two systematic reviews in men with metastatic (M1) and non-metastatic (M0)hormone-sensitive prostate cancer (HSPC)