Crowdsourced PAC Learning under Classification Noise

In this paper, we analyze PAC learnability from labels produced by crowdsourcing. In our setting, unlabeled examples are drawn from a distribution and labels are crowdsourced from workers who operate under classification noise, each with their own noise parameter. We develop an end-to-end crowdsourced PAC learning algorithm that takes unlabeled data points as input and outputs a trained classifier. Our three-step algorithm incorporates majority voting, pure-exploration bandits, and noisy-PAC learning. We prove several guarantees on the number of tasks labeled by workers for PAC learning in this setting and show that our algorithm improves upon the baseline by reducing the total number of tasks given to workers. We demonstrate the robustness of our algorithm by exploring its application to additional realistic crowdsourcing settings.Comment: 14 page

An Incentive Compatible Multi-Armed-Bandit Crowdsourcing Mechanism with Quality Assurance

Consider a requester who wishes to crowdsource a series of identical binary labeling tasks to a pool of workers so as to achieve an assured accuracy for each task, in a cost optimal way. The workers are heterogeneous with unknown but fixed qualities and their costs are private. The problem is to select for each task an optimal subset of workers so that the outcome obtained from the selected workers guarantees a target accuracy level. The problem is a challenging one even in a non strategic setting since the accuracy of aggregated label depends on unknown qualities. We develop a novel multi-armed bandit (MAB) mechanism for solving this problem. First, we propose a framework, Assured Accuracy Bandit (AAB), which leads to an MAB algorithm, Constrained Confidence Bound for a Non Strategic setting (CCB-NS). We derive an upper bound on the number of time steps the algorithm chooses a sub-optimal set that depends on the target accuracy level and true qualities. A more challenging situation arises when the requester not only has to learn the qualities of the workers but also elicit their true costs. We modify the CCB-NS algorithm to obtain an adaptive exploration separated algorithm which we call { \em Constrained Confidence Bound for a Strategic setting (CCB-S)}. CCB-S algorithm produces an ex-post monotone allocation rule and thus can be transformed into an ex-post incentive compatible and ex-post individually rational mechanism that learns the qualities of the workers and guarantees a given target accuracy level in a cost optimal way. We provide a lower bound on the number of times any algorithm should select a sub-optimal set and we see that the lower bound matches our upper bound upto a constant factor. We provide insights on the practical implementation of this framework through an illustrative example and we show the efficacy of our algorithms through simulations

Complexity of Judgment Aggregation

We analyse the computational complexity of three problems in judgment aggregation: (1) computing a collective judgment from a profile of individual judgments (the winner determination problem); (2) deciding whether a given agent can influence the outcome of a judgment aggregation procedure in her favour by reporting insincere judgments (the strategic manipulation problem); and (3) deciding whether a given judgment aggregation scenario is guaranteed to result in a logically consistent outcome, independently from what the judgments supplied by the individuals are (the problem of the safety of the agenda). We provide results both for specific aggregation procedures (the quota rules, the premisebased procedure, and a distance-based procedure) and for classes of aggregation procedures characterised in terms of fundamental axioms

Why States Create International Tribunals: A Response to Professors Posner and Yoo

A recent article in the California Law Review by Professors Eric Posner and John Yoo, Judicial Independence in International Tribunals, argues that the only effective international tribunals are dependent tribunals, by which the authors mean ad hoc tribunals staffed by judges closely controlled by governments through the power of reappointment or threats of retaliation. Independent tribunals, by contrast, meaning tribunals staffed by judges appointed on similar terms as those in domestic courts, pose a danger to international cooperation. According to Posner and Yoo, independent tribunals are suspect because they are more likely to allow moral ideals, ideological imperatives or the interests of other states to influence their judgments. In this response, we identify the many shortcomings in the theory, methodology, and empirics in Judicial Independence in International Tribunals. We do so to challenge the authors\u27 core conjecture: that formally dependent international tribunals are correlated with effective judicial outcomes. We then offer our own counter-theory; a theory of constrained independence in which states establish independent international tribunals to enhance the credibility of their commitments and then use more fine grained structural, political, and discursive mechanisms to limit the potential for judicial excesses

Why States Create International Tribunals: A Response to Professors Posner and Yoo

A recent article in the California Law Review by Professors Eric Posner and John Yoo, Judicial Independence in International Tribunals, argues that the only effective international tribunals are dependent tribunals, by which the authors mean ad hoc tribunals staffed by judges closely controlled by governments through the power of reappointment or threats of retaliation. Independent tribunals, by contrast, meaning tribunals staffed by judges appointed on similar terms as those in domestic courts, pose a danger to international cooperation. According to Posner and Yoo, independent tribunals are suspect because they are more likely to allow moral ideals, ideological imperatives or the interests of other states to influence their judgments. In this response, we identify the many shortcomings in the theory, methodology, and empirics in Judicial Independence in International Tribunals. We do so to challenge the authors\u27 core conjecture: that formally dependent international tribunals are correlated with effective judicial outcomes. We then offer our own counter-theory; a theory of constrained independence in which states establish independent international tribunals to enhance the credibility of their commitments and then use more fine grained structural, political, and discursive mechanisms to limit the potential for judicial excesses