Permutation-Invariant Consensus over Crowdsourced Labels

This Major Qualifying Project introduces a novel crowdsourcing consensus model and inference algorithm which we call PICA (Permutation-Invariant Crowdsourcing Aggregation) that is designed to recover the ground-truth labels of a dataset while being invariant to the class permutations enacted by the different annotators. The PICA model is constructed by endowing each annotator with a doubly-stochastic matrix (DSM), which models the probabilities that an annotator will perceive one class and transcribe it into another. We conduct simulations and experiments to show the advantage of PICA over similar models for three different clustering/labeling tasks, including aggregating dense image segmentations and clustering text passages. Our work was published in HCOMP 2018

On the Impossibility of Convex Inference in Human Computation

Human computation or crowdsourcing involves joint inference of the ground-truth-answers and the worker-abilities by optimizing an objective function, for instance, by maximizing the data likelihood based on an assumed underlying model. A variety of methods have been proposed in the literature to address this inference problem. As far as we know, none of the objective functions in existing methods is convex. In machine learning and applied statistics, a convex function such as the objective function of support vector machines (SVMs) is generally preferred, since it can leverage the high-performance algorithms and rigorous guarantees established in the extensive literature on convex optimization. One may thus wonder if there exists a meaningful convex objective function for the inference problem in human computation. In this paper, we investigate this convexity issue for human computation. We take an axiomatic approach by formulating a set of axioms that impose two mild and natural assumptions on the objective function for the inference. Under these axioms, we show that it is unfortunately impossible to ensure convexity of the inference problem. On the other hand, we show that interestingly, in the absence of a requirement to model "spammers", one can construct reasonable objective functions for crowdsourcing that guarantee convex inference.Comment: AAAI 201

크라우드소싱 시스템에서의 빠르고 신뢰성 높은 추론 알고리즘

학위논문 (박사) -- 서울대학교 대학원 : 공과대학 전기·컴퓨터공학부, 2021. 2. 정교민.As the need for large scale labeled data grows in various fields, the appearance of web-based crowdsourcing systems gives a promising solution to exploiting the wisdom of crowds efficiently in a short time with a relatively low budget. Despite their efficiency, crowdsourcing systems have an inherent problem in that responses from workers can be unreliable since workers are low-paid and have low responsibility. Although simple majority voting can be a natural solution, various research studies have sought to aggregate noisy responses to obtain greater reliability in results. In this dissertation, we propose novel iterative massage-passing style algorithms to infer the groundtruths from noisy answers, which can be directly applied to real crowdsourcing systems. While EM-based algorithms get the limelight in crowdsourcing systems due to their useful inference techniques, our proposed algorithms draw faster and more reliable answers through an iterative scheme based on the idea of low-rank matrix approximations. We show that the performance of our proposed iterative algorithms are order-optimal, which outperforms majority voting and EM-based algorithms. Unlike other researches solving simple binary-choice questions (yes & no), our studies cover more complex task types which contain multiple-choice questions, short-answer questions, K-approval voting, and real-valued vector regression.다양한 분야에서 라벨된 빅데이터를 필요로 하는 현재, 웹 기반 크라우드소싱 서비스들이 출범하며 상대적으로 적은 예산과 짧은 시간에도 효율적으로 사람들의 지혜를 활용할 수 있는 방법들이 제시되고 있다. 이러한 방법들의 효율성에도 불구하고, 크라우드소싱 시스템의 선천적인 문제점은 일을 맡은 사람들의 적은 보상 및 책임감 결여로 인해 그들의 응답을 완전히 신뢰할 수 없다는 점에 있다. 이에 다수결 방식이 자연스러운 해법으로 사용되지만, 보다 신뢰 높은 답을 얻어내기 위해 많은 연구들이 진행되고 있다. 본 박사학위 논문에서는 크라우드소싱 시스템에서 수많은 사람들로부터 받은 응답들을 모아 신뢰성 높은 응답을 추론하는 반복적 메세지전달 형태의 알고리즘들을 제시한다. 본 알고리즘들은 낮은랭크근사에 기반한 반복 추론 방법으로, 기존에 각광받던 EM 알고리즘들에 비해 더 빠르고 신뢰적인 정답을 추론해낸다. 더불어 본 알고리즘들의 추론 정확도가 최적에 매우 근접하며 다수결 방식 및 EM 알고리즘들의 정확도를 상회한다는 것을 이론적 증명 및 실험적 결과를 통해 제시한다. 본 연구는 실제 크라우드소싱에서 대다수의 응답 유형을 차지하는 객관식 응답, 주관식 응답, 복수 선택 응답, 및 실수 값 응답의 추론 문제를 다루며, 기존 양자택일 응답 추론 문제만을 다루는 기존 연구들과 큰 차별성을 가진다.1 Introduction 1 2 Background 9 2.1 Crowdsourcing Systems for Binary-choice Questions 9 2.1.1 Majority Voting 10 2.1.2 Expectation Maximization 11 2.1.3 Message Passing 11 3 Crowdsourcing Systems for Multiple-choice Questions 12 3.1 Related Work 13 3.2 Problem Setup 16 3.3 Inference Algorithm 17 3.3.1 Task Allocation 17 3.3.2 Multiple Iterative Algorithm 18 3.3.3 Task Allocation for General Setting 20 3.4 Applications 23 3.5 Analysis of Algorithms 25 3.5.1 Quality of Workers 25 3.5.2 Bound on the Average Error Probability 27 3.5.3 Proof of the Error Bounds 29 3.5.4 Proof of Sub-Gaussianity 32 3.6 Experimental Results 36 3.6.1 Comparison with Other Algorithms 37 3.6.2 Adaptive Scenario 38 3.6.3 Simulations on a Set of Various D Values 41 3.7 Conclusion 43 4 Crowdsourcing Systems for Multiple-choice Questions with K-Approval Voting 45 4.1 Related Work 47 4.2 Problem Setup 49 4.2.1 Problem Definition 49 4.2.2 Worker Model for Various (D, K) 50 4.3 Inference Algorithm 51 4.4 Analysis of Algorithms 53 4.4.1 Worker Model 55 4.4.2 Quality of Workers 56 4.4.3 Bound on the Average Error Probability 58 4.4.4 Proof of the Error Bounds 59 4.4.5 Proof of Sub-Gaussianity 62 4.4.6 Phase Transition 67 4.5 Experimental Results 68 4.5.1 Performance on the Average Error with q and l 68 4.5.2 Relationship between Reliability and y-message 69 4.5.3 Performance on the Average Error with Various (D, K) Pairs 69 4.6 Conclusion 72 5 Crowdsourcing Systems for Real-valued Vector Regression 73 5.1 Related Work 75 5.2 Problem Setup 77 5.3 Inference Algorithm 78 5.3.1 Task Message 79 5.3.2 Worker Message 80 5.4 Analysis of Algorithms 81 5.4.1 Worker Model 81 5.4.2 Oracle Estimator 84 5.4.3 Bound on the Average Error Probability 86 5.5 Experimental Results 91 5.5.1 Real Crowdsourcing Data 91 5.5.2 Verification of the Error Bounds with Synthetic data 96 5.6 Conclusion 98 6 Conclusions 99Docto

파라미터 학습 통한 데이터 잡음 및 간섭극복 연구

학위논문 (박사) -- 서울대학교 대학원 : 공과대학 전기·컴퓨터공학부, 2021. 2. 정교민.인공신경망 모델에 다량의 데이터를 학습시키는 방식은 컴퓨터 비전 및 자연어 처리 분야의 문제들을 해결하는데 새로운 패러다임으로 자리매김하였다. 기존 사람의 직관으로 모델을 설정하는 방식과 비교하여 높은 성능을 달성할 수 있었으나, 학습데이터의 양과 품질에 따라서 그 성능이 크게 좌우된다. 이렇게 인공 신경망을 효과적으로 훈련하려면 많은 양의 데이터를 모으는 것과 데이터의 품질을 저하시키는 요인을 파악하는 것이 중요하다. 본 연구에서는 라벨링된 데이터의 품질을 결정하는 주요 요인으로 알려져 있는 잡음(Noise)과 간섭(Interference)을 극복할 수 있는 기법을 제시한다. 연구자들은 일반적으로 웹기반의 크라우드 소싱시스템을 사용하여 다양한 사람들로부터 답변을 수집하여 데이터그룹을 구성한다\cite{simonyan2014very}. 그러나 사람들의 답변으로 얻는 데이터는 작업 지침에 대한 오해, 책임 부족 및 고유한 오류로 인해서 데이터 입력(Input)과 출력(Target)사이에 잡음이 포함된다. 본 연구에서는 이렇게 크라우드 소싱을 통해 라벨링된 데이터에 존재하는 잡음을 극복하기 위한 추론 알고리즘을 제안한다. 두번째로, 모델의 학습성능을 저하시키는 요인인 데이터간의 간섭을 다룬다. 잡음이 제거되어 정제된 입력과 출력을 라벨링된 데이터 샘플이라고 하면, 학습시에 샘플들 사이의 관계를 생각할 수 있다. 사람 수준의 인공지능에 도달하기 위해서는 하나의 모델이 하나의 문제만을 해결하는 것이 아니라 시간상 순차적으로 직면하는 여러 문제를 동시에 해결할 수 있어야 한다. 이러한 상황에서, 샘플들 사이에 간섭이 발생할 수 있고, 학계에서는 연속학습(Continual Learning)에서의 "Catastrophic Forgetting"또는 "Semantic Drift"으로 정의하고 있다. 본 연구에서는 이러한 간섭을 효과적으로 극복하기 위한 방법에 대한 연구를 다룬다. 앞서 언급한 데이터 잡음을 극복하기 위해서 첫 번째 장에서는 크라우드 소싱 시스템의 이산 객관식 및 실수 벡터 회귀 작업에 대한 새로운 추론 알고리즘을 각각 제안한다. 제안 된 알고리즘은 크라우드 소싱 모델을 그래프 모델(Graphical Model)로서 상정하고, 테스크와 답변을 주는 사람들간의 두 가지 유형의 메시지를 반복적으로 주고 받음으로써 각 작업의 정답과 각 작업자의 신뢰성을 추정 할 수 있다. 또한 이들의 평균 성능은 확률적 군중 모델을 이용하여 분석하고 입증한다. 이러한 성능에러 한계는 작업당 할당되는 사람들의 수와 작업자의 평균 신뢰성의해 결정된다. 사람들의 평균 신뢰도가 일정 수준을 넘어서면, 제안된 알고리즘의 평균 성능은 모든 작업자의 신뢰성을 알고있는 오라클 추정기 (이론적인 한계)에 수렴한다. 실제 데이터 세트와 합성 데이터 세트 모두에 대한 광범위한 실험을 통해, 제안된 알고리즘의 실제 성능이 이전의 state-of-the-art 알고리즘들 보다 우수하다는 것을 입증한다. 논문의 두 번째 장에서는 연속학습상황에서 데이터샘플사이에 발생하는 간섭을 해결하기 위해, 항상성기반의 메타 학습 구조 (Homeostatic Meta Model)를 제안한다. 구체적으로, 이전 테스크 중요한 학습 변수를 찾고 정규화에 선별적으로 적용하는 방법을 사용하는데, 제안된 모델은 이러한 정규화의 강도를 자동으로 제어한다. 이러한 기법은 새로운 학습을 진행할 때 이전에 획득한 지식을 최소한으로 잃어버리도록 인공신경망의 학습을 유도한다. 다양한 유형의 연속 학습 과제에서 제안된 방법을 검증하는데, 실험적으로 제안된 방법이 학습의 간섭완화 측면에서 기존 방법보다 우수하다는 점을 보인다.또한 기존 시냅스 가소성 기반 방법들에 비해 상대적으로 변화에 강인하다.제안된 모델에 의해 생성된 정규화의 강도 값은 시냅스에서 항상성 의 음의 피드백 메커니즘과 유사하게, 특정 범위 내에서 능동적으로 제어된다.Data-driven approaches based on neural networks have emerged as new paradigm to solve problems in computer vision and natural language processing fields. These approaches achieve better performance compared to existing human-design approaches (heuristic), however, these performance gains solely relies on a large amount of high quality labeled data. Accordingly, it is important to collect a large amount of data and improve the quality of data by analyzing degrading factors in order to well-train a model. In this dissertation, I propose iterative algorithms to relieve noise of labeled data in crowdsourcing system and meta architecture to alleviate interference among them in continual learning scenarios respectively. Researchers generally collect data using crowdsourcing system which utilizes human evaluations. However, human annotators' decisions may vary significantly due to misconceptions of task instructions, the lack of responsibility, and inherent noise. To relieve the noise in responses from crowd annotators, I propose novel inference algorithms for discrete multiple choice and real-valued vector regression tasks. Web-based crowdsourcing platforms are widely used for collecting large amount of labeled data. Due to low-paid workers and inherent noise, the quality of acquired data could be easily degraded. The proposed algorithms can overcome the noise by estimating the true answer of each task and a reliability of each worker updating two types of messages iteratively. For performance guarantee, the performances of the algorithms are theoretically proved under probabilistic crowd model. Interestingly, their performance bounds depend on the number of queries per task and the average quality of workers. Under a certain condition, each average performance becomes close to an oracle estimator which knows the reliability of every worker (theoretical upper bound). Through extensive experiments with both real-world and synthetic datasets, the practical performance of algorithms are verified. In fact, they are superior to other state-of-the-art algorithms. Second, when a model learns a sequence of tasks one by one (continual learning), previously learned knowledge may conflict with new knowledge. It is well-known phenomenon called "Catastrophic Forgetting" or "Semantic Drift". In this dissertation, we call the phenomena "Interference" since it occurs between two knowledge from labeled data separated in time. It is essential to control the amount of noise and interference for neural network to be well-trained. In the second part of dissertation, to solve the Interference among labeled data from consecutive tasks in continual learning scenario, a homeostasis-inspired meta learning architecture (HM) is proposed. The HM automatically controls the intensity of regularization (IoR) by capturing important parameters from the previous tasks and the current learning direction. By adjusting IoR, a learner can balance the amount of interference and degrees of freedom for its current learning. Experimental results are provided on various types of continual learning tasks. Those results show that the proposed method notably outperforms the conventional methods in terms of average accuracy and amount of the interference. In experiments, I verify that HM is relatively stable and robust compared to the existing Synaptic Plasticity based methods. Interestingly, the IoR generated by HM appears to be proactively controlled within a certain range, which resembles a negative feedback mechanism of homeostasis in synapses.Contents Abstract Contents List of Tables List of Figures 1 INTRODUCTION 1 2 Reliable multiple-choice iterative algorithm for crowdsourcing systems 6 2.1 Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2.2 Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 2.2.1 Task Allocation . . . . . . . . . . . . . . . . . . . . . . . . . 7 2.2.2 Multiple Iterative Algorithm . . . . . . . . . . . . . . . . . . 8 2.2.3 Task Allocation for General Setting . . . . . . . . . . . . . . 10 2.3 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2.4 Analysis of algorithms . . . . . . . . . . . . . . . . . . . . . . . . . 16 2.4.1 Quality of workers . . . . . . . . . . . . . . . . . . . . . . . 16 2.4.2 Bound on the Average Error Probability . . . . . . . . . . . . 18 2.4.3 Proof of the Theorem 1 . . . . . . . . . . . . . . . . . . . . . 20 2.4.4 Proof of Sub-Gaussianity . . . . . . . . . . . . . . . . . . . . 22 2.5 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 iii2.6 Related Literature . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 3 Reliable Aggregation Method for Vector Regression in Crowdsourcing 38 3.1 Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 3.2 Inference Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 3.2.1 Task Message . . . . . . . . . . . . . . . . . . . . . . . . . . 40 3.2.2 Worker Message . . . . . . . . . . . . . . . . . . . . . . . . 40 3.3 Experimental Results . . . . . . . . . . . . . . . . . . . . . . . . . . 42 3.3.1 Real crowdsourcing data . . . . . . . . . . . . . . . . . . . . 43 3.4 Performance Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . 47 3.4.1 Dirichlet crowd model . . . . . . . . . . . . . . . . . . . . . 48 3.4.2 Error Bound . . . . . . . . . . . . . . . . . . . . . . . . . . 49 3.4.3 Optimality of Oracle Estimator . . . . . . . . . . . . . . . . . 51 3.4.4 Performance Proofs . . . . . . . . . . . . . . . . . . . . . . . 52 3.5 Related Literature . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 4 Homeostasis-Inspired Meta Continual Learning 60 4.1 Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 4.1.1 Continual Learning . . . . . . . . . . . . . . . . . . . . . . . 60 4.1.2 Meta Learning . . . . . . . . . . . . . . . . . . . . . . . . . 62 4.2 Homeostatic Meta-Model . . . . . . . . . . . . . . . . . . . . . . . . 63 4.3 Preliminary Experiments and Findings . . . . . . . . . . . . . . . . . 66 4.3.1 Block-wise Permutation . . . . . . . . . . . . . . . . . . . . 67 4.3.2 Performance Metrics . . . . . . . . . . . . . . . . . . . . . . 68 4.4 Experiment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68 4.4.1 Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 4.4.2 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 4.4.3 Overall Performance . . . . . . . . . . . . . . . . . . . . . . 70 4.5 Related Literature . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 iv4.6 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71 5 Conclusion 78 Abstract (In Korean) 89Docto

Machine learning from crowds a systematic review of its applications

Crowdsourcing opens the door to solving a wide variety of problems that previ-ously were unfeasible in the field of machine learning, allowing us to obtain rela-tively low cost labeled data in a small amount of time. However, due to theuncertain quality of labelers, the data to deal with are sometimes unreliable, forcingpractitioners to collect information redundantly, which poses new challenges in thefield. Despite these difficulties, many applications of machine learning usingcrowdsourced data have recently been published that achieved state of the artresults in relevant problems. We have analyzed these applications following a sys-tematic methodology, classifying them into different fields of study, highlightingseveral of their characteristics and showing the recent interest in the use of crowd-sourcing for machine learning. We also identify several exciting research linesbased on the problems that remain unsolved to foster future research in this field

Considering Human Aspects on Strategies for Designing and Managing Distributed Human Computation

A human computation system can be viewed as a distributed system in which the processors are humans, called workers. Such systems harness the cognitive power of a group of workers connected to the Internet to execute relatively simple tasks, whose solutions, once grouped, solve a problem that systems equipped with only machines could not solve satisfactorily. Examples of such systems are Amazon Mechanical Turk and the Zooniverse platform. A human computation application comprises a group of tasks, each of them can be performed by one worker. Tasks might have dependencies among each other. In this study, we propose a theoretical framework to analyze such type of application from a distributed systems point of view. Our framework is established on three dimensions that represent different perspectives in which human computation applications can be approached: quality-of-service requirements, design and management strategies, and human aspects. By using this framework, we review human computation in the perspective of programmers seeking to improve the design of human computation applications and managers seeking to increase the effectiveness of human computation infrastructures in running such applications. In doing so, besides integrating and organizing what has been done in this direction, we also put into perspective the fact that the human aspects of the workers in such systems introduce new challenges in terms of, for example, task assignment, dependency management, and fault prevention and tolerance. We discuss how they are related to distributed systems and other areas of knowledge.Comment: 3 figures, 1 tabl

Enhancing the use of online 3d multimedia content through the analysis of user interactions

De plus en plus de contenus 3D interactifs sont disponibles sur la toile. Visualiser et manipuler ces contenus 3D en temps réel, de façon naturelle et intuitive, devient donc une nécessité. Les applications visées sont nombreuses : le e-commerce, l'éducation et la formation en ligne, la conception, ou l'architecture dans le contexte par exemple de musées virtuels ou de communautés virtuelles. L'utilisation de contenus 3D en ligne ne propose pas de remplacer les contenus traditionnels, tels que les textes, les images ou les vidéos, mais plutôt d'utiliser la 3D en complément, pour enrichir ces contenus. La toile est désormais une plate-forme où les contenus hypertexte, hypermédia, et 3D sont simultanément disponibles pour les utilisateurs. Cette utilisation des contenus 3D pose cependant deux questions principales. Tout d'abord, les interactions 3D sont souvent lourdes puisqu'elles comprennent de nombreux degrés de liberté; la navigation dans les contenus 3D peut s'en trouver inefficace et lente. Nous abordons ce problème en proposant un nouveau paradigme basé sur l'analyse des interactions (crowdsourcing). En analysant les interactions d'utilisateurs 3D, nous identifions des régions d'intérêt (ROI), et générons des recommandations pour les utilisateurs suivants. Ces recommandations permettent à la fois de réduire le temps d'interaction pour identifier une ROI d'un objet 3D et également de simplifier les interactions 3D nécessaires. De plus, les scènes ou objets 3D contiennent une information visuelle riche. Les sites Web traditionnels contiennent, eux, principalement des informations descriptives (textuelles) ainsi que des hyperliens pour permettre la navigation. Des sites contenants d'une part de l'information textuelle, et d'autre part de l'information 3D peuvent s'avérer difficile à appréhender pour les utilisateurs. Pour permettre une navigation cohérente entre les informations 3D et textuelles, nous proposons d'utiliser le crowdsourcing pour la construction d'associations sémantiques entre le texte et la visualisation en 3D. Les liens produits sont proposés aux utilisateurs suivants pour naviguer facilement vers un point de vue d'un objet 3D associé à un contenu textuel. Nous évaluons ces deux méthodes par des études expérimentales. Les évaluations montrent que les recommandations réduisent le temps d'interaction 3D. En outre, les utilisateurs apprécient l'association sémantique proposée, c'est-à-dire, une majorité d'utilisateurs indique que les recommandations ont été utiles pour eux, et préfèrent la navigation en 3D proposée qui consiste à utiliser les liens sémantiques ainsi que la souris par rapport à des interactions utilisant seulement la souris. ABSTRACT : Recent years have seen the development of interactive 3D graphics on the Web. The ability to visualize and manipulate 3D content in real time seems to be the next evolution of the Web for a wide number of application areas such as e-commerce, education and training, architecture design, virtual museums and virtual communities. The use of online 3D graphics in these application domains does not mean to substitute traditional web content of texts, images and videos, but rather acts as a complement for it. The Web is now a platform where hypertext, hypermedia, and 3D graphics are simultaneously available to users. This use of online 3D graphics, however, poses two main issues. First, since 3D interactions are cumbersome as they provide numerous degrees of freedom, 3D browsing may be inefficient. We tackle this problem by proposing a new paradigm based on crowdsourcing to ease online 3D interactions, that consists of analyzing 3D user interactions to identify Regions of Interest (ROIs), and generating recommendations to subsequent users. The recommendations both reduce 3D browsing time and simplify 3D interactions. Second, 3D graphics contain purely rich visual information of the concepts. On the other hand, traditional websites mainly contain descriptive information (text) with hyperlinks as navigation means. The problem is that viewing and interacting with the websites that use two very different mediums (hypertext and 3D graphics) may be complicated for users. To address this issue, we propose to use crowdsourcing for building semantic associations between texts and 3D visualizations. The produced links are suggested to upcoming users so that they can readily locate 3D visualization associated with a textual content. We evaluate the proposed methods with experimental user studies. The evaluations show that the recommendations reduce 3D interaction time. Moreover, the results from the user study showed that our proposed semantic association is appreciated by users, that is, a majority of users assess that recommendations were helpful for them, and browsing 3D objects using both mouse interactions and the proposed links is preferred compared to having only mouse interactions

Essays In Algorithmic Market Design Under Social Constraints

Rapid technological advances over the past few decades---in particular, the rise of the internet---has significantly reshaped and expanded the meaning of our everyday social activities, including our interactions with our social circle, the media, and our political and economic activities This dissertation aims to tackle some of the unique societal challenges underlying the design of automated online platforms that interact with people and organizations---namely, those imposed by legal, ethical, and strategic considerations. I narrow down attention to fairness considerations, learning with repeated trials, and competition for market share. In each case, I investigate the broad issue in a particular context (i.e. online market), and present the solution my research offers to the problem in that application. Addressing interdisciplinary problems, such as the ones in this dissertation, requires drawing ideas and techniques from various disciplines, including theoretical computer science, microeconomics, and applied statistics. The research presented here utilizes a combination of theoretical and data analysis tools to shed light on some of the key challenges in designing algorithms for today\u27s online markets, including crowdsourcing and labor markets, online advertising, and social networks among others