486 research outputs found
Crowdsourcing with Sparsely Interacting Workers
We consider estimation of worker skills from worker-task interaction data
(with unknown labels) for the single-coin crowd-sourcing binary classification
model in symmetric noise. We define the (worker) interaction graph whose nodes
are workers and an edge between two nodes indicates whether or not the two
workers participated in a common task. We show that skills are asymptotically
identifiable if and only if an appropriate limiting version of the interaction
graph is irreducible and has odd-cycles. We then formulate a weighted rank-one
optimization problem to estimate skills based on observations on an
irreducible, aperiodic interaction graph. We propose a gradient descent scheme
and show that for such interaction graphs estimates converge asymptotically to
the global minimum. We characterize noise robustness of the gradient scheme in
terms of spectral properties of signless Laplacians of the interaction graph.
We then demonstrate that a plug-in estimator based on the estimated skills
achieves state-of-art performance on a number of real-world datasets. Our
results have implications for rank-one matrix completion problem in that
gradient descent can provably recover rank-one matrices based on
off-diagonal observations of a connected graph with a single odd-cycle
Gradient descent for sparse rank-one matrix completion for crowd-sourced aggregation of sparsely interacting workers
We consider worker skill estimation for the singlecoin
Dawid-Skene crowdsourcing model. In
practice skill-estimation is challenging because
worker assignments are sparse and irregular due
to the arbitrary, and uncontrolled availability of
workers. We formulate skill estimation as a
rank-one correlation-matrix completion problem,
where the observed components correspond to
observed label correlation between workers. We
show that the correlation matrix can be successfully
recovered and skills identifiable if and only
if the sampling matrix (observed components) is
irreducible and aperiodic. We then propose an
efficient gradient descent scheme and show that
skill estimates converges to the desired global optima
for such sampling matrices. Our proof is
original and the results are surprising in light of
the fact that even the weighted rank-one matrix
factorization problem is NP hard in general. Next
we derive sample complexity bounds for the noisy
case in terms of spectral properties of the signless
Laplacian of the sampling matrix. Our proposed
scheme achieves state-of-art performance on a
number of real-world datasets.Published versio
Mouse Behavior Recognition with The Wisdom of Crowd
In this thesis, we designed and implemented a crowdsourcing system to annotatemouse behaviors in videos; this involves the development of a novel clip-based video labeling tools, that is more efficient than traditional labeling tools in crowdsourcing platform, as well as the design of probabilistic inference algorithms that predict the true labels and the workers' expertise from multiple workers' responses. Our algorithms are shown to perform better than majority vote heuristic. We also carried out extensive experiments to determine the effectiveness of our labeling tool, inference algorithms and the overall system
A Provably Improved Algorithm for Crowdsourcing with Hard and Easy Tasks
Crowdsourcing is a popular method used to estimate ground-truth labels by
collecting noisy labels from workers. In this work, we are motivated by
crowdsourcing applications where each worker can exhibit two levels of accuracy
depending on a task's type. Applying algorithms designed for the traditional
Dawid-Skene model to such a scenario results in performance which is limited by
the hard tasks. Therefore, we first extend the model to allow worker accuracy
to vary depending on a task's unknown type. Then we propose a spectral method
to partition tasks by type. After separating tasks by type, any Dawid-Skene
algorithm (i.e., any algorithm designed for the Dawid-Skene model) can be
applied independently to each type to infer the truth values. We theoretically
prove that when crowdsourced data contain tasks with varying levels of
difficulty, our algorithm infers the true labels with higher accuracy than any
Dawid-Skene algorithm. Experiments show that our method is effective in
practical applications
νλΌλ―Έν° νμ΅ ν΅ν λ°μ΄ν° μ‘μ λ° κ°μ극볡 μ°κ΅¬
νμλ
Όλ¬Έ (λ°μ¬) -- μμΈλνκ΅ λνμ : 곡과λν μ κΈ°Β·μ»΄ν¨ν°κ³΅νλΆ, 2021. 2. μ κ΅λ―Ό.μΈκ³΅μ κ²½λ§ λͺ¨λΈμ λ€λμ λ°μ΄ν°λ₯Ό νμ΅μν€λ λ°©μμ μ»΄ν¨ν° λΉμ λ° μμ°μ΄ μ²λ¦¬ λΆμΌμ λ¬Έμ λ€μ ν΄κ²°νλλ° μλ‘μ΄ ν¨λ¬λ€μμΌλ‘ μ리맀κΉνμλ€. κΈ°μ‘΄ μ¬λμ μ§κ΄μΌλ‘ λͺ¨λΈμ μ€μ νλ λ°©μκ³Ό λΉκ΅νμ¬ λμ μ±λ₯μ λ¬μ±ν μ μμμΌλ, νμ΅λ°μ΄ν°μ μκ³Ό νμ§μ λ°λΌμ κ·Έ μ±λ₯μ΄ ν¬κ² μ’μ°λλ€. μ΄λ κ² μΈκ³΅ μ κ²½λ§μ ν¨κ³Όμ μΌλ‘ νλ ¨νλ €λ©΄ λ§μ μμ λ°μ΄ν°λ₯Ό λͺ¨μΌλ κ²κ³Ό λ°μ΄ν°μ νμ§μ μ νμν€λ μμΈμ νμ
νλ κ²μ΄ μ€μνλ€. λ³Έ μ°κ΅¬μμλ λΌλ²¨λ§λ λ°μ΄ν°μ νμ§μ κ²°μ νλ μ£Όμ μμΈμΌλ‘ μλ €μ Έ μλ μ‘μ(Noise)κ³Ό κ°μ(Interference)μ 극볡ν μ μλ κΈ°λ²μ μ μνλ€.
μ°κ΅¬μλ€μ μΌλ°μ μΌλ‘ μΉκΈ°λ°μ ν¬λΌμ°λ μμ±μμ€ν
μ μ¬μ©νμ¬ λ€μν μ¬λλ€λ‘λΆν° λ΅λ³μ μμ§νμ¬ λ°μ΄ν°κ·Έλ£Ήμ ꡬμ±νλ€\cite{simonyan2014very}. κ·Έλ¬λ μ¬λλ€μ λ΅λ³μΌλ‘ μ»λ λ°μ΄ν°λ μμ
μ§μΉ¨μ λν μ€ν΄, μ±
μ λΆμ‘± λ° κ³ μ ν μ€λ₯λ‘ μΈν΄μ λ°μ΄ν° μ
λ ₯(Input)κ³Ό μΆλ ₯(Target)μ¬μ΄μ μ‘μμ΄ ν¬ν¨λλ€. λ³Έ μ°κ΅¬μμλ μ΄λ κ² ν¬λΌμ°λ μμ±μ ν΅ν΄ λΌλ²¨λ§λ λ°μ΄ν°μ μ‘΄μ¬νλ μ‘μμ 극볡νκΈ° μν μΆλ‘ μκ³ λ¦¬μ¦μ μ μνλ€.
λλ²μ§Έλ‘, λͺ¨λΈμ νμ΅μ±λ₯μ μ νμν€λ μμΈμΈ λ°μ΄ν°κ°μ κ°μμ λ€λ£¬λ€. μ‘μμ΄ μ κ±°λμ΄ μ μ λ μ
λ ₯κ³Ό μΆλ ₯μ λΌλ²¨λ§λ λ°μ΄ν° μνμ΄λΌκ³ νλ©΄, νμ΅μμ μνλ€ μ¬μ΄μ κ΄κ³λ₯Ό μκ°ν μ μλ€. μ¬λ μμ€μ μΈκ³΅μ§λ₯μ λλ¬νκΈ° μν΄μλ νλμ λͺ¨λΈμ΄ νλμ λ¬Έμ λ§μ ν΄κ²°νλ κ²μ΄ μλλΌ μκ°μ μμ°¨μ μΌλ‘ μ§λ©΄νλ μ¬λ¬ λ¬Έμ λ₯Ό λμμ ν΄κ²°ν μ μμ΄μΌ νλ€. μ΄λ¬ν μν©μμ, μνλ€ μ¬μ΄μ κ°μμ΄ λ°μν μ μκ³ , νκ³μμλ μ°μνμ΅(Continual Learning)μμμ "Catastrophic Forgetting"λλ "Semantic Drift"μΌλ‘ μ μνκ³ μλ€. λ³Έ μ°κ΅¬μμλ μ΄λ¬ν κ°μμ ν¨κ³Όμ μΌλ‘ 극볡νκΈ° μν λ°©λ²μ λν μ°κ΅¬λ₯Ό λ€λ£¬λ€.
μμ μΈκΈν λ°μ΄ν° μ‘μμ 극볡νκΈ° μν΄μ 첫 λ²μ§Έ μ₯μμλ ν¬λΌμ°λ μμ± μμ€ν
μ μ΄μ° κ°κ΄μ λ° μ€μ λ²‘ν° νκ· μμ
μ λν μλ‘μ΄ μΆλ‘ μκ³ λ¦¬μ¦μ κ°κ° μ μνλ€. μ μ λ μκ³ λ¦¬μ¦μ ν¬λΌμ°λ μμ± λͺ¨λΈμ κ·Έλν λͺ¨λΈ(Graphical Model)λ‘μ μμ νκ³ , ν
μ€ν¬μ λ΅λ³μ μ£Όλ μ¬λλ€κ°μ λ κ°μ§ μ νμ λ©μμ§λ₯Ό λ°λ³΅μ μΌλ‘ μ£Όκ³ λ°μμΌλ‘μ¨ κ° μμ
μ μ λ΅κ³Ό κ° μμ
μμ μ λ’°μ±μ μΆμ ν μ μλ€. λν μ΄λ€μ νκ· μ±λ₯μ νλ₯ μ κ΅°μ€ λͺ¨λΈμ μ΄μ©νμ¬ λΆμνκ³ μ
μ¦νλ€. μ΄λ¬ν μ±λ₯μλ¬ νκ³λ μμ
λΉ ν λΉλλ μ¬λλ€μ μμ μμ
μμ νκ· μ λ’°μ±μν΄ κ²°μ λλ€. μ¬λλ€μ νκ· μ λ’°λκ° μΌμ μμ€μ λμ΄μλ©΄, μ μλ μκ³ λ¦¬μ¦μ νκ· μ±λ₯μ λͺ¨λ μμ
μμ μ λ’°μ±μ μκ³ μλ μ€λΌν΄ μΆμ κΈ° (μ΄λ‘ μ μΈ νκ³)μ μλ ΄νλ€. μ€μ λ°μ΄ν° μΈνΈμ ν©μ± λ°μ΄ν° μΈνΈ λͺ¨λμ λν κ΄λ²μν μ€νμ ν΅ν΄, μ μλ μκ³ λ¦¬μ¦μ μ€μ μ±λ₯μ΄ μ΄μ μ state-of-the-art μκ³ λ¦¬μ¦λ€ λ³΄λ€ μ°μνλ€λ κ²μ μ
μ¦νλ€.
λ
Όλ¬Έμ λ λ²μ§Έ μ₯μμλ μ°μνμ΅μν©μμ λ°μ΄ν°μνμ¬μ΄μ λ°μνλ κ°μμ ν΄κ²°νκΈ° μν΄, νμμ±κΈ°λ°μ λ©ν νμ΅ κ΅¬μ‘° (Homeostatic Meta Model)λ₯Ό μ μνλ€. ꡬ체μ μΌλ‘, μ΄μ ν
μ€ν¬ μ€μν νμ΅ λ³μλ₯Ό μ°Ύκ³ μ κ·νμ μ λ³μ μΌλ‘ μ μ©νλ λ°©λ²μ μ¬μ©νλλ°, μ μλ λͺ¨λΈμ μ΄λ¬ν μ κ·νμ κ°λλ₯Ό μλμΌλ‘ μ μ΄νλ€. μ΄λ¬ν κΈ°λ²μ μλ‘μ΄ νμ΅μ μ§νν λ μ΄μ μ νλν μ§μμ μ΅μνμΌλ‘ μμ΄λ²λ¦¬λλ‘ μΈκ³΅μ κ²½λ§μ νμ΅μ μ λνλ€. λ€μν μ νμ μ°μ νμ΅ κ³Όμ μμ μ μλ λ°©λ²μ κ²μ¦νλλ°, μ€νμ μΌλ‘ μ μλ λ°©λ²μ΄ νμ΅μ κ°μμν μΈ‘λ©΄μμ κΈ°μ‘΄ λ°©λ²λ³΄λ€ μ°μνλ€λ μ μ 보μΈλ€.λν κΈ°μ‘΄ μλ
μ€ κ°μμ± κΈ°λ° λ°©λ²λ€μ λΉν΄ μλμ μΌλ‘ λ³νμ κ°μΈνλ€.μ μλ λͺ¨λΈμ μν΄ μμ±λ μ κ·νμ κ°λ κ°μ μλ
μ€μμ νμμ± μ μμ νΌλλ°± λ©μ»€λμ¦κ³Ό μ μ¬νκ², νΉμ λ²μ λ΄μμ λ₯λμ μΌλ‘ μ μ΄λλ€.Data-driven approaches based on neural networks have emerged as new paradigm to solve problems in computer vision and natural language processing fields. These approaches achieve better performance compared to existing human-design approaches (heuristic), however, these performance gains solely relies on a large amount of high quality labeled data. Accordingly, it is important to collect a large amount of data and improve the quality of data by analyzing degrading factors in order to well-train a model. In this dissertation, I propose iterative algorithms to relieve noise of labeled data in crowdsourcing system and meta architecture to alleviate interference among them in continual learning scenarios respectively.
Researchers generally collect data using crowdsourcing system which utilizes human evaluations. However, human annotators' decisions may vary significantly due to misconceptions of task instructions, the lack of responsibility, and inherent noise. To relieve the noise in responses from crowd annotators, I propose novel inference algorithms for discrete multiple choice and real-valued vector regression tasks. Web-based crowdsourcing platforms are widely used for collecting large amount of labeled data. Due to low-paid workers and inherent noise, the quality of acquired data could be easily degraded. The proposed algorithms can overcome the noise by estimating the true answer of each task and a reliability of each worker updating two types of messages iteratively. For performance guarantee, the performances of the algorithms are theoretically proved under probabilistic crowd model. Interestingly, their performance bounds depend on the number of queries per task and the average quality of workers. Under a certain condition, each average performance becomes close to an oracle estimator which knows the reliability of every worker (theoretical upper bound). Through extensive experiments with both real-world and synthetic datasets, the practical performance of algorithms are verified. In fact, they are superior to other state-of-the-art algorithms.
Second, when a model learns a sequence of tasks one by one (continual learning), previously learned knowledge may conflict with new knowledge. It is well-known phenomenon called "Catastrophic Forgetting" or "Semantic Drift". In this dissertation, we call the phenomena "Interference" since it occurs between two knowledge from labeled data separated in time. It is essential to control the amount of noise and interference for neural network to be well-trained.
In the second part of dissertation, to solve the Interference among labeled data from consecutive tasks in continual learning scenario, a homeostasis-inspired meta learning architecture (HM) is proposed. The HM automatically controls the intensity of regularization (IoR) by capturing important parameters from the previous tasks and the current learning direction. By adjusting IoR, a learner can balance the amount of interference and degrees of freedom for its current learning. Experimental results are provided on various types of continual learning tasks. Those results show that the proposed method notably outperforms the conventional methods in terms of average accuracy and amount of the interference. In experiments, I verify that HM is relatively stable and robust compared to the existing Synaptic Plasticity based methods. Interestingly, the IoR generated by HM appears to be proactively controlled within a certain range, which resembles a negative feedback mechanism of homeostasis in synapses.Contents
Abstract
Contents
List of Tables
List of Figures
1 INTRODUCTION 1
2 Reliable multiple-choice iterative algorithm for crowdsourcing systems 6
2.1 Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.2 Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.2.1 Task Allocation . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.2.2 Multiple Iterative Algorithm . . . . . . . . . . . . . . . . . . 8
2.2.3 Task Allocation for General Setting . . . . . . . . . . . . . . 10
2.3 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.4 Analysis of algorithms . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.4.1 Quality of workers . . . . . . . . . . . . . . . . . . . . . . . 16
2.4.2 Bound on the Average Error Probability . . . . . . . . . . . . 18
2.4.3 Proof of the Theorem 1 . . . . . . . . . . . . . . . . . . . . . 20
2.4.4 Proof of Sub-Gaussianity . . . . . . . . . . . . . . . . . . . . 22
2.5 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
iii2.6 Related Literature . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
3 Reliable Aggregation Method for Vector Regression in Crowdsourcing 38
3.1 Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
3.2 Inference Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
3.2.1 Task Message . . . . . . . . . . . . . . . . . . . . . . . . . . 40
3.2.2 Worker Message . . . . . . . . . . . . . . . . . . . . . . . . 40
3.3 Experimental Results . . . . . . . . . . . . . . . . . . . . . . . . . . 42
3.3.1 Real crowdsourcing data . . . . . . . . . . . . . . . . . . . . 43
3.4 Performance Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . 47
3.4.1 Dirichlet crowd model . . . . . . . . . . . . . . . . . . . . . 48
3.4.2 Error Bound . . . . . . . . . . . . . . . . . . . . . . . . . . 49
3.4.3 Optimality of Oracle Estimator . . . . . . . . . . . . . . . . . 51
3.4.4 Performance Proofs . . . . . . . . . . . . . . . . . . . . . . . 52
3.5 Related Literature . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
4 Homeostasis-Inspired Meta Continual Learning 60
4.1 Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
4.1.1 Continual Learning . . . . . . . . . . . . . . . . . . . . . . . 60
4.1.2 Meta Learning . . . . . . . . . . . . . . . . . . . . . . . . . 62
4.2 Homeostatic Meta-Model . . . . . . . . . . . . . . . . . . . . . . . . 63
4.3 Preliminary Experiments and Findings . . . . . . . . . . . . . . . . . 66
4.3.1 Block-wise Permutation . . . . . . . . . . . . . . . . . . . . 67
4.3.2 Performance Metrics . . . . . . . . . . . . . . . . . . . . . . 68
4.4 Experiment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
4.4.1 Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
4.4.2 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
4.4.3 Overall Performance . . . . . . . . . . . . . . . . . . . . . . 70
4.5 Related Literature . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
iv4.6 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
5 Conclusion 78
Abstract (In Korean) 89Docto
- β¦