32 research outputs found
Analysis of Crowdsourced Sampling Strategies for HodgeRank with Sparse Random Graphs
Crowdsourcing platforms are now extensively used for conducting subjective
pairwise comparison studies. In this setting, a pairwise comparison dataset is
typically gathered via random sampling, either \emph{with} or \emph{without}
replacement. In this paper, we use tools from random graph theory to analyze
these two random sampling methods for the HodgeRank estimator. Using the
Fiedler value of the graph as a measurement for estimator stability
(informativeness), we provide a new estimate of the Fiedler value for these two
random graph models. In the asymptotic limit as the number of vertices tends to
infinity, we prove the validity of the estimate. Based on our findings, for a
small number of items to be compared, we recommend a two-stage sampling
strategy where a greedy sampling method is used initially and random sampling
\emph{without} replacement is used in the second stage. When a large number of
items is to be compared, we recommend random sampling with replacement as this
is computationally inexpensive and trivially parallelizable. Experiments on
synthetic and real-world datasets support our analysis
Exploring Outliers in Crowdsourced Ranking for QoE
Outlier detection is a crucial part of robust evaluation for crowdsourceable
assessment of Quality of Experience (QoE) and has attracted much attention in
recent years. In this paper, we propose some simple and fast algorithms for
outlier detection and robust QoE evaluation based on the nonconvex optimization
principle. Several iterative procedures are designed with or without knowing
the number of outliers in samples. Theoretical analysis is given to show that
such procedures can reach statistically good estimates under mild conditions.
Finally, experimental results with simulated and real-world crowdsourcing
datasets show that the proposed algorithms could produce similar performance to
Huber-LASSO approach in robust ranking, yet with nearly 8 or 90 times speed-up,
without or with a prior knowledge on the sparsity size of outliers,
respectively. Therefore the proposed methodology provides us a set of helpful
tools for robust QoE evaluation with crowdsourcing data.Comment: accepted by ACM Multimedia 2017 (Oral presentation). arXiv admin
note: text overlap with arXiv:1407.763
Hybrid-MST: A hybrid active sampling strategy for pairwise preference aggregation
In this paper we present a hybrid active sampling strategy for pairwise
preference aggregation, which aims at recovering the underlying rating of the
test candidates from sparse and noisy pairwise labelling. Our method employs
Bayesian optimization framework and Bradley-Terry model to construct the
utility function, then to obtain the Expected Information Gain (EIG) of each
pair. For computational efficiency, Gaussian-Hermite quadrature is used for
estimation of EIG. In this work, a hybrid active sampling strategy is proposed,
either using Global Maximum (GM) EIG sampling or Minimum Spanning Tree (MST)
sampling in each trial, which is determined by the test budget. The proposed
method has been validated on both simulated and real-world datasets, where it
shows higher preference aggregation ability than the state-of-the-art methods
Hybrid-MST: A hybrid active sampling strategy for pairwise preference aggregation
In this paper we present a hybrid active sampling strategy for pairwise
preference aggregation, which aims at recovering the underlying rating of the
test candidates from sparse and noisy pairwise labelling. Our method employs
Bayesian optimization framework and Bradley-Terry model to construct the
utility function, then to obtain the Expected Information Gain (EIG) of each
pair. For computational efficiency, Gaussian-Hermite quadrature is used for
estimation of EIG. In this work, a hybrid active sampling strategy is proposed,
either using Global Maximum (GM) EIG sampling or Minimum Spanning Tree (MST)
sampling in each trial, which is determined by the test budget. The proposed
method has been validated on both simulated and real-world datasets, where it
shows higher preference aggregation ability than the state-of-the-art methods