2 research outputs found
Relevance Scoring of Triples Using Ordinal Logistic Classification - The Celosia Triple Scorer at WSDM Cup 2017
In this paper, we report our participation in the Task 2: Triple Scoring of
WSDM Cup challenge 2017. In this task, we were provided with triples of
"type-like" relations which were given human-annotated relevance scores ranging
from 0 to 7, with 7 being the "most relevant" and 0 being the "least relevant".
The task focuses on two such relations: profession and nationality. We built a
system which could automatically predict the relevance scores for unseen
triples. Our model is primarily a supervised machine learning based one in
which we use well-designed features which are used to a make a Logistic Ordinal
Regression based classification model. The proposed system achieves an overall
accuracy score of 0.73 and Kendall's tau score of 0.36.Comment: Triple Scorer at WSDM Cup 2017, see arXiv:1712.0808
Overview of the Triple Scoring Task at the WSDM Cup 2017
This paper provides an overview of the triple scoring task at the WSDM Cup
2017, including a description of the task and the dataset, an overview of the
participating teams and their results, and a brief account of the methods
employed. In a nutshell, the task was to compute relevance scores for
knowledge-base triples from relations, where such scores make sense. Due to the
way the ground truth was constructed, scores were required to be integers from
the range 0..7. For example, reasonable scores for the triples "Tim Burton
profession Director" and "Tim Burton profession Actor" would be 7 and 2,
respectively, because Tim Burton is well-known as a director, but he acted only
in a few lesser known movies.
The triple scoring task attracted considerable interest, with 52 initial
registrations and 21 teams who submitted a valid run before the deadline. The
winning team achieved an accuracy of 87%, that is, for that fraction of the
triples from the test set (which was revealed only after the deadline) the
difference to the score from the ground truth was at most 2. The best result
for the average difference from the test set scores was 1.50