We present HIGGINS, an engine for high quality Knowl- edge Acquisition (KA), placing special emphasis on its ar- chitecture. The distinguishing characteristic and novelty of HIGGINS lies in its special blending of two engines: An automated Information Extraction (IE) engine, aided by semantic resources, and a game-based, Human Computing engine (HC). We focus on KA from web data and text sources and, in particular, on deriving relationships between enti- ties. As a running application we utilise movie narratives, using which we wish to derive relationships among movie characters

Kumar Kondreddi, S.

Triantafillou, P.

Weikum, G.

English

Enlighten

       Kumar Kondreddi, S., Triantafillou, P., and Weikum, G. (2013) HIGGINS: where knowledge acquisition meets the crowds. In: International Conference on the World Wide Web, 13-17 May 2013, Rio de Janeiro, Brazil.   Copyright © 2013 International World Wide Web Conference Committee  A copy can be downloaded for personal non-commercial research or study, without prior permission or charge   The content must not be changed in any way or reproduced in any format or medium without the formal permission of the copyright holder(s)   When referring to this work, full bibliographic details must be given      http://eprints.gla.ac.uk/80439/       Deposited on:  13 September 2013          Enlighten – Research publications by members of the University of Glasgow http://eprints.gla.ac.uk HIGGINS: where Knowledge Acquisition meets the CrowdsSarath Kumar KondreddiMax Planck Institute for InformaticsSaarbrücken, Germanyskondred@mpi-inf.mpg.dePeter TriantafillouUniversity of PatrasRio-Patra, Greecepeter@ceid.upatras.grGerhard WeikumMax Planck Institute for InformaticsSaarbrücken, Germanyweikum@mpi-inf.mpg.deWe present HIGGINS, an engine for high quality Knowl-edge Acquisition (KA), placing special emphasis on its ar-chitecture. The distinguishing characteristic and novelty ofHIGGINS lies in its special blending of two engines: Anautomated Information Extraction (IE) engine, aided bysemantic resources, and a game-based, Human Computingengine (HC). We focus on KA from web data and text sourcesand, in particular, on deriving relationships between enti-ties. As a running application we utilise movie narratives,using which we wish to derive relationships among moviecharacters.1. THE RATIONALEKA critically relies on IE technology, combining methodsfrom pattern matching, computational linguistics, and statis-tical learning. Open IE methods can derive a wide diversityof relational facts (instances of binary relationships) basedon detecting and analyzing noun phrases (for entities) andverb-centric phrases (for relations), such as: “Vesper”“finallyfalls for”“Bond”. Hence, fully-automated IE plays a key rolewithin HIGGINS.However, there exist fundamental limitations of IE technol-ogy. IE methods can yield noisy or non-sensical relationships(knowledge triples) such as: “Vesper”“certainly has”“Bond”(from the sentence “. . . certainly has . . . sized up”). This oc-curs because fully automated IE generally faces fundamentalobstacles as input sentences often have extremely complexstructures, use of pronouns and other anaphoras, and am-biguous wording. The following snippet (from imdb.com) is adaunting example: “He quickly grabs Vesper and they kissin the stairway entrance to cover themselves.”Crowdsourcing (a.k.a Human Computing (HC)) recentlyhas been employed successfully as a means to help with taskswhere computer-performed, automated solutions are deemedinadequate. Our thesis is that HC is a natural alternative toovercome fundamental limitations of automated IE. It can taphuman intelligence and knowledge to assess candidate factsand to correct errors. Further, humans can readily recognizePermission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.WWW ’13, Rio de Janeiro, BrazilCopyright 20XX ACM X-XXXXX-XX-X/XX/XX ...$10.00.”surreal” relationships, (e.g., occurring in dreams, being imag-inary, contained in intentions, etc.), such as “Vesper Lynd”“pretends to love”“James Bond”, or ”Voldermort” ”plans tokill” ”Harry Potter” and note that they do not correspond totrue facts. Human intelligence can help to resolve pronounsin complex sentences or to identify erroneous paths in thedependency-parsing of natural language. Human knowledgeon special topics such as movies, books, or medicine can addnew facts, that may be entirely missing from the text, oreven help compile/derive facts, e.g., relationships betweencharacters, that are virtually impossible to extract automati-cally, as they may not be explicitly mentioned. Despite thisgreat potential, to our knowledge there have been no pre-vious attempts to employ HC and crowdsourcing platformsfor the di cult KA task of extracting relationships betweenindividual entities.For applications with human experts, (e.g., movies withmovie aficionados, or book lovers, or experts in diseases andmedicine, etc.), one would expect that HC can be nicely castinto game form, thus enticing more users to contribute onthe KA task. However, despite the inherent promises of HCfor KA, humans alone cannot carry this burden. First, thenumber of real experts is typically limited. Second, these ex-perts are not so likely to participate in online games. Hence,inevitably, HC output will contain a wide range from high-quality to highly noisy and incorrect facts. One may thinkthat these HC errors could be compensated by large-scalecrowdsourcing, with redundant Human Intelligence Tasks(HITS) and statistic reasoning over many contributors. How-ever, there is still the issue of the total cost: each HIT maycost a few cents only, but paying for hundreds of thousandsor millions of HITs quickly becomes prohibitive.2. STRATEGY, DESIGNPRINCIPLES, ANDARCHITECTUREHIGINS is a novel architecture that blends HC inputswith machine-centric IE and thus overcomes the limitationsof either a purely HC-based or purely IE-based approachto advanced KA. The key idea is to use automatic IE togenerate questions and candidate answers (for multiple-choicequestions), for a KA game. Our expectation is that this canimprove the quality of user contributions and reduce theoverall cost of crowdsourcing.HIGGINS’ strategy for KA of entity-relationship factsproceeds in two phases:1. IE phase: First we employ automated IE on appropriateWeb corpora, in order to derive candidates for relationGame Logs & Statistics Target entities Target  relations Context  entity Web Corpus Semantic Resources Statistical Language Models Entities & Relational Phrases Yago Wikipedia generate questions & candidate answers Higgins IE engine Higgins HC engine Figure 1: Overview of the Higgins Architectureinstances, with an open set of potential relations. Here,we use a suite of techniques from computational linguis-tics, including dependency parsing (with the StanfordParser) and pronoun resolution (with our own customizedmethod). The resulting triples are usually of very mixedquality, necessitating the second stage.2. HC phase: The sets of candidates from the IE stage andtheir underlying patterns are then used to generate HITsin game form. Abstractly, each HIT presents the userwith a knowledge quad of the form (c, e1, r, e2) where e1and e2 are entities, r is a relation, and c is a cue or textualcontext. One or more of the components e1, r, and e2can be empty slots (variables) to be filled by the user; wemay present a multiple-choice list to the user to pick themissing value. Concretely, the quads are presented in theform of questions, with candidate answers and additionalfree-text fields for entering further values. We focus onthe case where the relationship r is left to be filled, andboth entities and the context are given.The design principles for HIGGINS are as follows:1. The IE engine is tuned to work very aggressively (aimingfor high recall) capturing as many relational patterns aspossible, and we expand this set by specifically designedstatistical (translation) language models (LM’s).2. We use statistics and heuristics to generate interestingquestions about important entities and salient but notobvious relationships. Candidate answers for multiple-choice input are judiciously ranked, using corpus-collectedstatistics, and an additional diversification step serves toavoid boring the user with near-duplicate choices.3. The statistically derived relational phrases for candidateanswers are complemented by phrases from semanticresources, specifically, the WordNet and ConceptNet the-sauri and the ReVerb and PATTY collections of relationalpatterns All this information is combined by a mixturemodel for the LM that generates, expands, and ranksrelationships for a given context.Architecture. Figure 1 depicts the architecture and maincomponents of HIGGINS.3. RESULTS AND CONCLUSIONSHIGGINS contributes a systematic and principled way forgenerating HITs, based on novel statistical language modelsrelying on extensive large corpora, semantic resources, andon an extensive dictionary. The latter serves as a conduitbetween free-form, human expressions for entities and re-lationships and the corresponding system ”proper” namesfor these. The result is a meaningful engagement of humangame players, facilitating the validation and assessment offacts obtained from the IE engine, and the derivation of newknowledge.We have conducted extensive experiments using Wikipediamovie plots and crowds ”in the wild” (with CrowdFlower)and crowds of students in our lab. Our results show that (i)the full HIGGINS architecture can more than double recalland precision of derived facts (achieving recall and precisionnumbers over 70%) compared against a purely statisticalIE engine or a purely semantics-based IE engine. Further,only a small number of game players are required in order toachieve this performance - a fact that translates into reduceddollar-costs for human engagement.

HIGGINS: where knowledge acquisition meets the crowds

Enlighten: Publications

  
 
 
 
 
 
Kumar Kondreddi, S., Triantafillou, P., and Weikum, G. (2013) HIGGINS: 
where knowledge acquisition meets the crowds. In: International 
Conference on the World Wide Web, 13-17 May 2013, Rio de Janeiro, 
Brazil. 
 
 
Copyright © 2013 International World Wide Web Conference Committee 
 
A copy can be downloaded for personal non-commercial research or 
study, without prior permission or charge  
 
The content must not be changed in any way or reproduced in any format 
or medium without the formal permission of the copyright holder(s)  
 
When referring to this work, full bibliographic details must be given 
 
 
 
 
 
http://eprints.gla.ac.uk/80439/ 
 
 
 
 
 
 
Deposited on:  13 September 2013 
 
 
 
 
 
 
 
 
 
Enlighten – Research publications by members of the University of Glasgow 
http://eprints.gla.ac.uk 
HIGGINS: where Knowledge Acquisition meets the Crowds
Sarath Kumar Kondreddi
Max Planck Institute for Informatics
Saarbrücken, Germany
skondred@mpi-inf.mpg.de
Peter Triantafillou
University of Patras
Rio-Patra, Greece
peter@ceid.upatras.gr
Gerhard Weikum
Max Planck Institute for Informatics
Saarbrücken, Germany
weikum@mpi-inf.mpg.de
We present HIGGINS, an engine for high quality Knowl-
edge Acquisition (KA), placing special emphasis on its ar-
chitecture. The distinguishing characteristic and novelty of
HIGGINS lies in its special blending of two engines: An
automated Information Extraction (IE) engine, aided by
semantic resources, and a game-based, Human Computing
engine (HC). We focus on KA from web data and text sources
and, in particular, on deriving relationships between enti-
ties. As a running application we utilise movie narratives,
using which we wish to derive relationships among movie
characters.
1. THE RATIONALE
KA critically relies on IE technology, combining methods
from pattern matching, computational linguistics, and statis-
tical learning. Open IE methods can derive a wide diversity
of relational facts (instances of binary relationships) based
on detecting and analyzing noun phrases (for entities) and
verb-centric phrases (for relations), such as: “Vesper”“finally
falls for”“Bond”. Hence, fully-automated IE plays a key role
within HIGGINS.
However, there exist fundamental limitations of IE technol-
ogy. IE methods can yield noisy or non-sensical relationships
(knowledge triples) such as: “Vesper”“certainly has”“Bond”
(from the sentence “. . . certainly has . . . sized up”). This oc-
curs because fully automated IE generally faces fundamental
obstacles as input sentences often have extremely complex
structures, use of pronouns and other anaphoras, and am-
biguous wording. The following snippet (from imdb.com) is a
daunting example: “He quickly grabs Vesper and they kiss
in the stairway entrance to cover themselves.”
Crowdsourcing (a.k.a Human Computing (HC)) recently
has been employed successfully as a means to help with tasks
where computer-performed, automated solutions are deemed
inadequate. Our thesis is that HC is a natural alternative to
overcome fundamental limitations of automated IE. It can tap
human intelligence and knowledge to assess candidate facts
and to correct errors. Further, humans can readily recognize
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page. To copy otherwise, to
republish, to post on servers or to redistribute to lists, requires prior specific
permission and/or a fee.
WWW ’13, Rio de Janeiro, Brazil
Copyright 20XX ACM X-XXXXX-XX-X/XX/XX ...$10.00.
”surreal” relationships, (e.g., occurring in dreams, being imag-
inary, contained in intentions, etc.), such as “Vesper Lynd”
“pretends to love”“James Bond”, or ”Voldermort” ”plans to
kill” ”Harry Potter” and note that they do not correspond to
true facts. Human intelligence can help to resolve pronouns
in complex sentences or to identify erroneous paths in the
dependency-parsing of natural language. Human knowledge
on special topics such as movies, books, or medicine can add
new facts, that may be entirely missing from the text, or
even help compile/derive facts, e.g., relationships between
characters, that are virtually impossible to extract automati-
cally, as they may not be explicitly mentioned. Despite this
great potential, to our knowledge there have been no pre-
vious attempts to employ HC and crowdsourcing platforms
for the di cult KA task of extracting relationships between
individual entities.
For applications with human experts, (e.g., movies with
movie aficionados, or book lovers, or experts in diseases and
medicine, etc.), one would expect that HC can be nicely cast
into game form, thus enticing more users to contribute on
the KA task. However, despite the inherent promises of HC
for KA, humans alone cannot carry this burden. First, the
number of real experts is typically limited. Second, these ex-
perts are not so likely to participate in online games. Hence,
inevitably, HC output will contain a wide range from high-
quality to highly noisy and incorrect facts. One may think
that these HC errors could be compensated by large-scale
crowdsourcing, with redundant Human Intelligence Tasks
(HITS) and statistic reasoning over many contributors. How-
ever, there is still the issue of the total cost: each HIT may
cost a few cents only, but paying for hundreds of thousands
or millions of HITs quickly becomes prohibitive.
2. STRATEGY, DESIGNPRINCIPLES, AND
ARCHITECTURE
HIGINS is a novel architecture that blends HC inputs
with machine-centric IE and thus overcomes the limitations
of either a purely HC-based or purely IE-based approach
to advanced KA. The key idea is to use automatic IE to
generate questions and candidate answers (for multiple-choice
questions), for a KA game. Our expectation is that this can
improve the quality of user contributions and reduce the
overall cost of crowdsourcing.
HIGGINS’ strategy for KA of entity-relationship facts
proceeds in two phases:
1. IE phase: First we employ automated IE on appropriate
Web corpora, in order to derive candidates for relation
Game Logs & Statistics 
Target 
entities Target  relations 
Context  
entity 
Web Corpus 
Semantic Resources 
Statistical 
Language 
Models 
Entities & 
Relational 
Phrases 
Yago 
Wikipedia 
generate 
questions & 
candidate 
answers 
Higgins IE engine Higgins HC engine 
Figure 1: Overview of the Higgins Architecture
instances, with an open set of potential relations. Here,
we use a suite of techniques from computational linguis-
tics, including dependency parsing (with the Stanford
Parser) and pronoun resolution (with our own customized
method). The resulting triples are usually of very mixed
quality, necessitating the second stage.
2. HC phase: The sets of candidates from the IE stage and
their underlying patterns are then used to generate HITs
in game form. Abstractly, each HIT presents the user
with a knowledge quad of the form (c, e1, r, e2) where e1
and e2 are entities, r is a relation, and c is a cue or textual
context. One or more of the components e1, r, and e2
can be empty slots (variables) to be filled by the user; we
may present a multiple-choice list to the user to pick the
missing value. Concretely, the quads are presented in the
form of questions, with candidate answers and additional
free-text fields for entering further values. We focus on
the case where the relationship r is left to be filled, and
both entities and the context are given.
The design principles for HIGGINS are as follows:
1. The IE engine is tuned to work very aggressively (aiming
for high recall) capturing as many relational patterns as
possible, and we expand this set by specifically designed
statistical (translation) language models (LM’s).
2. We use statistics and heuristics to generate interesting
questions about important entities and salient but not
obvious relationships. Candidate answers for multiple-
choice input are judiciously ranked, using corpus-collected
statistics, and an additional diversification step serves to
avoid boring the user with near-duplicate choices.
3. The statistically derived relational phrases for candidate
answers are complemented by phrases from semantic
resources, specifically, the WordNet and ConceptNet the-
sauri and the ReVerb and PATTY collections of relational
patterns All this information is combined by a mixture
model for the LM that generates, expands, and ranks
relationships for a given context.
Architecture. Figure 1 depicts the architecture and main
components of HIGGINS.
3. RESULTS AND CONCLUSIONS
HIGGINS contributes a systematic and principled way for
generating HITs, based on novel statistical language models
relying on extensive large corpora, semantic resources, and
on an extensive dictionary. The latter serves as a conduit
between free-form, human expressions for entities and re-
lationships and the corresponding system ”proper” names
for these. The result is a meaningful engagement of human
game players, facilitating the validation and assessment of
facts obtained from the IE engine, and the derivation of new
knowledge.
We have conducted extensive experiments using Wikipedia
movie plots and crowds ”in the wild” (with CrowdFlower)
and crowds of students in our lab. Our results show that (i)
the full HIGGINS architecture can more than double recall
and precision of derived facts (achieving recall and precision
numbers over 70%) compared against a purely statistical
IE engine or a purely semantics-based IE engine. Further,
only a small number of game players are required in order to
achieve this performance - a fact that translates into reduced
dollar-costs for human engagement.


http://eprints.gla.ac.uk/80439/1/80439.pdf

HIGGINS: where knowledge acquisition meets the crowds

Abstract

Similar works

Full text

Available Versions

Enlighten

Enlighten: Publications