Arnaout, H.

Ghosh, S.

Razniewski, S.

Suchanek, F.

English

MPG.PuRe

On the Limits of Machine Knowledge:Completeness, Recall and Negationin Web-scale Knowledge BasesSimon Razniewski1, Hiba Arnaout1, Shrestha Ghosh1, Fabian Suchanek21Max Planck Institute for Informatics2Institut Polytechnique de Paris{srazniew,harnaout,ghoshs}@mpi-inf.mpg.de,fabian@suchanek.nameABSTRACTGeneral-purpose knowledge bases (KBs) are an important compo-nent of several data-driven applications. Pragmatically constructedfrom available web sources, these KBs are far from complete, whichposes a set of challenges in curation as well as consumption.In this tutorial we discuss how completeness, recall and nega-tion in DBs and KBs can be represented, extracted, and inferred.We proceed in 5 parts: (i) We introduce the logical foundationsof knowledge representation and querying under partial closed-world semantics. (ii) We show how information about recall can beidentified in KBs and in text, and (iii) how it can be estimated viastatistical patterns. (iv) We show how interesting negative state-ments can be identified, and (v) how recall can be targeted in acomparative notion.PVLDB Reference Format:Simon Razniewski, Hiba Arnaout, Shrestha Ghosh, Fabian Suchanek. Onthe Limits of Machine Knowledge:Completeness, Recall and Negationin Web-scale Knowledge Bases. PVLDB, 14(12): 3175 - 3177, 2021.doi:10.14778/3476311.34764011 MOTIVATIONWeb-scale knowledge bases (KBs) likeWikidata [32], DBpedia [2] orYago [30] are used in applications ranging from question-answeringto personal assistants. Pragmatically constructed from web re-sources, they focus on representing positive knowledge, i.e., state-ments that are true. They do not store negative statements. Theyare also incomplete, i.e., they do not contain all true statements inthe domain of interest. This means that if a statement is not in theKB, we do not know whether it is false in the real world or justabsent.This poses major challenges for the curation and applicationof KBs: First, KB curators may want to know where the KB isincomplete, so that they can prioritize their completion efforts. Thisholds in particular for KBs such as NELL [4], which want to auto-complete themselves. Second, KB applications need to know wherethe data is incomplete, so as to alert end users of quality issues.For example, a query for “the largest city in Japan” may return theThis work is licensed under the Creative Commons BY-NC-ND 4.0 InternationalLicense. Visit https://creativecommons.org/licenses/by-nc-nd/4.0/ to view a copy ofthis license. For any use beyond those covered by this license, obtain permission byemailing info@vldb.org. Copyright is held by the owner/author(s). Publication rightslicensed to the VLDB Endowment.Proceedings of the VLDB Endowment, Vol. 14, No. 12 ISSN 2150-8097.doi:10.14778/3476311.3476401wrong answer if Tokyo happens to be absent in the KB. Similarly,a KB that is used for question-answering in an enterprise settingneeds awareness of when a question surpasses its knowledge [22].This holds in particular for boolean questions, such as “Did Airbusproduce this plane”, where a “no” could come simply from missinginformation. Finally, a comprehensive answer to the request tosummarize the salient information about an entity should containalso salient facts that do not apply.Traditionally, KB construction and curation has focused on theaspects of provenance and accuracy [21, 33]. Yet recent years haveseen a maturing of formalisms for describing recall and negativeknowledge [1, 5, 18], as well as a rise of statistical and text-basedmethods for estimating recall [3, 7, 12–14, 17, 24, 29] and derivingnegative statements [1, 13]. Systematizing these approaches, andmaking them accessible to the general database audience, is thetopic of this tutorial. The tutorial will be of interest to theoreticiansas well as practitioners. It will inform the audience about the latestadvances in completeness assessment and negation, and equip themwith a repertoire of methodologies to better represent and assessthe recall of specific datasets.2 DETAILED DESCRIPTIONLength: Half day (3 hours)Outline(1) Introduction (10 minutes): We outline the gaps in exist-ing web-scale KBs [26], and motivate the importance of cap-turing information about completeness, recall and salientnegations in KBs with several use cases.(2) Logical foundations (20 minutes): We outline the logi-cal framework in which KBs operate, the open world as-sumption, the partial-closed world assumption (PCWA) [5,6, 8], the implications this framework has for query an-swering [25], as well as the formal semantics of complete-ness assertions, how they can be practically represented indatabases [15, 20, 28] and knowledge base [5].(3) Cardinalities from KBs and text as ground truth (45min):We explain the challenges in obtaining human groundtruth, and the role that relation cardinality information playsin recall assessment. In particular, we show how existingcardinality information inside KBs can be identified and usedto assess completeness [10], as well as how this informationcan be extracted from natural language documents [18].3175(4) Predictive recall assessment (45 min): We present threelines of approaches: (i) Supervised machine learning to iden-tify complete or incomplete regions of KBs [7], (ii) unsuper-vised statistical techniques, such as species sampling tech-niques from ecology [17, 31], density-based estimators [14]and statistical invariants about number distributions [29],(iii) linguistic theories about human conversations, whichtell in which contexts information is likely complete, and inwhich contexts it is not [24].(5) Identifying salient negations (30 min): We show whyexplicit negations are needed in open-world settings, andhow they can be automatically mined by locally inferringclosed-world topics from reference peer entities [1]. We con-trast this approach with text extraction based on searchengine query logs or Wikipedia text revisions [13], as well aslanguage-modelling-based extraction [27], and outline openissues in terms of ontology modelling.(6) Relative recall (30 min): We finally relax stricter absolutenotions of recall, and show how recall can be measuredin a relative manner, especially via extrinsic use cases likequestion answering and entity summarization [12, 23], bycomparison with open information extraction or externalreference resources [9, 19], and by comparison with othercomparable entities inside the KB [3, 11, 16].Practical sessions Each part will have a practical component,where participants familiarize themselves with sample tools forthe respective topic (for example, data quality tools like Recoin(https://www.wikidata.org/wiki/Wikidata:Recoin), ProWD (https://prowd.id),Wikinegata (https://d5demos.mpi-inf.mpg.de/negation), CounQER(https://counqer.mpi-inf.mpg.de/spo)), Cool-WD (cool-wd.inf.unibz.it).3 TUTORIAL MATERIALAttendees will be provided with slides, as well as link collections torelevant tools, code repositories and datasets, under a permissivelicense.4 AUDIENCEWe expect a broad audience for this tutorial that can be dividedinto three groups: (1) knowledge acquisition engineers interestedin tracking quality, and focusing extraction efforts; (2) applicationengineers interested in understanding knowledge quality; (3) formalsemantics researchers interested in modeling knowledge beyondthe open-world assumption.Correspondingly, the tutorial will be organized so that partici-pants with varying backgrounds can benefit. Only a basic familiaritywith general database modelling (e.g., ER data model) is required,beyond that, all important concepts will be introduced formally, aswell as by examples.5 PRESENTERSSimon Razniewski (primary contact) - Max Planck Institutefor Informatics, srazniew@mpi-inf.mpg.de, http://simonrazniewski.com. Simon Razniewski is a senior researcher at the Max PlanckInstitute for Informatics in Saarbrücken, Germany, where he headsthe Knowledge Base Construction and Quality research area. Hehas been a driver behind recent research around completeness,recall and negation in KBs, and has ample didactical experiencefrom university teaching, and conference tutorials on commonsenseknowledge (e.g., AAAI’21, WSDM’21).HibaArnaout -Max Planck Institute for Informatics, harnaout@mpi-inf.mpg.de, http://people.mpi-inf.mpg.de/~harnaout/. Hiba Arnaoutis a PhD student at the Max Planck Insitute for Informatics, inSaarbrücken, Germany. Her primary academic interests includeKnowledge Base quality and negation in Knowledge Bases. Hibahas authored an award-wining paper on interesting negative state-ments in Knowledge Bases, published at AKBC’20, and presentedat ISWC’20, as significant Web Semantic related work in a sister-conference.ShresthaGhosh -Max Planck Institute for Informatics, ghoshs@mpi-inf.mpg.de, https://people.mpi-inf.mpg.de/~ghoshs/. ShresthaGhoshis a PhD student at the Max Planck Institute for Informatics inSaarbrücken, Germany. Her primary research is on exploring setinformation in Knowledge Bases and text to improve recall oncount queries. She has published her work in JWS’20, ESWC’20 andpresented at the doctoral consortium track of ISWC’20.Fabian Suchanek - Institut Polytechnique de Paris,fabian@suchanek.name, http://suchanek.name. Fabian Suchanekis a professor at Institut Polytechnique de Paris in France, andthe creator of the YAGO knowledge base. He has ample didacticalexperience, and authored more than 100 publications in the areaof knowledge bases (with 12k citations in total), several of thesespecifically concerning completeness.6 RELATED EVENTSThis tutorial represents completely novel material and has notbeen presented anywhere so far. Modified versions of the proposalhave also been submitted to two complementary communities,semantic web (ISWC’21), and computational logics (KR’21), butthese conferences take place several months after VLDB, and thecommunities have different foci.We are not aware of any similar tutorials by other presenters inthe past.7 REQUIREMENTSA a live internet connection is beneficial to try out examples on web-deployed KBs, but we will show them in the presentation screentoo, so recordings can also be watched offline.REFERENCES[1] H. Arnaout, S. Razniewski, and G. Weikum. Enriching knowledge bases withinteresting negative statements. In AKBC, 2020.[2] S. Auer, C. Bizer, G. Kobilarov, J. Lehmann, R. Cyganiak, and Z. G. Ives. DBpedia:A nucleus for a web of open data. ISWC, 2007.[3] V. Balaraman, S. Razniewski, and W. Nutt. Recoin: relative completeness inWikidata. InWiki workshop at WWW, 2018.[4] A. Carlson, J. Betteridge, B. Kisiel, B. Settles, E. Hruschka, and T. Mitchell. Towardan architecture for never-ending language learning. In AAAI, 2010.[5] F. Darari, W. Nutt, G. Pirro, and S. Razniewski. Completeness statements aboutRDF data sources and their use for query answering. In ISWC, 2013.[6] M. Denecker, Á. Cortés-Calabuig, M. Bruynooghes, and O. Arieli. Towards alogical reconstruction of a theory for locally closed databases. TODS, 2008.[7] L. Galárraga, S. Razniewski, A. Amarilli, and F. M. Suchanek. Predicting com-pleteness in knowledge bases. In WSDM, 2017.[8] L. A. Galárraga, C. Teflioudi, K. Hose, and F. Suchanek. Amie: association rulemining under incomplete evidence in ontological knowledge bases. InWWW,2013.3176[9] K. Gashteovski, R. Gemulla, B. Kotnis, S. Hertling, and C. Meilicke. On aligningOpenIE extractions with knowledge bases: A case study. In Eval4NLP, 2020.[10] S. Ghosh, S. Razniewski, and G. Weikum. Uncovering hidden semantics of setinformation in knowledge bases. JWS, 2020.[11] L. C. Gleim, R. Schimassek, D. Hüser, M. Peters, C. Krämer, M. Cochez, andS. Decker. Schematree: Maximum-likelihood property recommendation forwikidata. In ESWC, 2020.[12] A. Hopkinson, A. Gurdasani, D. Palfrey, and A. Mittal. Demand-weighted com-pleteness prediction for a knowledge base. In NAACL, 2018.[13] G. Karagiannis, I. Trummer, S. Jo, S. Khandelwal, X. Wang, and C. Yu. Mining an“anti-knowledge base” fromWikipedia updates with applications to fact checkingand beyond. VLDB, 2019.[14] J. Lajus and F. M. Suchanek. Are all people married? determining obligatoryattributes in knowledge bases. In WWW, 2018.[15] W. Lang, R. V. Nehme, E. Robinson, and J. F. Naughton. Partial results in databasesystems. In SIGMOD, 2014.[16] M. Luggen, J. Audiffren, D. Difallah, and P. Cudré-Mauroux. Wiki2prop: Amultimodal approach for predicting wikidata properties from wikipedia. InWWW, 2021.[17] M. Luggen, D. Difallah, C. Sarasua, G. Demartini, and P. Cudré-Mauroux. Non-parametric class completeness estimators for collaborative knowledge graphs -the case of Wikidata. In ISWC, 2019.[18] P. Mirza, S. Razniewski, F. Darari, and G. Weikum. Enriching knowledge baseswith counting quantifiers. In ISWC, 2018.[19] B. D. Mishra, N. Tandon, and P. Clark. Domain-targeted, high precision knowl-edge extraction. TACL, 2017.[20] A. Motro. Integrity= validity+completeness. TODS, 1989.[21] H. Paulheim. Knowledge graph refinement: A survey of approaches and evalua-tion methods. SWJ, 2017.[22] P. Rajpurkar, R. Jia, and P. Liang. Know what you don’t know: Unanswerablequestions for squad. In ACL, 2018.[23] S. Razniewski and P. Das. Structured knowledge: Have we made progress? anextrinsic study of KB coverage over 19 years. In CIKM, 2020.[24] S. Razniewski, N. Jain, P. Mirza, and G. Weikum. Coverage of informationextraction from sentences and paragraphs. In EMNLP, 2019.[25] R. Reiter. On closed world data bases. In Readings in artificial intelligence. 1981.[26] D. Ringler and H. Paulheim. One knowledge graph to rule them all? analyzingthe differences between DBpedia, YAGO, Wikidata & co. In KI, 2017.[27] T. Safavi and D. Koutra. Generating negative commonsense knowledge. KR2ML,2020.[28] Z. Shang, W. Brackenbury, A. J. Elmore, and M. J. Franklin. Cyadb: a databasethat covers your ask. VLDB, 2018.[29] A. Soulet, A. Giacometti, B. Markhoff, and F. M. Suchanek. Representativenessof knowledge bases with the generalized Benford’s law. In ISWC, 2018.[30] F. M. Suchanek, G. Kasneci, and G. Weikum. Yago: a core of semantic knowledge.In WWW, 2007.[31] B. Trushkowsky, T. Kraska, M. J. Franklin, and P. Sarkar. Crowdsourced enumer-ation queries. In ICDE, 2013.[32] D. Vrandečić and M. Krötzsch. Wikidata: A free collaborative knowledge base.CACM, 2014.[33] A. Zaveri, A. Rula, A. Maurino, R. Pietrobon, J. Lehmann, and S. Auer. Qualityassessment for linked data: A survey. SWJ, 2016.3177

On the Limits of Machine Knowledge: {C}ompleteness, Recall and Negation in Web-scale Knowledge Bases

http://hdl.handle.net/21.11116/0000-0009-6546-7

On the Limits of Machine Knowledge: {C}ompleteness, Recall and Negation in Web-scale Knowledge Bases

Abstract

Similar works

Full text

Available Versions

MPG.PuRe