Applications such as computational fact checking and data-to-text generation exploit the relationship between relational data and natural language text. Despite promising results in these areas, state of the art solutions simply fail in managing “data-ambiguity”, i.e., the case when there are multiple interpretations of the relationship between the textual sentence and the relational data. To tackle this problem, we introduce Pythia, a system that, given a relational table D, generates textual sentences that contain factual ambiguities w.r.t. the data in D. Such sentences can then be used to train target applications in handling data-ambiguity. In this demonstration, we first show how our system generates data ambiguous sentences for a given table in an unsupervised fashion by data profiling and query generation. We then demonstrate how two existing applications benefit from Pythia’s generated sentences, improving the state-of-the-art results. The audience will interact with Pythia by changing input parameters in an interactive fashion, including the upload of their own dataset to see what data ambiguous sentences are generated for it

Badaro Gilbert

Papotti Paolo

Saeed Mohammed

Santoro Donatello

Veltri Enzo

English

Archivio della Ricerca - Università della Basilicata

Pythia: Unsupervised Generation of Ambiguous Textual Claimsfrom Relational DataEnzo Veltrienzo.veltri@unibas.itUniversity of BasilicataDonatello Santorodonatello.santoro@unibas.itUniversity of BasilicataGilbert Badarogilbert.badaro@eurecom.frEURECOMMohammed Saeedmohammed.saeed@eurecom.frEURECOMPaolo Papottipapotti@eurecom.frEURECOMABSTRACTApplications such as computational fact checking and data-to-textgeneration exploit the relationship between relational data andnatural language text. Despite promising results in these areas, stateof the art solutions simply fail in managing “data-ambiguity”, i.e.,the case when there are multiple interpretations of the relationshipbetween the textual sentence and the relational data. To tackle thisproblem, we introduce Pythia, a system that, given a relationaltable𝐷 , generates textual sentences that contain factual ambiguitiesw.r.t. the data in 𝐷 . Such sentences can then be used to train targetapplications in handling data-ambiguity.In this demonstration, we first show how our system generatesdata ambiguous sentences for a given table in an unsupervised fash-ion by data profiling and query generation. We then demonstratehow two existing applications benefit from Pythia’s generatedsentences, improving the state-of-the-art results. The audience willinteract with Pythia by changing input parameters in an interac-tive fashion, including the upload of their own dataset to see whatdata ambiguous sentences are generated for it.ACM Reference Format:Enzo Veltri, Donatello Santoro, Gilbert Badaro, Mohammed Saeed, and PaoloPapotti. 2022. Pythia: Unsupervised Generation of Ambiguous TextualClaims from Relational Data. In Proceedings of the 2022 International Confer-ence on Management of Data (SIGMOD ’22), June 12–17, 2022, Philadelphia,PA, USA.ACM, New York, NY, USA, 4 pages. https://doi.org/10.1145/3514221.35201641 INTRODUCTIONAmbiguity is common in natural language in many forms [7]. Wefocus on the ambiguity of a factual sentence w.r.t. the data in a table,or relation. The problem of data ambiguities in sentences is relevantfor many natural language processing (NLP) applications that userelational data, from computational fact checking [4, 6, 11] to data-to-text generation [8, 10], and question answering in general [2, 9].Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than ACMmust be honored. Abstracting with credit is permitted. To copy otherwise, or republish,to post on servers or to redistribute to lists, requires prior specific permission and/or afee. Request permissions from permissions@acm.org.SIGMOD ’22, June 12–17, 2022, Philadelphia, PA, USA© 2022 Association for Computing Machinery.ACM ISBN 978-1-4503-9249-5/22/06. . . $15.00https://doi.org/10.1145/3514221.3520164Table 1: Dataset 𝐷 . The sentence “Carter has higher shootingthan Smith” is data ambiguous w.r.t. 𝐷 .Player Team FG% 3FG% fouls apps𝑡1 Carter LA 56 47 4 5𝑡2 Smith SF 55 50 4 7𝑡3 Carter SF 60 51 3 3Consider a fact checking application, where the goal is to verifya textual claim against relational data, and the sentence “Carter LAhas higher shooting than Smith SF” to be checked against the dataset𝐷 in Table 1. Even as humans, it is hard to state if the sentenceis true or false w.r.t. the data in 𝐷 . The challenge is due to thetwo different meanings that can be matched to shooting percentage:the claim can refer to statistics about Field Goal (FG%) or 3-pointField Goal (3FG%). We refer to this issue as data ambiguity, i.e.,the existence of more than one unique interpretation of a factualsentence w.r.t. the data for a human reader.Sentences with data ambiguities are not present in existing cor-pora with sentences annotated w.r.t. the data, which leads targetapplications to simply fail in these scenarios. In existing fact check-ing applications, the sentence above leads to a single interpretation,that is incorrect in 50% of the cases. This problem is importantin practice. In the log of an online application for fact checking(https://coronacheck.eurecom.fr/), we found that more than 20 thou-sand claims submitted by the users are data ambiguous, over 80%of the submitted sentences!While traditionally high quality corpora of annotated text sen-tences come from extensive and expensive manual work, in thiswork we demonstrate that by exploiting deep learning over theinput table, we can automatically generate SQL scripts that outputdata ambiguous annotated sentences. SQL enables the efficient gen-eration of a large number of annotated sentences for a given table,which in turn can be used as training data to significantly advancethe target downstream applications.In this demonstration we show Pythia, a system for the auto-matic generation of sentences with data ambiguities, which effec-tively solves the problem of crafting training examples at scale.More precisely, we first introduce our solution based on data profil-ing, query generation, and a novel algorithm to detect ambiguousattributes (Section 2). We then illustrate how the users will engagewith Pythia with interactive parameters, including the ability toload their own datasets and automatically obtain data ambiguoussentences. We will also demonstrate how the generated sentencesare used as training data to improve the coverage and quality oftwo target applications (Section 3).Figure 1: Pythia takes as input any relational table and generates data ambiguous sentences.2 SYSTEM OVERVIEWConsider again the sentence about the higher shooting percentage.The sentence can be obtained from a SQL query over 𝐷 comparingthe FG% and 3FG% attributes for pairs of players.q1: SELECT CONCAT(b1.Player, b1.Team, 'has highershooting than', b2.Player, b2.Team)FROM D b1, D b2WHERE b1.Player <> b2.Player ANDb1.Team <> b2.Team ANDb1.FG% > b2.FG% AND b1.3FG% < b2.3FG%Another example is “Carter has higher 3FG% than Smith”. Thereare two players named ‘Carter’ and the sentence is true for the onein team ‘SF’, but it is false for the one in team ‘LA’. This sentencecan also be generated with a query over 𝐷 .Given a relation, the key idea is to model a sentence as theresult of a query over the data. However, coming up with the queryautomatically from a given relational table is challenging. Consideragain query𝑞1 above. First, the ambiguous attribute pair is not givenwith the table, e.g., FG% and 3FG% in 𝑞1must be identified. Second,an ambiguous sentence requires a label, words in the Select clausethat plausibly refer to the two attributes, e.g., “shooting” in 𝑞1 is thelabel for FG% and 3FG%. To handle these challenges, we restrict thequery generation search space by making use of templates, whichget instantiated with deep learning models predicting the queryparameters from the given table, as detailed next.Data Ambiguity with A-Queries. Sentences can be data ambigu-ous in different ways. For example, the claim 𝑠1: “Carter LA hasbetter shooting than Smith SF” is ambiguous w.r.t. relation 𝐷 inTable 1 because even as humans we cannot state for sure if it refersto regular Field Point statistics or performance for the 3 point range.This is an example of attribute ambiguity over two attributes, butthe same sentence can be ambiguous to more than two attributes.Sentence 𝑠2: “Carter commit 4 fouls” shows a different kind ofdata ambiguity. As rows are identified by two attributes in 𝐷 , areference to only part of the composite key makes it impossible toidentify the right player. This is a row ambiguity over two rows,but the same sentence can be ambiguous to more rows.A sentence can also be ambiguous in the two dimensions, bothover the attributes and the rows. With more dimensions, also thenumber of possible interpretations increases. We therefore dis-tinguish between attribute, row, and full for the structure of theambiguity. We further distinguish between contradictory and uni-form sentences. The first type leads to interpretations that haveopposite factual match w.r.t. the data. Sentences 𝑠1 and 𝑠2 are bothcontradictory, as one interpretation is confirmed from the data andthe other is refuted. For uniform sentences, all interpretations haveequal matching w.r.t. the data, e.g., sentence “Carter has more ap-pearances than Smith” is refuted for both Carter players in 𝐷 . Wefound the distinction between contradictory and uniform sentencescrucial in target applications as they guarantee more control overthe output corpora.An ambiguity query (or a-query) executed over table 𝐷 returns𝑠1, . . . , 𝑠𝑛 sentences with data ambiguity w.r.t. 𝐷 . Consider again𝑞1 defined over 𝐷 , it returns all the pairs of distinct players thatlead to contradictory sentences. The same query can be modifiedto obtain uniform sentences by changing one operator in one ofthe WHERE clause comparing statistics, e.g., change ‘<’ to ‘>’ inthe last clause. Finally, by removing the last two WHERE clauses,the query returns both contradictory and uniform sentences.From the Table to the Sentences. A-queries can be obtainedfrom parametrized SQLA-Query Templates. The parameters identifythe elements of the query and can be assigned to (i) relation andattribute names, (ii) a set of labels, i.e. words with a meaning thatapplies for two or more attributes, (iii) comparison operators.Given an A-Query Template is possible to generate multiple a-queries. Pythia already stores multiple handcrafted SQL templatesthat cover different ambiguity types and can be widely used withdifferent datasets, but it also allows users to add other customtemplates using a predefined syntax.Figure 1 summarizes the overall process. Starting from dataset 𝐷 ,key/FDs profiling methods and our ambiguous attribute discoveryalgorithm identify the metadata to fill up the A-Templates providedin Pythia. All compatible operators are used by default. Metadata,operators, and templates can be refined and extended manuallyby users. Once the template is instantiated, the A-Query is readyfor execution over 𝐷 to obtain the data ambiguous sentences, eachannotated with the relevant cells in 𝐷 .Parameter values to instantiate the templates are automaticallypopulated by Pythia using existing profiling methods and a noveldeep learning method to find attribute ambiguities with the corre-sponding ambiguous words. This method takes as input all attributepairs in a schema and uses a pre-trained model [3] fine-tuned withexamples obtained with weak-supervision over a corpus of tables.We remark our focus on generating sentences with genuine dataambiguity, which is in contrast to other cases where the correctmeaning of the text is clear to a human but an algorithm detects2Figure 2: Pythia configuration of a scenario for a new table.more than one interpretation in the data. For example, “Carter SFhas played 3 times” clearly refers to the player Appearances (apps),but an algorithm could be uncertain between fouls and apps.3 DEMONSTRATIONThe demonstration will be organized in two parts. In the first part,we focus on the data ambiguity sentences from a given table. In thesecond part, we show how the generated corpus can improve threetarget applications.3.1 Dataset GenerationAt the beginning of the demo, the visitor will be able to usepreloaded scenarios or create a new one. Figure 2 shows the con-figuration of a new scenario. First, the visitor uploads a dataset.Then metadata such as primary keys, composite keys and FDs canbe manually provided or obtained with profiling [1]. The core ofFigure 3: Output visualizationwith data ambiguous sentencesand corresponding cells from the table.Pythia is built on top of A-Query Templates. Pythia comes with aset of three generic default templates that were manually craftedfrom the analysis of an online fact checking application’s log. Thethree templates cover more than 90% of user submitted claims withdata ambiguities. More A-Query Templates can be provided bythe visitor. Parameters’ values of the templates are automaticallypopulated by Pythia using the schema metadata and a novel deeplearning module to find ambiguous attributes and the word thatdescribes both attributes (label). Pythia allows the visitor to sub-mit more ambiguous attributes and labels, ultimately submittingfeedback that further improves the underlying ML module.After the scenario’s definition, the visitor will generate ambigu-ous sentences as depicted in Figure 3. To control the type of ambi-guities of the generated dataset, it is possible to select the structuretype, i.e., the A-Query Templates to use, strategies (Contradictory,Uniform True, Uniform False), and the number of output sentencesper a-query. Depending on the dataset and schema size, the numberof generated sentences can vary between thousands and millionseven for medium size datasets. The system outputs for every gen-erated sentence the pair of template and a-query that generated it,together with the data in the table that relates to the sentence. Inthis step, the users can also fine-tune templates and queries inter-actively. The generated corpus can then be exported in CSV, JSON,and application specific formats.33.2 Use casesThe visitor will use seven preloaded datasets. Training data fordownstream applications will be generated using three datasets,the remaining datasets will be used for test data. For each datasetthe visitor will generate sentences with contradictory, uniform trueand uniform false strategies. We denote with 𝑃𝑡 and 𝑃𝑑 the trainingand developing (testing) data generated with Pythia, respectively.Table 2: Impact of 𝑃𝑡 data on table-to-text generation.BLEU ROUGEOriginal (ToTTo [8]) 0.547 0.202ToTTo fine-tuned with Pythia’s output 37.631 0.631ToTTo. ToTTo [8] is a dataset for table-to-text generation,i.e., given a set of cells, the goal is to generate a sentencedescribing the input data. ToTTo contains annotated sentencesthat involve reasoning and comparisons among rows, columns,or cells. The visitor will test a T5 model [3] trained on theoriginal ToTTo for sentence generation. The results will showthat ambiguities are not handled in ToTTo, and thus, the modelsimply fails when it makes predictions over 𝑃𝑑 . The visitor willthen use the 𝑃𝑡 to fine-tune the model. Such fine-tuning step willimprove the performance of the model in terms of BLEU andROUGEmetrics according to the results in Table 2 (higher is better).Table 3: Impact of 𝑃𝑡 data on fact checking (Feverous).CLASS P R F1Ambiguous n.a. n.a. n.a.Feverous Baseline Supports 0.33 1.00 0.49Refutes 0.00 0.00 0.00Ambiguous 0.92 0.85 0.89Feverous Baseline trained on 𝑃𝑡 Supports 0.96 0.95 0.95Refutes 0.88 0.95 0.92Feverous. Feverous [5] is a dataset for fact checking containingtextual claims. Every claim is annotated with the data identifiedby human annotators to Support or Refute a claim. We show theattendant that a model trained with the Feverous baseline pipelinefails on the classification of an ambiguous claim from 𝑃𝑑 , as theAmbiguous label is not present in the original dataset. Training thesame model with 𝑃𝑡 , which contains also examples of ambiguity,radically improves the performance for all classes. Table 3 reportsthe results in terms of precision, recall, and f-measure.CoronaCheck. As a second fact checking application, we willshow the verification of statistical claims related to COVID-19 [6].We incorporate the contributions of Pythia to improve an existingsystem (https://coronacheck.eurecom.fr/). The original system wasnot able to handle attribute ambiguity. Also, row ambiguity causedthe original system to hallucinate with lower classification accu-racy for ambiguous claims. One of the tables used for verifying thestatistical claims consists of the following attributes, among others:country, date, total_confirmed, new_confirmed, total_fatality_rate,total_mortality_rate. From the analysis of the log of claims submit-ted by users, an example of common attribute ambiguity is betweenattributes total_fatality_rate and total_mortality_rate for sentencescontaining “death rate”. Another example of attribute ambiguityis between total_confirmed and new_confirmed when sentencesthat only mention “cases” are verified. Examples of row ambiguityconsist of claims that refer to two records with same location butdifferent timestamps, such as “In France, 10k confirmed cases havebeen reported” (today or yesterday?).Table 4: Impact of 𝑃𝑡 data on fact checking (CoronaCheck).Users’ Accuracy AccuracyClaims original original+ PythiaRow Amb. 44 25/44 36/44Attribute Amb. 9 0/9 6/9No Ambiguity 6 6/6 6/6Total 50 31/50 42/50Pythia enabled us to improve CoronaCheck by automaticallygenerating ambiguity aware training data. The new corpora havebeen used to train new classifiers that allowed the extension ofthe original system. We consider 50 claims from the log of Coro-naCheck and we study the types of ambiguities they include. Asshown in Table 4, 88% include a row or attribute ambiguities. 83%of those ambiguous claims have row ambiguity, and 17% have bothattribute and row ambiguity. The original CoronaCheck fails toclassify claims with ambiguity. However, training the system withthe output of Pythia leads to handling most of the row and at-tribute ambiguity. To show the benefit of using Pythia comparedto the original system, we consider the following example: “June2021: France has 111,244 Covid-19 deaths”. (True for total_deaths,False for new_deaths). The original system returns a false decisionby checking against new_deaths only. With Pythia support, thedecision is True for total_deaths and False otherwise.REFERENCES[1] Ziawasch Abedjan, Lukasz Golab, Felix Naumann, and Thorsten Papenbrock.2018. Data Profiling. Morgan & Claypool Publishers.[2] Wenhu Chen, Ming-Wei Chang, Eva Schlinger, William Yang Wang, andWilliam W. Cohen. 2021. Open Question Answering over Tables and Text. InICLR. OpenReview.net.[3] Colin Raffel et al. 2020. Exploring the Limits of Transfer Learning with a UnifiedText-to-Text Transformer. Journal of Machine Learning Research 21, 140 (2020).[4] Preslav Nakov et al. 2021. Automated Fact-Checking for Assisting Human Fact-Checkers. In IJCAI. 4551–4558.[5] Rami Aly et al. 2021. FEVEROUS: Fact Extraction and VERification Over Unstruc-tured and Structured information. In Thirty-fifth Conference on Neural InformationProcessing Systems (NIPS - Datasets and Benchmarks).[6] Georgios Karagiannis, Mohammed Saeed, Paolo Papotti, and Immanuel Trummer.2020. Scrutinizer: A Mixed-Initiative Approach to Large-Scale, Data-DrivenClaim Verification. Proc. VLDB Endow. 13, 11 (2020), 2508–2521.[7] [n.d.]. 2021. Notes on Ambiguity. https://cs.nyu.edu/~davise/ai/ambiguity.html.(2021).[8] Ankur P. Parikh, Xuezhi Wang, Sebastian Gehrmann, Manaal Faruqui, BhuwanDhingra, Diyi Yang, and Dipanjan Das. 2020. ToTTo: A Controlled Table-To-TextGeneration Dataset. In EMNLP. ACL, 1173–1186.[9] James Thorne, Majid Yazdani, Marzieh Saeidi, Fabrizio Silvestri, Sebastian Riedel,and Alon Y. Levy. 2021. From Natural Language Processing to Neural Databases.Proc. VLDB Endow. 14, 6 (2021), 1033–1039.[10] Immanuel Trummer. 2019. Data Vocalization with CiceroDB. In CIDR.www.cidrdb.org.[11] You Wu, Pankaj K. Agarwal, Chengkai Li, Jun Yang, and Cong Yu. 2017. Compu-tational Fact Checking through Query Perturbations. ACM Trans. Database Syst.42, 1 (2017), 4:1–4:41.4

Pythia: Unsupervised generation of ambiguous textual claims from relational data

https://iris.unibas.it/bitstream/11563/157086/2/42.SIGMOD2022.pdf

Pythia: Unsupervised generation of ambiguous textual claims from relational data

Abstract

Similar works

Full text

Available Versions

Archivio della Ricerca - Università della Basilicata