International audienceIn this paper, we present a bill-type document generator capable of supplying on demand all the mass of documents that a learning system needs. The lack of administrative documents has long been a handicap because of the confidentiality of this type of document. In addition, this generator allowed us to solve the problem of annotations since they are done automatically during the generation and put directly in XML-GEDI form. Then, to show the interest of the generator, we proposed a system of invoice recognition based on graph convolutional neural network. The experiments took place in excellent conditions since we had all the possibilities to vary the classes, the samples in the classes, and their parameters

Belaïd, Abdel

Belaïd, Yolande

Blanchard, Jérôme

INRIA a CCSD electronic archive server

HAL Id: hal-02460869https://hal.inria.fr/hal-02460869Submitted on 30 Jan 2020HAL is a multi-disciplinary open accessarchive for the deposit and dissemination of sci-entific research documents, whether they are pub-lished or not. The documents may come fromteaching and research institutions in France orabroad, or from public or private research centers.L’archive ouverte pluridisciplinaire HAL, estdestinée au dépôt et à la diffusion de documentsscientifiques de niveau recherche, publiés ou non,émanant des établissements d’enseignement et derecherche français ou étrangers, des laboratoirespublics ou privés.Automatic generation of a custom corpora for invoiceanalysis and recognitionJérôme Blanchard, Yolande Belaïd, Abdel BelaïdTo cite this version:Jérôme Blanchard, Yolande Belaïd, Abdel Belaïd. Automatic generation of a custom corpora forinvoice analysis and recognition. ICDAR-WIADAR, Sep 2019, Sydney, Australia. ￿hal-02460869￿Automatic generation of a custom corpora for invoice analysis and recognitionJérôme BlanchardFAIR[&]SMART11 Rempart St Thiébault,57000 Metz, Francejerome.blanchard@fairandsmart.comYolande Belaı̈dUniversité de Lorraine - LORIACampus scientifique54500 Vandoeuvre, Franceyolande.belaid@loria.frAbdel Belaı̈dUniversité de Lorraine - LORIACampus scientifique54500 Vandoeuvre, Franceabdel.belaid@loria.frAbstract—In this paper, we present a bill-type documentgenerator capable of supplying on demand all the mass ofdocuments that a learning system needs. The lack of ad-ministrative documents has long been a handicap because ofthe confidentiality of this type of document. In addition, thisgenerator allowed us to solve the problem of annotations sincethey are done automatically during the generation and putdirectly in XML-GEDI form. Then, to show the interest of thegenerator, we proposed a system of invoice recognition basedon graph convolutional neural network. The experiments tookplace in excellent conditions since we had all the possibilitiesto vary the classes, the samples in the classes, and theirparameters.Keywords-Invoice generator, GEDI format, Graph Convolu-tional Neural NetworkI. INTRODUCTIONIn order to extract named entities from scanned invoice’sdocuments, we are using an Artificial Intelligence (AI) sys-tem that has been trained on annotated invoices taken fromreal life. We faced a common problem in AI training dueto the fact that our invoices corpora was not big enough toproduce relevant results using deep learning neural network.We realized that building a well annotated corpora is verycostly, generates intellectual property problems and is noteasy to customize when you want to add or modify someannotations. We decided to built a software to automate thegeneration of a custom corpora, based on an invoice datamodel made of real data. Our methodology could be used forany other data set that need an automated annotation basedground truth. The generated data (images and annotations)can be used solely or mixed with real life data in order toimprove an existing dataset, especially for AI training.This generation of on-demand documents goes with theidea of wanting to process streaming data without classlimits. However, most existing systems like Intellix [4],ITESOFT [5], [6], smartFIX [7] and others [8], [9] createspecific models limited to the data they have. Due to their de-pendence on seeing the template beforehand, these systemscannot accurately extract information from unseen layoutsof invoices. CloudScan [10] is perhaps the only model sofar which can handle unseen layout invoices quite well. Itis based on classification of word n-grams into entities ofinterest instead of mapping of words to fields.We propose a generic approach to deal with all the entitiesin the invoice in or outside table. We model the wholeinvoice document as a document graph of words, then weclassify each word in the document into classes of interestto extract through a graph convolutional network (GCN) andfinally we group the words of same classes together to obtainthe final entities. The power of our system lies in the graphmodeling and GCN, which takes into account the features ofits neighboring words and their inter-relationships to decidethe class of a word [1].The paper is organized as follows: Section II presents thegenerator and its different possibilities. Section III shows theinvoice analysis approach. Finally, we give some conclusionremarks in IV.II. INVOICE GENERATIONBy analyzing real invoices, determined that it was nearlyalways composed of the same elements, some mandatoryor always present like: a Head zone containing CompanyInformations, Client Informations, Invoice Numbers andDates; a central zone containing Products, Payment Infosand some ID Numbers and a Footer zone containing variousinformations but at least the Totals. Others elements areoptional or their position can vary a lot like: Company Logo,some Company Informations like VAT Number, SIRET,Expedition Type and Shipping Cost, etc.In order to be able to produce any kind of invoice, wefirst defined a model containing all possible invoice data (seeFigure 1). Model structure is critical for the accuracy of thetraining and for the final usage. It must be fine grain enoughto ensure good annotation accuracy. We defined classes thatwe want to extract or classify using the AI process. Wedecided to use an object based model to define all thecomponents of an invoice. All those objects are organizedin a tree where root node is the invoice itself.All objects that compose the data tree have to provide agenerator for its own type. This generator can use severalkinds of data source: random generation, data source basedgeneration, set of values, set of regular expressions. Thus,generation of an invoice data results in crawling the treeto generate each node using its own generation implemen-tation. Container nodes are responsible for propagating theFigure 1: Invoice Model Schema.generation to their children.A. Context generationFor each invoice generation, some definitive choices aremade during tree generation and those choices have impacton the rest of the generation: for example, the generationlanguage, currency or date format. As soon as the currencyis chosen it has to be taken into consideration for manyelements of the tree. To enforce this coherency across allnodes of the invoice, we are using a dedicated structure, theGeneration Context, which is propagated to all nodes duringthe tree generation. Some context choices can have impactwhere it seems it should not: the choice of the companylogo can be directly linked with the country previously setin the context avoiding inconsistency on logo slogan that arenot in the same language than the rest of the invoice. Thus,the generators of some model nodes can be increased withnewly discovered context dependencies by adding heuristicsin this node generator based on context values.B. Data generation strategiesThe first strategy for generation is to use a set ofregular expressions to produce data. This works wellfor invoice numbers or invoice dates. Those nodesare using predefined regular expression based on reallife invoice data observation. Then, we use a library(https://github.com/mifmif/Generex) that is able to randomlygenerate any value in the whole regular expression domainpossibilities. The regular expressions can also be conditionedwith some generation context’s choices, some are dedicatedto specific language or country for example.A second strategy for generation is to use real life opendatabases. For example, addresses are randomly generatedusing public base for all countries that we want to support.This can create more realistic invoice using, for a Belgiuminvoice, real Belgium addresses. We also use, for labels orsome textual parts of an invoice, predefined set of all valuesthat we already observed on real invoices. Those samplesare ordered by language and/or by country.Another generation strategy is to include some pseudorandom in some heuristic choices, sometimes with a statis-tical distribution of results. For example, the client’s billingaddress is often the same than the client’s shipping addressso we used a generation proportionality of 80% the sameand 20% another adress. In the same way, command date,shipping date and billing date are generated using someheuristics that avoid a shipping date prior or too far ofthe command date. We also use some heuristics for thegeneration of the products included in the invoice: dependingon the product category, the product random quantity is notthe same (it is reasonably possible to buy 10 ink cartridgebut not to buy 10 microwaves).C. Global generation processThe global generation process consists of fixing or ran-domizing some context elements (country, language) and ofstarting an iterative process to be able to generate as muchas invoice data we need.The generation strategies as much as the data sources(products, addresses, companies logos, etc.) and the regularexpressions for elements generation can be easily expendedand included in a new generation cycle by improving nodegenerators heuristics or data sources.D. Graphical representation of the invoice modelOnce the data model is generated, we can build agraphical representation of that invoice. We decided togenerate invoices in PDF that we can export in an imageformat (TIFF or JPG). The graphical generation consistsin drawing all pieces of invoice data model onto a singlepage using a dedicated Layout. The layout is responsibleof placing model data on the sheet but also to producethe annotations corresponding to the graphical positionand size of all elements; image is generated in a PDFflow and annotations in an XML flow (in the GEDI for-mat (https://sourceforge.net/projects/gedigroundtruth/)). Asfor the invoice model, layout elements are built in a treestructure.E. Graphical elements generationWe created a set of graphical components with basicbehavior of typical graphical toolkits but also with theannotation process built-in. We started by implementingthe smallest object of our kit: the Text Box. This basiccomponent consist of a part of text which can be drawninto the picture. More complexity is added to this simplebehavior like, the desired font, the padding, the box size(allowing to split text if end of line is reached), color,background color, line height. More than only graphicalconsideration, we give also annotation consideration like theentity name that corresponds to this text.F. Annotation generationThe annotation generation consists of creating an XMLelement based on GEDI schema. The annotation is includedFigure 2: Annotation generation schema.in an XML stream and include position and box size of thetext zone but also the text itself and the annotation classthat we want to learn. Layout is also responsible, by usinginvoice model data, to fix annotation classes.G. Layout for invoicesIn order to produce invoices with complex layouts, wehave developed graphical objects that acts as containersfor Text Box. Thus, we have Horizontal Container, VerticalContainer, Table Row, etc. We also defined some specificelements for graphical purpose only like Graphic Lines,Images, Borders, etc.H. Fixed based layoutFixed based layout are a simple static based combinationof existing elements to produce a “clone” of a real invoiceand to use it for the generation of many invoices with variousdata. This allows to have a ground truth very close to thereality but with a bigger number of invoices. Nevertheless,we need to have many existing samples in order to have avaried ground truth and layout integration can be costly inthis case.I. Generic layoutWe decided also to work on generic layout by definingparts of an invoice that could produce random behavior.We defined Global Zones (Header, Product Table, ProductFooter, Global Footer). Header Zone will be divided infour regions allowing to place randomly global components:Company Information, Order information, Billing Address,Shipping Address. The rendering of all components andregions will be randomized using specific graphical repre-sentation (background color, font size, border, etc.). This willallow to have mixed rendering layout based on a rich graph-ical library. The Table 3 shows the different element locationand neighbor relationships, while the Table I highlightsthe elements appearing in the different blocks, indicatingwhether they are mandatory or optional relationshipsThe Figure 4 shows an example of generated invoices, onefrom actual invoice (a) and one generated randomly (b).Figure 3: Main table elements and their location and inter-relationships.Figure 4: Examples of generated invoices: (a) from actualinvoice by varying the information inside the blocks, and (b)randomly generated.III. INVOICE ANALYSISThe global system has been published in [1]. We are goingto just summarize here the main idea of the system. Theimage is run through an OCR engine (Tesseract). From thisoutput, we only take word zones and ignore all other zonessuch as graphic line, area, image, etc., because the higherup zones are composed of word zones and accumulate moreocr segmentation errors. Then, features are calculated foreach word. These words are used to model the completedocument as a graph with words as nodes and edges depict-ing neighborhood relationships. This document graph is fedto a graph node classifier which classifies each word intoclasses of interest. Finally words belonging to same classesare grouped together following left to right ordering to formentities.Text feature calculation is basically converting the wordtext into a meaningful vector representation. For this task,we use BPEmb [13]. We obtain a 317 dimensional featurevector of every word in the document. The last 17 elementsare reserved for the existence of keywords like date, zipcode,city, etc.The whole document is modeled as a graph with wordsTable I: Intra-element constraintsType Delivery / Billing Invoice information Order Information Product table Summary table / totalsLast name / First Name Company Name Invoice number Order number QuantityMandatory No. and street No. and street Date NameZipCode, City name ZipCode, City name Unit Price excl taxTotal Price excl taxCountry Country Location Order date Product referenceTel number Tel number Client number Client number Unit price incl tax@mail Fax N° Company Name Payment mode DiscountOptional @mail Payment Total Final unit price incl tax@site Sending date Eco participationSending mode VATParcel numberas nodes and edges denoting nearest neighbors of a word in4 major directions. This is different from, for example, thesystem of ITESOFT [5] using a star graph for each importantword. For modeling, we were inspired by the system of[14] who used a Graph Convolutional Networks (GCN). Weused more than 3000 images generated by our generator,by selecting those which seemed to us quite representativeof the sought variations. In current system, we scan eachimage in 300 dpi. The invoices are annotated at word levelby providing each word a ground truth class.IV. CONCLUSIONIn this paper, we presented two ways to automaticallygenerate invoices, inspired by actual invoices to which weapplied random variations of a few elements. The otheris completely free, by simply considering the elementsthat must appear in the invoice, but by introducing interand intra-element relations. The organization scheme of theelements in the page was complex because the constraintsare numerous. We opted for reasonable constraints leading tothe production of fairly realistic bills. This work has allowedus to generate enough well annotated invoice samples toallow learning and recognition in good conditions.REFERENCES[1] D. Lohani, A. Belaı̈d and Y. Belaı̈d. An Invoice ReadingSystem using a Graph Convolutional Network. IWRR, Dec2018, PERTH, Australia. 2018.[2] B. Klein, S. Agne and A. Dengel: Results of a study on invoice-reading systems in Germany. DAS, pp. 451-462, 2004.[3] D. Nadeau and S. Sekine: A survey of named entity recognitionand classification. Lingvisticae Investigationes, 30, pp. 3-26,2007.[4] D. Schuster and K. Muthmann and D. Esser and A. Schill andM. Berger and C. Weidling and K. Aliyev and A. Hofmeier:Intellix-end-user trained information extraction for documentarchiving. ICDAR, 2013, pp. 101-105.[5] M. Rusinol, T. Benkhelfallah and V. Poulain d’Andecy: Fieldextraction from administrative documents by incremental struc-tural templates. ICDAR, 2013, pp. 1100-1104.[6] H. Hamza, Y. Belaı̈d and A. Belaı̈d: A case-based reasoningapproach for invoice structure extraction. ICDAR, 2007, pp.327-331.[7] A. Dengel and B. Klein: smartfix: A requirements-drivensystem for document analysis and understanding. DAS, 2002,pp. 433-444.[8] F. Cesarini, E. Francesconi, M. Gori and G. Soda: Analysisand understanding of multi-class invoices. ICDAR, 2003, pp.102-114.[9] V. P. d’Andecy, E. Hartman and M. Rusiñol: Field extractionby hybrid incremental and a priori structural templates. DAS,2018, pp. 251-256.[10] R. B. Palm, O. Winther and F. Laws: Cloudscan-aconfiguration-free invoice analysis system using recurrent neu-ral networks. ICDAR 2017, pp. 406-413.[11] T. Kasar, T. K. Bhowmik and A. Belaı̈d: Table informationextraction and structure recognition using query patterns. IC-DAR, 2015, pp. 1086-1090.[12] R. Sennrich, B. Haddow and A. Birch: Neural machinetranslation of rare words with subword units. 54th AnnualMeeting of the Association for Computational Linguistics,2016, pp. 1715–1725[13] B. Heinzerling and M. Strube: BPEmb: Tokenization-free Pre-trained Subword Embeddings in 275 Languages. In: LREC2018, pp. 2989-2993.[14] T. N. Kipf and M. Welling: Semi-supervised classificationwith graph convolutional networks. ICLR, 2017, pp. 1-14.[15] M. Defferrard, X. Bresson and P. Vandergheynst: Convolu-tional neural networks on graphs with fast localized spectralfiltering. In: Advances in Neural Information Processing Sys-tems, 2016, pp. 3844-3852[16] R. Smith: An overview of the tesseract ocr engine. In ICDAR2007, pp. 629-633.[17] D. P. Kingma and J. Ba: Adam: A method for stochasticoptimization. arXiv preprint arXiv:1412.6980, 2014.

Automatic generation of a custom corpora for invoice analysis and recognition

https://hal.inria.fr/hal-02460869/document

Automatic generation of a custom corpora for invoice analysis and recognition

Abstract

Similar works

Full text

Available Versions

INRIA a CCSD electronic archive server