Autonomous Consolidation of Heterogeneous Record-Structured HTML Data in Chameleon

While progress has been made in querying digital information contained in XML and HTML documents, success in retrieving information from the so called hidden Web (data behind Web forms) has been modest. There has been a nascent trend of developing autonomous tools for extracting information from the hidden Web. Automatic tools for ontology generation, wrapper generation, Weborm querying, response gathering, etc., have been reported in recent research. This thesis presents a system called Chameleon for automatic querying of and response gathering from the hidden Web. The approach to response gathering is based on automatic table structure identification, since most information repositories of the hidden Web are structured databases, and so the information returned in response to a query will have regularities. Information extraction from the identified record structures is performed based on domain knowledge corresponding to the domain specified in a query. So called domain plug-ins are used to make the dynamically generated wrappers domain-specific, rather than conventionally used document-specific

Autonomous Consolidation of Heterogeneous Record-Structured HTML Data in Chameleon

While progress has been made in querying digital information contained in XML and HTML documents, success in retrieving information from the so called hidden Web (data behind Web forms) has been modest. There has been a nascent trend of developing autonomous tools for extracting information from the hidden Web. Automatic tools for ontology generation, wrapper generation, Weborm querying, response gathering, etc., have been reported in recent research. This thesis presents a system called Chameleon for automatic querying of and response gathering from the hidden Web. The approach to response gathering is based on automatic table structure identification, since most information repositories of the hidden Web are structured databases, and so the information returned in response to a query will have regularities. Information extraction from the identified record structures is performed based on domain knowledge corresponding to the domain specified in a query. So called domain plug-ins are used to make the dynamically generated wrappers domain-specific, rather than conventionally used document-specific

A Novel Approach to Data Extraction on Hyperlinked Webpages

The World Wide Web has an enormous amount of useful data presented as HTML tables. These tables are often linked to other web pages, providing further detailed information to certain attribute values. Extracting schema of such relational tables is a challenge due to the non-existence of a standard format and a lack of published algorithms. We downloaded 15,000 web pages using our in-house developed web-crawler, from various web sites. Tables from the HTML code were extracted and table rows were labeled with appropriate class labels. Conditional random fields (CRF) were used for the classification of table rows, and a nondeterministic finite automaton (NFA) algorithm was designed to identify simple, complex, hyperlinked, or non-linked tables. A simple schema for non-linked tables was extracted and for the linked-tables, relational schema in the form of primary and foreign-keys (PK and FK) were developed. Child tables were concatenated with the parent table’s attribute value (PK), serving as foreign keys (FKs). Resultantly, these tables could assist with performing better and stronger queries using the join operation. A manual checking of the linked web table results revealed a 99% precision and 68% recall values. Our 15,000-strong downloadable corpus and a novel algorithm will provide the basis for further research in this field.publishedVersio

Towards a Large Corpus of Richly Annotated Web Tables for Knowledge Base Population

Ell B, Hakimov S, Braukmann P, et al. Towards a Large Corpus of Richly Annotated Web Tables for Knowledge Base Population. Presented at the Fifth international workshop on Linked Data for Information Extraction (LD4IE) at ISWC2017, Vienna.Web Table Understanding in the context of Knowledge Base Population and the Semantic Web is the task of i) linking the content of tables retrieved from the Web to an RDF knowledge base, ii) of building hypotheses about the tables' structures and contents, iii) of extracting novel information from these tables, and iv) of adding this new information to a knowledge base. Knowledge Base Population has gained more and more interest in the last years due to the increased demand in large knowledge graphs which became relevant for Artificial Intelligence applications such as Question Answering and Semantic Search. In this paper we describe a set of basic tasks which are relevant for Web Table Understanding in the mentioned context. These tasks incrementally enrich a table with hypotheses about the table's content. In doing so, in the case of multiple interpretations, selecting one interpretation and thus deciding against other interpretations is avoided as much as possible. By postponing these decision, we enable table understanding approaches to decide by themselves, thus increasing the usability of the annotated table data. We present statistics from analyzing and annotating 1.000.000 tables from the Web Table Corpus 2015 and make this dataset as well as our code available online

Ekstraksi Tabel HTML Bentuk Column-Row Wise ke dalam Basis Data

Pada halaman web, tabel adalah bagian penting dari masalah yang dijelaskan dalam sebuah artikel. Tabel yang terdapat pada halaman web berbeda dari tabel dalam basis data. Tabel di halaman web cenderung tidak memiliki aturan atau bentuk standar. Salah satu bentuk tabel yang tidak standar pada halaman web adalah column-row wise. Penelitian ini menawarkan pendekatan untuk mengekstraksi isi tabel sedemikian sehingga arti dari keterkaitan antara dua atribut dan data dalam tabel column-row wise tidak hilang. Data yang diekstrak disimpan ke dalam basis data yang membentuk tiga tabel, yaitu tabel yang menyimpan atribut pertama, tabel yang menyimpan atribut kedua, dan tabel yang menyimpan atribut pertama, kedua, dan data dari atribut pertama dan kedua. Penelitian ini menghasilkan sebuah algoritma untuk mengekstrak data dari tabel yang berbentuk column-row wise pada sebuah halaman web. Algoritma yang dihasilkan dari penelitian ini diharapkan dapat diimplementasikan dalam berbagai bahasa pemrograman. Untuk pengujian, algoritma telah diimplementasikan dengan Bahasa pemrograman Python dan berhasil melakukan ekstraksi tabel dan menyimpannya dalam basis data.   Abstract  Tables are an important part of a web page. The table contains tabulations of data or information that you want to convey from the web page. This data tabulation can be used for comparisons with similar tables or as a trigger for action. However, tables on web pages are independent of webpage makers. There is no standard form or layout for a table on a web page. One of the table layouts on a web page is column-row wise. This study offers an approach for extracting table contents such that the meaning of the linkage between two attributes and a data in the column-row wise table is not disappeared. The extracted data is stored into a database that forms three tables, ie the table that stores the first attribute, the table that stores the second attribute, and the table that stores the first, second, and second attributes of the two attributes. Output of this research is an algorithm to extract data of column-row wise table in a web page. The algorithm generated from this research is expected to be implemented in various programming languages. For testing, the algorithm is implemented in Python and success to extract table and save the data into database. Cyclomatic complexity number of the proposed algorithm is 12. This means that the complexity of the proposed algorithm is still high

BlogForever D2.6: Data Extraction Methodology

This report outlines an inquiry into the area of web data extraction, conducted within the context of blog preservation. The report reviews theoretical advances and practical developments for implementing data extraction. The inquiry is extended through an experiment that demonstrates the effectiveness and feasibility of implementing some of the suggested approaches. More specifically, the report discusses an approach based on unsupervised machine learning that employs the RSS feeds and HTML representations of blogs. It outlines the possibilities of extracting semantics available in blogs and demonstrates the benefits of exploiting available standards such as microformats and microdata. The report proceeds to propose a methodology for extracting and processing blog data to further inform the design and development of the BlogForever platform