Web Data Extraction, Applications and Techniques: A Survey

Web Data Extraction is an important problem that has been studied by means of different scientific tools and in a broad range of applications. Many approaches to extracting data from the Web have been designed to solve specific problems and operate in ad-hoc domains. Other approaches, instead, heavily reuse techniques and algorithms developed in the field of Information Extraction. This survey aims at providing a structured and comprehensive overview of the literature in the field of Web Data Extraction. We provided a simple classification framework in which existing Web Data Extraction applications are grouped into two main classes, namely applications at the Enterprise level and at the Social Web level. At the Enterprise level, Web Data Extraction techniques emerge as a key tool to perform data analysis in Business and Competitive Intelligence systems as well as for business process re-engineering. At the Social Web level, Web Data Extraction techniques allow to gather a large amount of structured data continuously generated and disseminated by Web 2.0, Social Media and Online Social Network users and this offers unprecedented opportunities to analyze human behavior at a very large scale. We discuss also the potential of cross-fertilization, i.e., on the possibility of re-using Web Data Extraction techniques originally designed to work in a given domain, in other domains.Comment: Knowledge-based System

Systems Database and Artificial Intelligence Group

We describe a method to extract tabular data from web pages. Rather than just analyzing the DOM tree, we also exploit visual cues in the rendered version of the document to extract data from tables which are not explicitly marked with an HTML table element. To detect tables, we rely on a variant of the well-known X-Y cut algorithm as used in the OCR community. We implemented the system by directly accessing Mozilla’s box model that contains the positional data for all HTML elements of a given web page. 1

Using Visual Cues for Extraction of Tabular Data from Arbitrary HTML Documents

We describe a method to extract tabular data from web pages. Rather than just analyzing the DOM tree, we also exploit visual cues in the rendered version of the document to extract data from tables which are not explicitly marked with an HTML table element. To detect tables, we rely on a variant of the well-known X-Y cut algorithm as used in the OCR community. We implemented the system by directly accessing Mozilla's box model that contains the positional data for all HTML elements of a given web page

Web information extraction using eupeptic data in Web tables

Abstract. By leveraging on the redundant information on the Web, we are building a Web information extraction system that concentrates on eupeptic data in Web tables. We use the term eupeptic to describe such representations of information that allow for easy interpretation of the subject–predicate–object nature of individual data items. The system mimics a human approach to information gathering. It explicitly uses visual cues on rendered Web pages to locate tabular data; it uses keywords to identify relevant chunks of data that gets processed on a deeper level; and it expands its initial search to include more pages when it spots eupeptic data

Towards Domain-Independent Information Extraction from Web Tables

Traditionally, information extraction from web tables has focused on small, more or less homogeneous corpora, often based on assumptions about the use of <table> tags. A multitude of different HTML implementations of web tables make these approaches difficult to scale. In this paper, we approach the problem of domain-independent information extraction from web tables by shifting our attention from the tree-based representation of web pages to a variation of the two-dimensional visual box model used by web browsers to display the information on the screen. The thereby obtained topological and style information allows us to fill the gap created by missing domain-specific knowledge about content and table templates. We believe that, in a future step, this approach can become the basis for a new way of large-scale knowledge acquisition from the current “Visual Web.

Feature-based Object Identification for Web Automation ∗

In this paper, we address automatic identification of common functional structures on web pages, a fundamental problem for web automation applications and graphical user interface testing. In contrast to other approaches, we aim to identify relevant patterns without relying on the source code of a web page or keywords, utilizing mostly geometrical and visually perceptible properties. We achieve this by transforming pages into an independent geometrical representation, on top of which we extract a set of features that allows us to employ traditional machine learning techniques for the identification task. We evaluate this approach by analyzing three typical scenarios, reviewing the obtained information retrieval key metrics and estimating the relevance of the chosen features. Our initial results demonstrate the feasibility of the proposed approach. Keywords web object identification; web automation; web accessibility; machine learning; web page visual representation 1