Improving Retrieval Accuracy in Main Content Extraction from HTML Web Documents

The rapid growth of text based information on the World Wide Web and various applications making use of this data motivates the need for efficient and effective methods to identify and separate the “main content” from the additional content items, such as navigation menus, advertisements, design elements or legal disclaimers. Firstly, in this thesis, we study, develop, and evaluate R2L, DANA, DANAg, and AdDANAg, a family of novel algorithms for extracting the main content of web documents. The main concept behind R2L, which also provided the initial idea and motivation for the other three algorithms, is to use well particularities of Right-to-Left languages for obtaining the main content of web pages. As the English character set and the Right-to-Left character set are encoded in different intervals of the Unicode character set, we can efficiently distinguish the Right-to-Left characters from the English ones in an HTML file. This enables the R2L approach to recognize areas of the HTML file with a high density of Right-to-Left characters and a low density of characters from the English character set. Having recognized these areas, R2L can successfully separate only the Right-to-Left characters. The first extension of the R2L, DANA, improves effectiveness of the baseline algorithm by employing an HTML parser in a post processing phase of R2L for extracting the main content from areas with a high density of Right-to-Left characters. DANAg is the second extension of the R2L and generalizes the idea of R2L to render it language independent. AdDANAg, the third extension of R2L, integrates a new preprocessing step to normalize the hyperlink tags. The presented approaches are analyzed under the aspects of efficiency and effectiveness. We compare them to several established main content extraction algorithms and show that we extend the state-of-the-art in terms of both, efficiency and effectiveness. Secondly, automatically extracting the headline of web articles has many applications. We develop and evaluate a content-based and language-independent approach, TitleFinder, for unsupervised extraction of the headline of web articles. The proposed method achieves high performance in terms of effectiveness and efficiency and outperforms approaches operating on structural and visual features.Das rasante Wachstum von textbasierten Informationen im World Wide Web und die Vielfalt der Anwendungen, die diese Daten nutzen, macht es notwendig, effiziente und effektive Methoden zu entwickeln, die den Hauptinhalt identifizieren und von den zusätzlichen Inhaltsobjekten wie z.B. Navigations-Menüs, Anzeigen, Design-Elementen oder Haftungsausschlüssen trennen. Zunächst untersuchen, entwickeln und evaluieren wir in dieser Arbeit R2L, DANA, DANAg und AdDANAg, eine Familie von neuartigen Algorithmen zum Extrahieren des Inhalts von Web-Dokumenten. Das grundlegende Konzept hinter R2L, das auch zur Entwicklung der drei weiteren Algorithmen führte, nutzt die Besonderheiten der Rechts-nach-links-Sprachen aus, um den Hauptinhalt von Webseiten zu extrahieren. Da der lateinische Zeichensatz und die Rechts-nach-links-Zeichensätze durch verschiedene Abschnitte des Unicode-Zeichensatzes kodiert werden, lassen sich die Rechts-nach-links-Zeichen leicht von den lateinischen Zeichen in einer HTML-Datei unterscheiden. Das erlaubt dem R2L-Ansatz, Bereiche mit einer hohen Dichte von Rechts-nach-links-Zeichen und wenigen lateinischen Zeichen aus einer HTML-Datei zu erkennen. Aus diesen Bereichen kann dann R2L die Rechts-nach-links-Zeichen extrahieren. Die erste Erweiterung, DANA, verbessert die Wirksamkeit des Baseline-Algorithmus durch die Verwendung eines HTML-Parsers in der Nachbearbeitungsphase des R2L-Algorithmus, um den Inhalt aus Bereichen mit einer hohen Dichte von Rechts-nach-links-Zeichen zu extrahieren. DANAg erweitert den Ansatz des R2L-Algorithmus, so dass eine Sprachunabhängigkeit erreicht wird. Die dritte Erweiterung, AdDANAg, integriert eine neue Vorverarbeitungsschritte, um u.a. die Weblinks zu normalisieren. Die vorgestellten Ansätze werden in Bezug auf Effizienz und Effektivität analysiert. Im Vergleich mit mehreren etablierten Hauptinhalt-Extraktions-Algorithmen zeigen wir, dass sie in diesen Punkten überlegen sind. Darüber hinaus findet die Extraktion der Überschriften aus Web-Artikeln vielfältige Anwendungen. Hierzu entwickeln wir mit TitleFinder einen sich nur auf den Textinhalt beziehenden und sprachabhängigen Ansatz. Das vorgestellte Verfahren ist in Bezug auf Effektivität und Effizienz besser als bekannte Ansätze, die auf strukturellen und visuellen Eigenschaften der HTML-Datei beruhen

A model for the detection of breast cancer using machine learning and thermal images in a mobile environment

Breast cancer is the most common cancer amongst women and one of the deadliest. Various modalities exist which image the breasts, all with a focus on early detection; thermography is one such method. It is a non-invasive test, which is safe and can be used for a wide variety of breast densities. It functions by analysing thermal patterns captured via an infrared camera of the surface of the breast. Advances in infrared and mobile technology enable this modality to be mobile based; allowing a high degree of portability at a lower cost. Furthermore, as technology has improved, machine learning has played a larger role in medical practices by offering unbiased, consistent, and timely second opinions. Machine learning algorithms are able to classify medical images automatically if offered in the correct format. This study aims to provide a model, which integrates breast cancer detection, thermal imaging, machine learning, and mobile technology. The conceptual model is theorised from three literature studies regarding: identifiable aspects of breast cancer through thermal imaging, the mobile ecosystem, and classification using machine learning algorithms. The model is implemented and evaluated using an experiment designed to classify automatically thermal breast images of the same quality that mobile attachable thermal cameras are able to capture. The experiment contrasts various combinations of segmentation methods, extracted features, and classification algorithms. Promising results were shown in the experiment with a high degree of accuracy obtained. The successful results obtained from the experimentation process validates the feasibility of the model

A model for the detection of breast cancer using machine learning and thermal images in a mobile environment

Breast cancer is the most common cancer amongst women and one of the deadliest. Various modalities exist which image the breasts, all with a focus on early detection; thermography is one such method. It is a non-invasive test, which is safe and can be used for a wide variety of breast densities. It functions by analysing thermal patterns captured via an infrared camera of the surface of the breast. Advances in infrared and mobile technology enable this modality to be mobile based; allowing a high degree of portability at a lower cost. Furthermore, as technology has improved, machine learning has played a larger role in medical practices by offering unbiased, consistent, and timely second opinions. Machine learning algorithms are able to classify medical images automatically if offered in the correct format. This study aims to provide a model, which integrates breast cancer detection, thermal imaging, machine learning, and mobile technology. The conceptual model is theorised from three literature studies regarding: identifiable aspects of breast cancer through thermal imaging, the mobile ecosystem, and classification using machine learning algorithms. The model is implemented and evaluated using an experiment designed to classify automatically thermal breast images of the same quality that mobile attachable thermal cameras are able to capture. The experiment contrasts various combinations of segmentation methods, extracted features, and classification algorithms. Promising results were shown in the experiment with a high degree of accuracy obtained. The successful results obtained from the experimentation process validates the feasibility of the model

Unsupervised learning of mDTD extraction patterns for Web text mining

This paper presents a new extraction pattern, called modified Document Type Definition (mDTD), which relies on analytical interpretation to identify extraction target from the contents of the Web documents. From conventional DTD in XML documents, we develop two major extensions: first, we introduce an extended content model with type-specific operators and keywords, and second, we refine the way to interpret the conventional DTD rules. As the result of the two, bur mDTD becomes freely represent HTML structures and extraction targets. The goal of mDTD is to overcome the current major barriers, that is, domain portability (with minimal human intervention) and high performance, on information extraction. The human experts compose an mDTD as seed rules, and then our system automatically extracts a set of instances by the mDTD from structured documents on the Web. We use the extracted instances as Sequential mDTD Learner (SmL) inputs to generate new mDTD rules based on part-of-speech tags and features for lexical similarity. This process does not require any hand-annotated corpus. We have experimented with 330 Korean and 220 English Web documents on audio and video shopping sites. The average extraction precision is 91.3% for Korean and 81.9% for English. (C) 2003 Elsevier Science Ltd. All rights reserved.X117sciescopu

Safety and Reliability - Safe Societies in a Changing World

The contributions cover a wide range of methodologies and application areas for safety and reliability that contribute to safe societies in a changing world. These methodologies and applications include: - foundations of risk and reliability assessment and management - mathematical methods in reliability and safety - risk assessment - risk management - system reliability - uncertainty analysis - digitalization and big data - prognostics and system health management - occupational safety - accident and incident modeling - maintenance modeling and applications - simulation for safety and reliability analysis - dynamic risk and barrier management - organizational factors and safety culture - human factors and human reliability - resilience engineering - structural reliability - natural hazards - security - economic analysis in risk managemen