Autonomous Sensor Data Cleaning in Stream Mining Setting

Background: Internet of Things (IoT), earth observation and big scientific experiments are sources of extensive amounts of sensor big data today. We are faced with large amounts of data with low measurement costs. A standard approach in such cases is a stream mining approach, implying that we look at a particular measurement only once during the real-time processing. This requires the methods to be completely autonomous. In the past, very little attention was given to the most time-consuming part of the data mining process, i.e. data pre-processing. Objectives: In this paper we propose an algorithm for data cleaning, which can be applied to real-world streaming big data. Methods/Approach: We use the short-term prediction method based on the Kalman filter to detect admissible intervals for future measurements. The model can be adapted to the concept drift and is useful for detecting random additive outliers in a sensor data stream. Results: For datasets with low noise, our method has proven to perform better than the method currently commonly used in batch processing scenarios. Our results on higher noise datasets are comparable. Conclusions: We have demonstrated a successful application of the proposed method in real-world scenarios including the groundwater level, server load and smart-grid data

ASSIGNING KEYWORDS TO DOCUMENTS USING MACHINE LEARNING

This paper describes the usage of machine learning techniques to assign keywords to documents. The large hierarchy of documents available on the Web, the Yahoo hierarchy, is used here as a real-world problem domain. Machine learning techniques developed for learning on text data are used here in the hierarchical classification structure. The high number of features is reduced by taking into account the hierarchical structure and using a feature subset selection based on the method used in information retrieval. Documents are represented as word-vectors that include word sequences (n-grams) instead of just single words. The hierarchical structure of the examples and class values is taken into account when defining the subproblems and forming training examples for them. Additionally, a hierarchical structure of class values is used in classification, where only promising paths in the hierarchy are considered

Contextualized Question Answering

The paper describes a system which enables accurate and easy-to-use contextualized question answering and it provides document overview functionalities. The possibility of asking natural language questions enables a friendly interaction for the user.The contextualization is achieved by using an ontology. The answers are provided based on a domain specific document collection of choice. The approach consists of several phases as follows: data preparation, data enhancement, data indexing and handling questions. Every module uses state of the art technologies that are shown to work in a complex pipeline to make available question answering on top of a given document repository with the context of ontologies, such as Cyc, ASFA and WordNet. The functioning of the proposed approach is demonstrated on English document collections on Aquatic Sciences and Fisheries — ASFA, using Cyc ontology, ASFA thesaurus as domain specific ontology and WordNet as general ontology. Experimental evaluation has shown that the usage of ontologies increases the number of answers retrieved by about 60%. However, the number of answers that are actually correct increases by only 40% when using ontologies