Cognitive approaches and optical multispectral data for semi-automated classification of landforms in a rugged mountainous area

This paper introduces a new open source, knowledge-based framework for automatic interpretation of remote sensing images, called InterIMAGE. This framework owns a flexible modular architecture, in which image processing operators can be associated to both root and leaf nodes of the semantic network, which constitutes a differential strategy in comparison to other object-based image analysis platforms currently available. The architecture, main features as well as an overview on the interpretation strategy implemented in InterIMAGE is presented. The paper also reports an experiment on the classification of landforms. Different geomorphometric and textural attributes obtained from ASTER/Terra images were combined with fuzzy logic and drove the interpretation semantic network. Object-based statistical agreement indices, estimated from a comparison between the classified scene and a reference map, were used to assess the classification accuracy. The InterIMAGE interpretation strategy yielded a classification result with strong agreement and proved to be effective for the extraction of landforms

Semantic Text Analysis on Social Networks and Data Processing: Review and Future Directions

Social network usage is growing exponentially in the most up-to-date decade; though social networks are becoming increasingly popular every day, many users are continuously active social network users. Using Twitter, LinkedIn, Facebook, and other social media sites has become the most convenient way for people. There is an enormous quantity of data produced by users of social networks. The most common part of modern research analysis is instrumental for many social network analysis applications. However, people actively utilize social networking sites and diverse uses of these sites. social media sites handle an immense amount of knowledge and answer these three computational problems, noise, dynamism, and scale. Semantic comprehension of the document, image, and video exchanged in a social network was also an essential topic in network analysis. Utilizing data processing provides vast datasets such as averages, laws, and patterns to discover practical knowledge. Using social media, data analysis was primarily used for machine learning, analysis, information extraction, statistical modelling, data preprocessing, and data interpretation processes. This research intentions to deliver an inclusive overview of social network research and application analyze state-of-the-art social media data analysis methods by reviewing basic concepts, social networks and elements social network research is linked to. Semantic ways of manipulating text in social networks are then clarified, and literature discusses studies before on these themes. Next, the evolving methods in research on social network analysis are discussed, especially in analyzing semantic text on social networks. Finally, subjects and opportunities for future research directions are explained

Multimodal Grounding for Language Processing

This survey discusses how recent developments in multimodal processing facilitate conceptual grounding of language. We categorize the information flow in multimodal processing with respect to cognitive models of human information processing and analyze different methods for combining multimodal representations. Based on this methodological inventory, we discuss the benefit of multimodal grounding for a variety of language processing tasks and the challenges that arise. We particularly focus on multimodal grounding of verbs which play a crucial role for the compositional power of language.Comment: The paper has been published in the Proceedings of the 27 Conference of Computational Linguistics. Please refer to this version for citations: https://www.aclweb.org/anthology/papers/C/C18/C18-1197

Fireground location understanding by semantic linking of visual objects and building information models

This paper presents an outline for improved localization and situational awareness in fire emergency situations based on semantic technology and computer vision techniques. The novelty of our methodology lies in the semantic linking of video object recognition results from visual and thermal cameras with Building Information Models (BIM). The current limitations and possibilities of certain building information streams in the context of fire safety or fire incident management are addressed in this paper. Furthermore, our data management tools match higher-level semantic metadata descriptors of BIM and deep-learning based visual object recognition and classification networks. Based on these matches, estimations can be generated of camera, objects and event positions in the BIM model, transforming it from a static source of information into a rich, dynamic data provider. Previous work has already investigated the possibilities to link BIM and low-cost point sensors for fireground understanding, but these approaches did not take into account the benefits of video analysis and recent developments in semantics and feature learning research. Finally, the strengths of the proposed approach compared to the state-of-the-art is its (semi -)automatic workflow, generic and modular setup and multi-modal strategy, which allows to automatically create situational awareness, to improve localization and to facilitate the overall fire understanding

Multi modal multi-semantic image retrieval

PhDThe rapid growth in the volume of visual information, e.g. image, and video can overwhelm users’ ability to find and access the specific visual information of interest to them. In recent years, ontology knowledge-based (KB) image information retrieval techniques have been adopted into in order to attempt to extract knowledge from these images, enhancing the retrieval performance. A KB framework is presented to promote semi-automatic annotation and semantic image retrieval using multimodal cues (visual features and text captions). In addition, a hierarchical structure for the KB allows metadata to be shared that supports multi-semantics (polysemy) for concepts. The framework builds up an effective knowledge base pertaining to a domain specific image collection, e.g. sports, and is able to disambiguate and assign high level semantics to ‘unannotated’ images. Local feature analysis of visual content, namely using Scale Invariant Feature Transform (SIFT) descriptors, have been deployed in the ‘Bag of Visual Words’ model (BVW) as an effective method to represent visual content information and to enhance its classification and retrieval. Local features are more useful than global features, e.g. colour, shape or texture, as they are invariant to image scale, orientation and camera angle. An innovative approach is proposed for the representation, annotation and retrieval of visual content using a hybrid technique based upon the use of an unstructured visual word and upon a (structured) hierarchical ontology KB model. The structural model facilitates the disambiguation of unstructured visual words and a more effective classification of visual content, compared to a vector space model, through exploiting local conceptual structures and their relationships. The key contributions of this framework in using local features for image representation include: first, a method to generate visual words using the semantic local adaptive clustering (SLAC) algorithm which takes term weight and spatial locations of keypoints into account. Consequently, the semantic information is preserved. Second a technique is used to detect the domain specific ‘non-informative visual words’ which are ineffective at representing the content of visual data and degrade its categorisation ability. Third, a method to combine an ontology model with xi a visual word model to resolve synonym (visual heterogeneity) and polysemy problems, is proposed. The experimental results show that this approach can discover semantically meaningful visual content descriptions and recognise specific events, e.g., sports events, depicted in images efficiently. Since discovering the semantics of an image is an extremely challenging problem, one promising approach to enhance visual content interpretation is to use any associated textual information that accompanies an image, as a cue to predict the meaning of an image, by transforming this textual information into a structured annotation for an image e.g. using XML, RDF, OWL or MPEG-7. Although, text and image are distinct types of information representation and modality, there are some strong, invariant, implicit, connections between images and any accompanying text information. Semantic analysis of image captions can be used by image retrieval systems to retrieve selected images more precisely. To do this, a Natural Language Processing (NLP) is exploited firstly in order to extract concepts from image captions. Next, an ontology-based knowledge model is deployed in order to resolve natural language ambiguities. To deal with the accompanying text information, two methods to extract knowledge from textual information have been proposed. First, metadata can be extracted automatically from text captions and restructured with respect to a semantic model. Second, the use of LSI in relation to a domain-specific ontology-based knowledge model enables the combined framework to tolerate ambiguities and variations (incompleteness) of metadata. The use of the ontology-based knowledge model allows the system to find indirectly relevant concepts in image captions and thus leverage these to represent the semantics of images at a higher level. Experimental results show that the proposed framework significantly enhances image retrieval and leads to narrowing of the semantic gap between lower level machinederived and higher level human-understandable conceptualisation

NON-PARAMETRIC STATISTICAL APPROACH TO CORRECT SATELLITE RAINFALL DATA IN NEAR-REAL-TIME FOR RAIN BASED FLOOD NOWCASTING

Floods resulting from intense rainfall are one of the most disastrous hazards in many regions of the world since they contribute greatly to personal injury and to property damage mainly as a result of their ability to strike with little warning. The possibility to give an alert about a flooding situation at least a few hours before helps greatly to reduce the damage. Therefore, scores of flood forecasting systems have been developed during the past few years mainly at country level and regional level. Flood forecasting systems based only on traditional methods such as return period of flooding situations or extreme rainfall events have failed on most occasions to forecast flooding situations accurately because of changes on territory in recent years by extensive infrastructure development, increased frequency of extreme rainfall events over recent decades, etc. Nowadays, flood nowcasting systems or early warning systems which run on real- time precipitation data are becoming more popular as they give reliable forecasts compared to traditional flood forecasting systems. However, these kinds of systems are often limited to developed countries as they need well distributed gauging station networks or sophisticated surface-based radar systems to collect real-time precipitation data. In most of the developing countries and in some developed countries also, precipitation data from available sparse gauging stations are inadequate for developing representative aerial samples needed by such systems. As satellites are able to provide a global coverage with a continuous temporal availability, currently the possibility of using satellite-based rainfall estimates in flood nowcasting systems is being highly investigated. To contribute to the world's requirement for flood early warning systems, ITHACA developed a global scale flood nowcasting system that runs on near-real-time satellite rainfall estimates. The system was developed in cooperation with United Nations World Food Programme (WFP), to support the preparedness phase of the WFP like humanitarian assistance agencies, mainly in less developed countries. The concept behind this early warning system is identifying critical rainfall events for each hydrological basin on the earth with past rainfall data and using them to identify floodable rainfall events with real time rainfall data. The individuation of critical rainfall events was done with a hydrological analysis using 3B42 rainfall data which is the most accurate product of Tropical Rainfall Measuring Mission (TRMM) Multi-satellite Precipitation Analysis (TMPA) dataset. These critical events have been stored in a database and when a rainfall event is found in real-time which is similar or exceeds the event in the database an alert is issued for the basin area. The most accurate product of TMPA (3B42) is derived by applying bias adjustments to real time rainfall estimates using rain gauge data, thus it is available for end-users 10-15 days after each calendar month. The real time product of TMPA (3B42RT) is released approximately 9 hours after real-time and lacks of such kind of bias adjustments using rain gauge data as rain gauge data are not available in real time. Therefore, to have reliable alerts it is very important to reduce the uncertainty of 3B42RT product before using it in the early warning system. For this purpose, a statistical approach was proposed to make near real- time bias adjustments for the near real time product of TMPA (3B42RT). In this approach the relationship between the bias adjusted rainfall data product (3B42) and the real-time rainfall data product (3B42RT) was analyzed on the basis of drainage basins for the period from January 2003 to December 2007, and correction factors were developed for each basin worldwide to perform near real-time bias adjusted product estimation from the real-time rainfall data product (3B42RT). The accuracy of the product was analyzed by comparing with gauge rainfall data from Bangladesh and it was found that the uncertainty of the product is less even than the most accurate product of TMPA dataset (3B42