Knowledge Graphs Meet Multi-Modal Learning: A Comprehensive Survey

Knowledge Graphs (KGs) play a pivotal role in advancing various AI applications, with the semantic web community's exploration into multi-modal dimensions unlocking new avenues for innovation. In this survey, we carefully review over 300 articles, focusing on KG-aware research in two principal aspects: KG-driven Multi-Modal (KG4MM) learning, where KGs support multi-modal tasks, and Multi-Modal Knowledge Graph (MM4KG), which extends KG studies into the MMKG realm. We begin by defining KGs and MMKGs, then explore their construction progress. Our review includes two primary task categories: KG-aware multi-modal learning tasks, such as Image Classification and Visual Question Answering, and intrinsic MMKG tasks like Multi-modal Knowledge Graph Completion and Entity Alignment, highlighting specific research trajectories. For most of these tasks, we provide definitions, evaluation benchmarks, and additionally outline essential insights for conducting relevant research. Finally, we discuss current challenges and identify emerging trends, such as progress in Large Language Modeling and Multi-modal Pre-training strategies. This survey aims to serve as a comprehensive reference for researchers already involved in or considering delving into KG and multi-modal learning research, offering insights into the evolving landscape of MMKG research and supporting future work.Comment: Ongoing work; 41 pages (Main Text), 55 pages (Total), 11 Tables, 13 Figures, 619 citations; Paper list is available at https://github.com/zjukg/KG-MM-Surve

The role of knowledge in determining identity of long-tail entities

The NIL entities do not have an accessible representation, which means that their identity cannot be established through traditional disambiguation. Consequently, they have received little attention in entity linking systems and tasks so far. Given the non-redundancy of knowledge on NIL entities, the lack of frequency priors, their potentially extreme ambiguity, and numerousness, they form an extreme class of long-tail entities and pose a great challenge for state-of-the-art systems. In this paper, we investigate the role of knowledge when establishing the identity of NIL entities mentioned in text. What kind of knowledge can be applied to establish the identity of NILs? Can we potentially link to them at a later point? How to capture implicit knowledge and fill knowledge gaps in communication? We formulate and test hypotheses to provide insights to these questions. Due to the unavailability of instance-level knowledge, we propose to enrich the locally extracted information with profiling models that rely on background knowledge in Wikidata. We describe and implement two profiling machines based on state-of-the-art neural models. We evaluate their intrinsic behavior and their impact on the task of determining identity of NIL entities

Methodologies for transforming data to information and advancing the understanding of water resources systems towards integrated water resources management

2017 Summer.Includes bibliographical references.The majority of river basins in the world, have undergone a great deal of transformations in terms of infrastructure and water management practices in order to meet increasing water needs due to population growth and socio-economic development. Surface water and groundwater systems are interwoven with environmental and socio-economic ones. The systems' dynamic nature, their complex interlinkages and interdependencies are inducing challenges for integrated water resources management. Informed decision-making process in water resources is deriving from a systematic analysis of the available data with the utilization of tools and models, by examining viable alternatives and their associated tradeoffs under the prism of a set of prudent priorities and expert knowledge. In an era of increasing volume and variety of data about natural and anthropogenic systems, opportunities arise for further enhancing data integration in problem-solving approaches and thus support decision-making for water resources planning and management. Although there is a plethora of variables monitored in various spatial and temporal scales, particularly in the United States, in real life, for water resources applications there are rarely, if ever, perfect data. Developing more systematic procedures to integrate the available data and harness their full potential of generating information, will improve the understanding of water resources systems and assist at the same time integrated water resources management efforts. The overarching objective of this study is to develop tools and approaches to overcome data obstacles in water resources management. This required the development of methodologies that utilize a wide range of water and environmental datasets in order to transform them into reliable and valuable information, which would address unanswered questions about water systems and water management practices, contributing to implementable efforts of integrated water resources management. More specifically, the objectives of this research are targeted in three complementary topics: drought, water demand, and groundwater supply. In this regard, their unified thread is the common quest for integrated river basin management (IRBM) under changing water resources conditions. All proposed methodologies have a common area of application namely the South Platte basin, located within Colorado. The area is characterized by limited water resources with frequent drought intervals. A system's vulnerability to drought due to the different manifestations of the phenomenon (meteorological, agricultural, hydrological, socio-economic and ecological) and the plethora of factors affecting it (precipitation patterns, the supply and demand trends, the socioeconomic background etc.) necessitates an integrated approach for delineating its magnitude and spatiotemporal extent and impacts. Thus, the first objective was to develop an implementable drought management policy tool based on the standardized drought vulnerability index framework and expanding it in order to capture more of drought's multifaceted effects. This study illustrated the advantages of a more transparent data rigorous methodology, which minimizes the need for qualitative information replacing it with a more quantitative one. It is believed that such approach may convey drought information to decision makers in a holistic manner and at the same time avoid the existing practices of broken linkages and fragmentation of reported drought impacts. Secondly, a multi-scale (well, HUC-12, and county level) comparative analysis framework was developed to identify the characteristics of the emergent water demand for unconventional oil and gas development. This effort revealed the importance of local conditions in well development patterns that influence water demand, the magnitude of water consumption in local scales in comparison to other water uses, the strategies of handling flowback water, and the need for additional data, and improved data collection methods for a detailed water life-cycle analysis including the associated tradeoffs. Finally, a novel, easy to implement, and computationally low cost methodology was developed for filling gaps in groundwater level time series. The proposed framework consists of four main components, namely: groundwater level time series; data (groundwater level, recharge and pumping) from a regional physically-based groundwater flow model; autoregressive integrated moving average with external inputs modeling; and the Ensemble Smoother (ES) technique. The methodology's efficacy to predict accurately groundwater levels was tested by conducting three numerical experiments at eighteen alluvial wells. The results suggest that the framework could serve as a valuable tool in gaining further insight of alluvium aquifer dynamics by filling missing groundwater level data in an intermittent or continuous (with relative short span) fashion. Overall, it is believed that this research has important implications in water resources decision making by developing implementable frameworks which advance further the understanding of water systems and may aid in integrated river basin management efforts

Model-based Joint Analysis of Safety and Security:Survey and Identification of Gaps

We survey the state-of-the-art on model-based formalisms for safety and security joint analysis, where safety refers to the absence of unintended failures, and security to absence of malicious attacks. We conduct a thorough literature review and - as a result - we consider fourteen model-based formalisms and compare them with respect to several criteria: (1) Modelling capabilities and Expressiveness: which phenomena can be expressed in these formalisms? To which extent can they capture safety-security interactions? (2) Analytical capabilities: which analysis types are supported? (3) Practical applicability: to what extent have the formalisms been used to analyze small or larger case studies? Furthermore, (1) we present more precise definitions for safety-security dependencies in tree-like formalisms; (2) we showcase the potential of each formalism by modelling the same toy example from the literature and (3) we present our findings and reflect on possible ways to narrow highlighted gaps. In summary, our key findings are the following: (1) the majority of approaches combine tree-like formal models; (2) the exact nature of safety-security interaction is still ill-understood and (3) diverse formalisms can capture different interactions; (4) analyzed formalisms merge modelling constructs from existing safety- and security-specific formalisms, without introducing ad hoc constructs to model safety-security interactions, or (5) metrics to analyze trade offs. Moreover, (6) large case studies representing safety-security interactions are still missing

Improving data management through automatic information extraction model in ontology for road asset management

lRoads are a critical component of transportation infrastructure, and their effective maintenance is paramount in ensuring their continued functionality and safety. This research proposes a novel information management approach based on state-of-the-art deep learning models and ontologies. The approach can automatically extract, integrate, complete, and search for project knowledge buried in unstructured text documents. The approach on the one hand facilitates implementation of modern management approaches, i.e., advanced working packaging to delivery success road management projects, on the other hand improves information management practices in the construction industry

Stock Market Prediction via Deep Learning Techniques: A Survey

The stock market prediction has been a traditional yet complex problem researched within diverse research areas and application domains due to its non-linear, highly volatile and complex nature. Existing surveys on stock market prediction often focus on traditional machine learning methods instead of deep learning methods. Deep learning has dominated many domains, gained much success and popularity in recent years in stock market prediction. This motivates us to provide a structured and comprehensive overview of the research on stock market prediction focusing on deep learning techniques. We present four elaborated subtasks of stock market prediction and propose a novel taxonomy to summarize the state-of-the-art models based on deep neural networks from 2011 to 2022. In addition, we also provide detailed statistics on the datasets and evaluation metrics commonly used in the stock market. Finally, we highlight some open issues and point out several future directions by sharing some new perspectives on stock market prediction

A Survey on Semantic Processing Techniques

Semantic processing is a fundamental research domain in computational linguistics. In the era of powerful pre-trained language models and large language models, the advancement of research in this domain appears to be decelerating. However, the study of semantics is multi-dimensional in linguistics. The research depth and breadth of computational semantic processing can be largely improved with new technologies. In this survey, we analyzed five semantic processing tasks, e.g., word sense disambiguation, anaphora resolution, named entity recognition, concept extraction, and subjectivity detection. We study relevant theoretical research in these fields, advanced methods, and downstream applications. We connect the surveyed tasks with downstream applications because this may inspire future scholars to fuse these low-level semantic processing tasks with high-level natural language processing tasks. The review of theoretical research may also inspire new tasks and technologies in the semantic processing domain. Finally, we compare the different semantic processing techniques and summarize their technical trends, application trends, and future directions.Comment: Published at Information Fusion, Volume 101, 2024, 101988, ISSN 1566-2535. The equal contribution mark is missed in the published version due to the publication policies. Please contact Prof. Erik Cambria for detail

Sensors for process and structural health monitoring of aerospace composites: a review

Structural Health Monitoring (SHM) is a promising approach to overcome the unpredictable failure behaviour of composite materials and further foster their use in aerospace industry with increased confidence. SHM may require a complex system, including sensors, wiring and cabling, data acquisition devices and software, data storage equipment, power equipment and algorithms for signal processing, involving a multidisciplinary team for its adequate development considering the operational environment and requirements of a certain application. This review paper focuses on the most promising type of sensors, laboratory made and commercially available, for SHM of aerospace composites. Sensing principles, characteristics, embedding procedures, sensor/ host materials interactions and acquired sensor data/ material behaviour are discussed. The use of sensors for in-situ process monitoring, specifically for curing and mould filling monitoring in liquid composite moulding processes are discussed. General considerations for the development of SHM systems for the aerospace environment are also briefly mentioned.The authors acknowledge the support of the European Regional Development Fund [grant number NORTE-01-0145-FEDER-000015]; and of the European Space Agency through the Network/Partnering Initiative Program