2D+3D Indoor Scene Understanding from a Single Monocular Image

Scene understanding, as a broad field encompassing many subtopics, has gained great interest in recent years. Among these subtopics, indoor scene understanding, having its own specific attributes and challenges compared to outdoor scene under- standing, has drawn a lot of attention. It has potential applications in a wide variety of domains, such as robotic navigation, object grasping for personal robotics, augmented reality, etc. To our knowledge, existing research for indoor scenes typically makes use of depth sensors, such as Kinect, that is however not always available. In this thesis, we focused on addressing the indoor scene understanding tasks in a general case, where only a monocular color image of the scene is available. Specifically, we first studied the problem of estimating a detailed depth map from a monocular image. Then, benefiting from deep-learning-based depth estimation, we tackled the higher-level tasks of 3D box proposal generation, and scene parsing with instance segmentation, semantic labeling and support relationship inference from a monocular image. Our research on indoor scene understanding provides a comprehensive scene interpretation at various perspectives and scales. For monocular image depth estimation, previous approaches are limited in that they only reason about depth locally on a single scale, and do not utilize the important information of geometric scene structures. Here, we developed a novel graphical model, which reasons about detailed depth while leveraging geometric scene structures at multiple scales. For 3D box proposals, to our best knowledge, our approach constitutes the first attempt to reason about class-independent 3D box proposals from a single monocular image. To this end, we developed a novel integrated, differentiable framework that estimates depth, extracts a volumetric scene representation and generates 3D proposals. At the core of this framework lies a novel residual, differentiable truncated signed distance function module, which is able to handle the relatively low accuracy of the predicted depth map. For scene parsing, we tackled its three subtasks of instance segmentation, se- mantic labeling, and the support relationship inference on instances. Existing work typically reasons about these individual subtasks independently. Here, we leverage the fact that they bear strong connections, which can facilitate addressing these sub- tasks if modeled properly. To this end, we developed an integrated graphical model that reasons about the mutual relationships of the above subtasks. In summary, in this thesis, we introduced novel and effective methodologies for each of three indoor scene understanding tasks, i.e., depth estimation, 3D box proposal generation, and scene parsing, and exploited the dependencies on depth estimates of the latter two tasks. Evaluation on several benchmark datasets demonstrated the effectiveness of our algorithms and the benefits of utilizing depth estimates for higher-level tasks

Knowledge Extraction from Textual Resources through Semantic Web Tools and Advanced Machine Learning Algorithms for Applications in Various Domains

Nowadays there is a tremendous amount of unstructured data, often represented by texts, which is created and stored in variety of forms in many domains such as patients' health records, social networks comments, scientific publications, and so on. This volume of data represents an invaluable source of knowledge, but unfortunately it is challenging its mining for machines. At the same time, novel tools as well as advanced methodologies have been introduced in several domains, improving the efficacy and the efficiency of data-based services. Following this trend, this thesis shows how to parse data from text with Semantic Web based tools, feed data into Machine Learning methodologies, and produce services or resources to facilitate the execution of some tasks. More precisely, the use of Semantic Web technologies powered by Machine Learning algorithms has been investigated in the Healthcare and E-Learning domains through not yet experimented methodologies. Furthermore, this thesis investigates the use of some state-of-the-art tools to move data from texts to graphs for representing the knowledge contained in scientific literature. Finally, the use of a Semantic Web ontology and novel heuristics to detect insights from biological data in form of graph are presented. The thesis contributes to the scientific literature in terms of results and resources. Most of the material presented in this thesis derives from research papers published in international journals or conference proceedings

소셜 네트워크와 이커머스 플랫폼에서의 잠재 네트워크 마이닝

학위논문(박사) -- 서울대학교대학원 : 공과대학 컴퓨터공학부, 2023. 2. 권태경.웹 기반 서비스의 폭발적인 발달로 사용자들은 온라인 상에서 폭넓게 연결되고 있다. 온라인 플랫폼 상에서, 사용자들은 서로에게 영향을 주고받으며 의사 결정에 그들의 경험과 의견을 반영하는 경향을 보인다. 본 학위 논문에서는 대표적인 온라인 플랫폼인 소셜 네트워크 서비스와 이커머스 플랫폼에서의 사용자 행동에 대해 연구하였다. 온라인 플랫폼에서의 사용자 행동은 사용자와 플랫폼 구성 요소 간의 관계로 표현할 수 있다. 사용자의 구매는 사용자와 상품 간의 관계로, 사용자의 체크인은 사용자와 장소 간의 관계로 나타내진다. 여기에 행동의 시간과 레이팅, 태그 등의 정보가 포함될 수 있다. 본 연구에서는 두 플랫폼에서 정의된 사용자의 행동 그래프에 영향을 미치는 잠재 네트워크를 파악하는 연구를 제시한다. 위치 기반의 소셜 네트워크 서비스의 경우 특정 장소에 방문하는 체크인 형식으로 많은 포스트가 만들어지는데, 사용자의 장소 방문은 사용자 간에 사전에 존재하는 친구 관계에 의해 영향을 크게 받는다. 사용자 활동 네트워크의 저변에 잠재된 사용자 간의 관계를 파악하는 것은 활동 예측에 도움이 될 수 있으며, 이를 위해 본 논문에서는 비지도학습 기반으로 활동 네트워크로부터 사용자 간 사회적 관계를 추출하는 연구를 제안하였다. 기존에 연구되었던 방법들은 두 사용자가 동시에 방문하는 행위인 co-visitation을 중점적으로 고려하여 사용자 간의 관계를 예측하거나, 네트워크 임베딩 또는 그래프 신경망(GNN)을 사용하여 표현 학습을 수행하였다. 그러나 이러한 접근 방식은 주기적인 방문이나 장거리 이동 등으로 대표되는 사용자의 행동 패턴을 잘 포착하지 못한다. 행동 패턴을 더 잘 학습하기 위해, ANES는 사용자 컨텍스트 내에서 사용자와 관심 지점(POI) 간의 측면(Aspect) 지향 관계를 학습한다. ANES는 User-POI 이분 그래프의 구조에서 사용자의 행동을 여러 개의 측면으로 나누고, 각각의 관계를 고려하여 행동 패턴을 추출하는 최초의 비지도학습 기반 접근 방식이다. 실제 LBSN 데이터에서 수행된 광범위한 실험에서, ANES는 기존에 제안되었던 기법들보다 높은 성능을 보여준다. 위치 기반 소셜 네트워크와는 다르게, 이커머스의 리뷰 시스템에서는 사용자들이 능동적인 팔로우/팔로잉 등의 행위를 수행하지 않고도 플랫폼에 의해 서로의 정보를 주고받고 영향력을 행사하게 된다. 이와 같은 사용자들의 행동 특성은 리뷰 스팸에 의해 쉽게 악용될 수 있다. 리뷰 스팸은 실제 사용자의 의견을 숨기고 평점을 조작하여 잘못된 정보를 전달하는 방식으로 이루어진다. 나는 이를 해결하기 위해 사용자 리뷰 데이터에서 사용자 간 사전 공모성(Collusiveness)의 가능성을 찾고, 이를 스팸 탐지에 활용한 방법인 SC-Com을 제안한다. SC-Com은 행동의 공모성으로부터 사용자 간 공모 점수를 계산하고 해당 점수를 바탕으로 전체 사용자를 유사한 사용자들의 커뮤니티로 분류한다. 그 후 스팸 유저와 일반 유저를 구별하는 데에 중요한 그래프 기반의 특징을 추출하여 감독 학습 기반의 분류기의 입력 데이터로 활용하는 방법을 제시한다. SC-Com은 공모성을 갖는 스팸 유저의 집합을 효과적으로 탐지한다. 실제 데이터셋을 이용한 실험에서, SC-Com은 기존 논문들 대비 스팸 탐지에 뛰어난 성능을 보여주었다. 위 논문에서 다양한 데이터에 대해 연구된 암시적 연결망 탐지 모델은 레이블이 없는 데이터에 대해서도 사전에 연결되었을 가능성이 높은 사용자들을 예측하므로, 실시간 위치 데이터나, 앱 사용 데이터 등의 다양한 데이터에서 활용할 수 있는 유용한 정보를 제공하여 광고 추천 시스템이나, 악성 유저 탐지 등의 분야에서 기여할 수 있을 것으로 기대한다.Following the exploding usage on online services, people are connected with each other more broadly and widely. In online platforms, people influence each other, and have tendency to reflect their opinions in decision-making. Social Network Services (SNSs) and E-commerce are typical example of online platforms. User behaviors in online platforms can be defined as relation between user and platform components. A user's purchase is a relationship between a user and a product, and a user's check-in is a relationship between a user and a place. Here, information such as action time, rating, tag, etc. may be included. In many studies, platform user behavior is represented in graph form. At this time, the elements constituting the nodes of the graph are composed of objects such as users and products and places within the platform, and the interaction between the platform elements and the user can be expressed as two nodes being connected. In this study, I present studies to identify potential networks that affect the user's behavior graph defined on the two platforms. In ANES, I focus on representation learning for social link inference based on user trajectory data. While traditional methods predict relations between users by considering hand-crafted features, recent studies first perform representation learning using network/node embedding or graph neural networks (GNNs) for downstream tasks such as node classification and link prediction. However, those approaches fail to capture behavioral patterns of individuals ingrained in periodical visits or long-distance movements. To better learn behavioral patterns, this paper proposes a novel scheme called ANES (Aspect-oriented Network Embedding for Social link inference). ANES learns aspect-oriented relations between users and Point-of-Interests (POIs) within their contexts. ANES is the first approach that extracts the complex behavioral pattern of users from both trajectory data and the structure of User-POI bipartite graphs. Extensive experiments on several real-world datasets show that ANES outperforms state-of-the-art baselines. In contrast to active social networks, people are connected to other users regardless of their intentions in some platforms, such as online shopping websites and restaurant review sites. They do not have any information about each other in advance, and they only have a common point which is that they have visited or have planned to visit same place or purchase a product. Interestingly, users have tendency to be influenced by the review data on their purchase intentions. Unfortunately, this instinct is easily exploited by opinion spammers. In SC-Com, I focus on opinion spam detection in online shopping services. In many cases, my decision-making process is closely related to online reviews. However, there have been threats of opinion spams by hired reviewers increasingly, which aim to mislead potential customers by hiding genuine consumers opinions. Opinion spams should be filed up collectively to falsify true information. Fortunately, I propose the way to spot the possibility to detect them from their collusiveness. In this paper, I propose SC-Com, an optimized collusive community detection framework. It constructs the graph of reviewers from the collusiveness of behavior and divides a graph by communities based on their mutual suspiciousness. After that, I extract community-based and temporal abnormality features which are critical to discriminate spammers from other genuine users. I show that my method detects collusive opinion spam reviewers effectively and precisely from their collective behavioral patterns. In the real-world dataset, my approach showed prominent performance while only considering primary data such as time and ratings. These implicit network inference models studied on various data in this thesis predicts users who are likely to be pre-connected to unlabeled data, so it is expected to contribute to areas such as advertising recommendation systems and malicious user detection by providing useful information.Chapter 1 Introduction 1 Chapter 2 Social link Inference in Location-based check-in data 5 2.1 Background 5 2.2 Related Work 12 2.3 Location-based Social Network Service Data 15 2.4 Aspect-wise Graph Decomposition 18 2.5 Aspect-wise Graph learning 19 2.6 Inferring Social Relation from User Representation 21 2.7 Performance Analysis 23 2.8 Discussion and Implications 26 2.9 Summary 34 Chapter 3 Detecting collusiveness from reviews in Online platforms and its application 35 3.1 Background 35 3.2 Related Work 39 3.3 Online Review Data 43 3.4 Collusive Graph Projection 44 3.5 Reviewer Community Detection 47 3.6 Review Community feature extraction and spammer detection 51 3.7 Performance Analysis 53 3.8 Discussion and Implications 55 3.9 Summary 62 Chapter 4 Conclusion 63박

Algorithm and System Co-design for Efficient Subgraph-based Graph Representation Learning

Subgraph-based graph representation learning (SGRL) has been recently proposed to deal with some fundamental challenges encountered by canonical graph neural networks (GNNs), and has demonstrated advantages in many important data science applications such as link, relation and motif prediction. However, current SGRL approaches suffer from scalability issues since they require extracting subgraphs for each training or test query. Recent solutions that scale up canonical GNNs may not apply to SGRL. Here, we propose a novel framework SUREL for scalable SGRL by co-designing the learning algorithm and its system support. SUREL adopts walk-based decomposition of subgraphs and reuses the walks to form subgraphs, which substantially reduces the redundancy of subgraph extraction and supports parallel computation. Experiments over six homogeneous, heterogeneous and higher-order graphs with millions of nodes and edges demonstrate the effectiveness and scalability of SUREL. In particular, compared to SGRL baselines, SUREL achieves 10$\times$ speed-up with comparable or even better prediction performance; while compared to canonical GNNs, SUREL achieves 50% prediction accuracy improvement.Comment: This is an extended version of the full paper to appear in PVLDB 15.11(VLDB 2022

Bilingual dictionary generation and enrichment via graph exploration

In recent years, we have witnessed a steady growth of linguistic information represented and exposed as linked data on the Web. Such linguistic linked data have stimulated the development and use of openly available linguistic knowledge graphs, as is the case with the Apertium RDF, a collection of interconnected bilingual dictionaries represented and accessible through Semantic Web standards. In this work, we explore techniques that exploit the graph nature of bilingual dictionaries to automatically infer new links (translations). We build upon a cycle density based method: partitioning the graph into biconnected components for a speed-up, and simplifying the pipeline through a careful structural analysis that reduces hyperparameter tuning requirements. We also analyse the shortcomings of traditional evaluation metrics used for translation inference and propose to complement them with new ones, both-word precision (BWP) and both-word recall (BWR), aimed at being more informative of algorithmic improvements. Over twenty-seven language pairs, our algorithm produces dictionaries about 70% the size of existing Apertium RDF dictionaries at a high BWP of 85% from scratch within a minute. Human evaluation shows that 78% of the additional translations generated for dictionary enrichment are correct as well. We further describe an interesting use-case: inferring synonyms within a single language, on which our initial human-based evaluation shows an average accuracy of 84%. We release our tool as free/open-source software which can not only be applied to RDF data and Apertium dictionaries, but is also easily usable for other formats and communities.This work was partially funded by the Prêt-à-LLOD project within the European Union’s Horizon 2020 research and innovation programme under grant agreement no. 825182. This article is also based upon work from COST Action CA18209 NexusLinguarum, “European network for Web-centred linguistic data science”, supported by COST (European Cooperation in Science and Technology). It has been also partially supported by the Spanish projects TIN2016-78011-C4-3-R and PID2020-113903RB-I00 (AEI/FEDER, UE), by DGA/FEDER, and by the Agencia Estatal de Investigación of the Spanish Ministry of Economy and Competitiveness and the European Social Fund through the “Ramón y Cajal” program (RYC2019-028112-I)

Social Network Data Management

With the increasing usage of online social networks and the semantic web's graph structured RDF framework, and the rising adoption of networks in various fields from biology to social science, there is a rapidly growing need for indexing, querying, and analyzing massive graph structured data. Facebook has amassed over 500 million users creating huge volumes of highly connected data. Governments have made RDF datasets containing billions of triples available to the public. In the life sciences, researches have started to connect disparate data sets of research results into one giant network of valuable information. Clearly, networks are becoming increasingly popular and growing rapidly in size, requiring scalable solutions for network data management. This thesis focuses on the following aspects of network data management. We present a hierarchical index structure for external memory storage of network data that aims to maximize data locality. We propose efficient algorithms to answer subgraph matching queries against network databases and discuss effective pruning strategies to improve performance. We show how adaptive cost models can speed up subgraph matching query answering by assigning budgets to index retrieval operations and adjusting the query plan while executing. We develop a cloud oriented social network database, COSI, which handles massive network datasets too large for a single computer by partitioning the data across multiple machines and achieving high performance query answering through asynchronous parallelization and cluster-aware heuristics. Tracking multiple standing queries against a social network database is much faster with our novel multi-view maintenance algorithm, which exploits common substructures between queries. To capture uncertainty inherent in social network querying, we define probabilistic subgraph matching queries over deterministic graph data and propose algorithms to answer them efficiently. Finally, we introduce a general relational machine learning framework and rule-based language, Probabilistic Soft Logic, to learn from and probabilistically reason about social network data and describe applications to information integration and information fusion

이미지의 의미적 이해를 위한 시각적 관계의 이용

학위논문(박사)--서울대학교 대학원 :융합과학기술대학원 융합과학부(디지털정보융합전공),2019. 8. 곽노준.이미지를 이해하는 것은 컴퓨터 비전 분야에서 가장 근본적인 목적 중 하나이다. 이러한 이해는 다양한 산업 분야의 문제를 해결 할 수 있는 혁신이 될 수 있다. 최근 딥러닝의 발전과 함께, 이미지에서 객관적인 요소를 인식하는 기술은 매우 발전되어 왔다. 그러나 시각 정보를 제대로 이해하기 위해서는 사람처럼 맥락 정보를 이해하는 것이 중요하다. 인간은 주로 직접적인 시각정보와 함께 맥락을 이해하여 의미 있는 지식 정보로 활용한다. 본 논문에서는 객체간의 의미적 관계정보를 구축하과 활용하는 방법론을 제시하여 보다 나은 이미지의 이해 방법을 연구하였다. 첫 번째로, 다이어그램에서 관계 지식을 표현하는 관계 그래프를 생성하는 알고리즘을 제안하였다. 다이어그램이 가진 정보를 축약하는 능력이 다른 형태의 지식 저장 방법에 비해 뛰어나지만, 그에 따라 해석하기에는 다양한 요소와 유연한 레이아웃 때문에 풀기 어려운 문제였다. 우리는 다이어그램에서 객체를 찾고 그것들의 관계를 찾는 통합 네트워크를 제안한다. 그리고 이러한 능동적인 그래프 생성을 위한 특수 모듈은 DGGN을 제안한다. 이 모듈의 성능을 나타내기 위해 모듈안의 활성화 게이트의 정보 역학을 비주얼라이즈 하여 분석하였다. 또한 공개된 다이어그램 데이터셋에서 기존의 알고리즘을 뛰어넘는 성능을 증명하였다.마지막으로 질의 응답 데이터셋을 이용한 실험으로 향후 다양한 응용 가능성도 증명하였다. 두 번째로, 우리는 현존하는 질의 응답 데이터셋 중 가장 복잡한 형태를 가진 교과서에서 질의응답 (TQA) 문제를 풀기위한 솔루션을 제안하였다. TQA 데이터셋은 질문 파트와 본문 파트 모두에 이미지와 텍스트 형태를 가진 데이터를 포함하고 있다. 이러한 복잡한 구조를 해결하기 위해 우리는 f-GCN이라는 다중 모달 그래프를 처리할 수 있는 모듈을 제안하였다. 이 모듈을 통해 보다 효율적으로 다중 모달을 그래프 형태로 처리하여 활용하기 쉬운 피쳐로 바꿔줄 수 있다. 그 다음으로 교과서의 경우 다양한 주제가 포함되어 있고 그에 따라 용어나 내용이 겹치지 않고 기술되어 있다. 그로인해 완전 새로운 내용의 문제를 풀어야하는 out-of-domain 이슈가 있다. 이를 해결하기위해 정답을 보지 않고 본문만으로 자가 학습을 하는 알고리즘을 제안하였다. 이 두 알고리즘을 통해 기존 연구보다 훨씬 좋은 성능을 보이는 실험 결과를 제시하였고 각각의 모듈의 기능성에 대해 검증하였다. 마지막으로, 인간과 물건의 관계정보를 활용하여 객체 검출을 약지도 학습으로 배우는 프레임워크를 제안하였다. 객체 검출 문제를 풀기위해 노동력이 많이 필요한 데이터 라벨링 작업이 필요하다. 그 중 가장 노력이 많이 필요한 위치 라벨링인데, 새로운 방법론은 인간과 물건의 관계를 이용하여 이부분을 해결하였다. 우리는 RRPN이란 모듈을 제안하여 인간의 포즈정보와 관계에 관한 동사를 이용하여 처음보는 물건의 위치를 추정할 수 있다. 이를 통해 새롭게 배우는 목표 라벨에 대해, 정답 라벨 없이 위치를 추정하여 학습할 수 있어 훨씬 적은 노력만 사용해도 된다. 또한 RRPN은 추가 방식의 구조로 다양한 태스크에 관한 네트워크에 추가 할 수 있다. HICO-DET 데이터셋을 사용하여 실험한 결과 현재의 지도학습을 대신할 가능성을 보여주었다. 또한 우리 모델이 처음 본 물건의 위치를 잘 추정하고 있음을 시각화를 통해 보여주었다.Understanding an image is one of the fundamental goals of computer vision and can provide important breakthroughs for various industries. In particular, the ability to recognize objective instances such as objects and poses has been developed due to recent deep learning approaches. However, deeply comprehending a visual scene requires higher understanding, such as is found in human beings. Humans usually exploit contextual information from visual inputs to detect meaningful features. In this dissertation, visual relation in various contexts, from the construction phase to the application phase, is studied with three tasks. We first propose a new algorithm for constructing relation graphs that contains relational knowledge in diagrams . Although diagrams contain richer information compared to individual image-based or language-based data, proper solutions for automatically understanding diagrams have not been proposed due to their innate multimodality and the arbitrariness of their layouts. To address this problem, we propose a unified diagram-parsing network for generating knowledge from diagrams based on an object detector and a recurrent neural network designed for a graphical structure. Specifically, we propose a dynamic graph-generation network that is based on dynamic memory and graph theory. We explore the dynamics of information in a diagram with the activation of gates in gated recurrent unit (GRU) cells. Using publicly available diagram datasets, our model demonstrates a state-of-the-art result that outperforms other baselines. Moreover, further experiments on question answering demonstrate the potential of the proposed method for use in various applications. Next, we introduce a novel algorithm to solve the Textbook Question Answering (TQA) task; this task describes more realistic QA (Question Answering) problems compared to other recent tasks. We mainly focus on two issues related to the analysis of the TQA dataset. First, solving the TQA problems requires an understanding of multimodal contexts in complicated input data. To overcome this issue of extracting knowledge features from long text lessons and merging them with visual features, we establish a context graph from texts and images and propose a new module f-GCN based on graph convolutional networks (GCN). Second, in the TQA dataset , scientific terms are not spread over the chapters and subjects are split. To overcome this so-called ``out-of-domain issue, before learning QA problems we introduce a novel, self-supervised, open-set learning process without any annotations. The experimental results indicate that our model significantly outperforms prior state-of-the-art methods. Moreover, ablation studies confirm that both methods (incorporating f-GCN to extract knowledge from multimodal contexts and our newly proposed, self-supervised learning process) are effective for TQA problems. Third, we introduce a novel, weakly supervised object detection (WSOD) paradigm to detect objects belonging to rare classes that do not have many examples. We use transferable knowledge from human-object interactions (HOI). While WSOD has lower performance than full supervision, we mainly focus on HOI that can strongly supervise complex semantics in images. Therefore, we propose a novel module called the ``relational region proposal network (RRPN) that outputs an object-localizing attention map with only human poses and action verbs. In the source domain, we fully train an object detector and the RRPN with full supervision of HOI. With transferred knowledge about the localization map from the trained RRPN, a new object detector can learn unseen objects with weak verbal supervisions of HOI without bounding box annotations in the target domain. Because the RRPN is designed as an add-on type, we can apply it not only to object detection but also to other domains such as semantic segmentation. The experimental results using a HICO-DET dataset suggest the possibility that the proposed method can be a cheap alternative for the current supervised object detection paradigm. Moreover, qualitative results demonstrate that our model can properly localize unseen objects in HICO-DET and V-COCO datasets.1. Introduction 1 1.1 Problem Definition 4 1.2 Motivation 6 1.3 Challenges 7 1.4 Contributions 9 1.4.1 Generating Visual Relation Graphs from Diagrams 9 1.4.2 Application of the Relation Graph in Textbook Question Answering 10 1.4.3 Weakly Supervised Object Detection with Human-object Interaction 11 1.5 Outline 11 2. Background 13 2.1 Visual relationships 13 2.2 Neural networks on a graph 16 2.3 Human-object interaction 17 3. Generating Visual Relation Graphs from Diagrams 18 3.1 Related Work 20 3.2 Proposed Method 21 3.2.1 Detecting Constituents in a Diagram 21 3.2.2 Generating a Graph of relationships 22 3.2.3 Multi-task Training and Cascaded Inference 27 3.2.4 Details of Post-processing 29 3.3 Experiment 29 3.3.1 Datasets 29 3.3.2 Baseline 32 3.3.3 Metrics 32 3.3.4 Implementation Details 33 3.3.5 Quantitative Results 35 3.3.6 Qualitative Results 37 3.4 Discussion 38 3.5 Conclusion 41 4. Application of the Relation Graph in Textbook Question Answering 46 4.1 Related Work 48 4.2 Problem 50 4.3 Proposed Method 53 4.3.1 Multi-modal Context Graph Understanding 53 4.3.2 Multi-modal Problem Solving 55 4.3.3 Self-supervised open-set comprehension 57 4.3.4 Process of Building Textual Context Graph 61 4.4 Experiment 62 4.4.1 Implementation Details 62 4.4.2 Dataset 62 4.4.3 Baselines 63 4.4.4 Quantitative Results 64 4.4.5 Qualitative Results 67 4.5 Conclusion 70 5. Weakly Supervised Object Detection with Human-object Interaction 77 5.1 Related Work 80 5.2 Algorithm Overview 81 5.3 Proposed Method 84 5.3.1 Training on the Source classes Ds 86 5.3.2 Training on the Target classes Dt 89 5.4 Experiment 90 5.4.1 Implementation details 90 5.4.2 Dataset and Pre-processing 91 5.4.3 Metrics 91 5.4.4 Comparison with different feature combination 92 5.4.5 Comparison with different attention loss balance and box threshold 95 5.4.6 Comparison with prior works 96 5.4.7 Qualitative results 96 5.5 Conclusion 100 6. Concluding Remarks 105 6.1 Summary 105 6.2 Limitation and Future Directions 106Docto

Graph Mining for Cybersecurity: A Survey

The explosive growth of cyber attacks nowadays, such as malware, spam, and intrusions, caused severe consequences on society. Securing cyberspace has become an utmost concern for organizations and governments. Traditional Machine Learning (ML) based methods are extensively used in detecting cyber threats, but they hardly model the correlations between real-world cyber entities. In recent years, with the proliferation of graph mining techniques, many researchers investigated these techniques for capturing correlations between cyber entities and achieving high performance. It is imperative to summarize existing graph-based cybersecurity solutions to provide a guide for future studies. Therefore, as a key contribution of this paper, we provide a comprehensive review of graph mining for cybersecurity, including an overview of cybersecurity tasks, the typical graph mining techniques, and the general process of applying them to cybersecurity, as well as various solutions for different cybersecurity tasks. For each task, we probe into relevant methods and highlight the graph types, graph approaches, and task levels in their modeling. Furthermore, we collect open datasets and toolkits for graph-based cybersecurity. Finally, we outlook the potential directions of this field for future research

Cyber physical security of avionic systems

“Cyber-physical security is a significant concern for critical infrastructures. The exponential growth of cyber-physical systems (CPSs) and the strong inter-dependency between the cyber and physical components introduces integrity issues such as vulnerability to injecting malicious data and projecting fake sensor measurements. Traditional security models partition the CPS from a security perspective into just two domains: high and low. However, this absolute partition is not adequate to address the challenges in the current CPSs as they are composed of multiple overlapping partitions. Information flow properties are one of the significant classes of cyber-physical security methods that model how inputs of a system affect its outputs across the security partition. Information flow supports traceability that helps in detecting vulnerabilities and anomalous sources, as well as helps in rendering mitigation measures. To address the challenges associated with securing CPSs, two novel approaches are introduced by representing a CPS in terms of a graph structure. The first approach is an automated graph-based information flow model introduced to identify information flow paths in the avionics system and partition them into security domains. This approach is applied to selected aspects of the avionic systems to identify the vulnerabilities in case of a system failure or an attack and provide possible mitigation measures. The second approach is based on graph neural networks (GNN) to classify the graphs into different security domains. Using these two approaches, successful partitioning of the CPS into different security domains is possible in addition to identifying their optimal coverage. These approaches enable designers and engineers to ensure the integrity of the CPS. The engineers and operators can use this process during design-time and in real-time to identify failures or attacks on the system”--Abstract, page iii