Graph Distillation for Action Detection with Privileged Modalities

We propose a technique that tackles action detection in multimodal videos under a realistic and challenging condition in which only limited training data and partially observed modalities are available. Common methods in transfer learning do not take advantage of the extra modalities potentially available in the source domain. On the other hand, previous work on multimodal learning only focuses on a single domain or task and does not handle the modality discrepancy between training and testing. In this work, we propose a method termed graph distillation that incorporates rich privileged information from a large-scale multimodal dataset in the source domain, and improves the learning in the target domain where training data and modalities are scarce. We evaluate our approach on action classification and detection tasks in multimodal videos, and show that our model outperforms the state-of-the-art by a large margin on the NTU RGB+D and PKU-MMD benchmarks. The code is released at http://alan.vision/eccv18_graph/.Comment: ECCV 201

Zero-Shot Deep Domain Adaptation

Domain adaptation is an important tool to transfer knowledge about a task (e.g. classification) learned in a source domain to a second, or target domain. Current approaches assume that task-relevant target-domain data is available during training. We demonstrate how to perform domain adaptation when no such task-relevant target-domain data is available. To tackle this issue, we propose zero-shot deep domain adaptation (ZDDA), which uses privileged information from task-irrelevant dual-domain pairs. ZDDA learns a source-domain representation which is not only tailored for the task of interest but also close to the target-domain representation. Therefore, the source-domain task of interest solution (e.g. a classifier for classification tasks) which is jointly trained with the source-domain representation can be applicable to both the source and target representations. Using the MNIST, Fashion-MNIST, NIST, EMNIST, and SUN RGB-D datasets, we show that ZDDA can perform domain adaptation in classification tasks without access to task-relevant target-domain training data. We also extend ZDDA to perform sensor fusion in the SUN RGB-D scene classification task by simulating task-relevant target-domain representations with task-relevant source-domain data. To the best of our knowledge, ZDDA is the first domain adaptation and sensor fusion method which requires no task-relevant target-domain data. The underlying principle is not particular to computer vision data, but should be extensible to other domains.Comment: This paper is accepted to the European Conference on Computer Vision (ECCV), 201

Recent Advances in Transfer Learning for Cross-Dataset Visual Recognition: A Problem-Oriented Perspective

This paper takes a problem-oriented perspective and presents a comprehensive review of transfer learning methods, both shallow and deep, for cross-dataset visual recognition. Specifically, it categorises the cross-dataset recognition into seventeen problems based on a set of carefully chosen data and label attributes. Such a problem-oriented taxonomy has allowed us to examine how different transfer learning approaches tackle each problem and how well each problem has been researched to date. The comprehensive problem-oriented review of the advances in transfer learning with respect to the problem has not only revealed the challenges in transfer learning for visual recognition, but also the problems (e.g. eight of the seventeen problems) that have been scarcely studied. This survey not only presents an up-to-date technical review for researchers, but also a systematic approach and a reference for a machine learning practitioner to categorise a real problem and to look up for a possible solution accordingly

Security and Privacy for Mobile Social Networks

With the ever-increasing demands of people's social interactions, traditional online social networking applications are being shifted to the mobile ones, enabling users' social networking and interactions anywhere anytime. Due to the portability and pervasiveness of mobile devices, such as smartphones, wearable devices and tablets, Mobile Social Network (MSN), as a promising social network platform, has become increasingly popular and brought immense benefits. In MSN, users can easily discover and chat with social friends in the vicinity even without the Internet; vehicle drivers and passengers can exchange traffic information, videos or images with other vehicles on the road; customers in a shopping mall can share sale information and recommend it to their friends. With MSNs, massive opportunities are created to facilitate people's social interactions and enlarge the inherent social circle. However, the flourish of MSNs also hinges upon fully understanding and managing the challenges, such as security threats and privacy leakage. Security and privacy concerns rise as the boom of MSN applications comes up, but few users have paid adequate attentions to protect their privacy-sensitive information from disclosing. First of all, to initiate social interactions, users sometimes exchange their social interests or preferences with each other (including strangers in the vicinity) without sufficient protections. As such, some private information may be inferred from the exchanged social interests by attackers and untrusted users. Secondly, some malicious attackers might forge fake identities or false contents, such as spam and advertisements, to disrupt MSNs or mislead other users. These attackers could even collude and launch a series of security threats to MSNs. In addition, massive social network data are usually stored in untrusted cloud servers, where data confidentiality, authentication, access control and privacy are of paramount importance. Last but not least, the trade-off between data availability and privacy should be taken into account when the data are stored, queried and processed for various MSN applications. Therefore, novel security and privacy techniques become essential for MSN to provide sufficient and adjustable protections. In this thesis, we focus on security and privacy for MSNs. Based on the MSN architecture and emerging applications, we first investigate security and privacy requirements for MSNs and introduce several challenging issues, i.e., spam, misbehaviors and privacy leakage. To tackle these problems, we propose efficient security and privacy preservation schemes for MSNs. Specifically, the main contributions of this thesis can be three-fold. Firstly, to address the issues of spam in autonomous MSNs, we propose a personalized fine-grained spam filtering scheme (PIF), which exploits social characteristics during data delivery. The PIF allows users to create personalized filters according to their social interests, and enables social friends to hold these filters, discarding the unwanted data before delivery. We also design privacy-preserving coarse-grained and fine-grained filtering mechanisms in the PIF to not only enable the filtering but also prevent users' private information included in the filters from disclosing to untrusted entities. Secondly, to detect misbehaviors during MSN data sharing, we propose a social-based mobile Sybil detection scheme (SMSD). The SMSD detects Sybil attackers by differentiating the abnormal pseudonym changing and contact behaviors, since Sybil attackers frequently or rapidly change their pseudonyms to cheat legitimate users. As the volume of contact data from users keeps increasing, the SMSD utilizes local cloud servers to store and process the users' contact data such that the burden of mobile users is alleviated. The SMSD also detects the collusion attacks and prevents user's data from malicious modification when employing the untrusted local cloud server for the detection. Thirdly, to achieve the trade-off between privacy and data availability, we investigate a centralized social network application, which exploits social network to enhance human-to-human infection analysis. We integrate social network data and health data to jointly analyze the instantaneous infectivity during human-to-human contact, and propose a novel privacy-preserving infection analysis approach (PIA). The PIA enables the collaboration among different cloud servers (i.e., social network cloud server and health cloud server). It employs a privacy-preserving data query method based on conditional oblivious transfer to enable data sharing and prevent data from disclosing to untrusted entities. A privacy-preserving classification-based infection analysis method is also proposed to enable the health cloud server to infer infection spread but preserve privacy simultaneously. Finally, we summarize the thesis and share several open research directions in MSNs. The developed security solutions and research results in this thesis should provide a useful step towards better understanding and implementing secure and privacy-preserving MSNs