Dynamic Deep Multi-modal Fusion for Image Privacy Prediction

With millions of images that are shared online on social networking sites, effective methods for image privacy prediction are highly needed. In this paper, we propose an approach for fusing object, scene context, and image tags modalities derived from convolutional neural networks for accurately predicting the privacy of images shared online. Specifically, our approach identifies the set of most competent modalities on the fly, according to each new target image whose privacy has to be predicted. The approach considers three stages to predict the privacy of a target image, wherein we first identify the neighborhood images that are visually similar and/or have similar sensitive content as the target image. Then, we estimate the competence of the modalities based on the neighborhood images. Finally, we fuse the decisions of the most competent modalities and predict the privacy label for the target image. Experimental results show that our approach predicts the sensitive (or private) content more accurately than the models trained on individual modalities (object, scene, and tags) and prior privacy prediction works. Also, our approach outperforms strong baselines, that train meta-classifiers to obtain an optimal combination of modalities.Comment: Accepted by The Web Conference (WWW) 201

Context-Dependent Privacy-Aware Photo Sharing based on Machine Learning

Photo privacy has raised a growing concern with the advancements of image analytics, face recognition, and deep learning techniques widely applied on social media. If properly deployed, these powerful techniques can in turn assist people in enhancing their online privacy. One possible approach is to build a strong, automatic and dynamic access control mechanism based on analyzing the image content and learning users sharing behavior. This paper presents a model for context-dependent and privacy-aware photo sharing based on machine learning. The proposed model utilizes image semantics and requester contextual information to decide whether or not to share a particular picture with a specific requester at certain context, and if yes, at which granularity. To evaluate the proposed model, we conducted a user study on 23 subjects and collected a dataset containing 1’018 manually annotated images with 12’216 personalized contextual sharing decisions. Evaluation experiments were performed and the results show a promising performance of the proposed model for photo sharing decision making. Furthermore, the influences of different types of features on decision making have been investigated, the results of which validate the usefulness of pre-defined features and imply a significant variance between users sharing behaviors and privacy attitudes

Privacy-Friendly Photo Sharing and Relevant Applications Beyond

Popularization of online photo sharing brings people great convenience, but has also raised concerns for privacy. Researchers proposed various approaches to enable image privacy, most of which focus on encrypting or distorting image visual content. In this thesis, we investigate novel solutions to protect image privacy with a particular emphasis on online photo sharing. To this end, we propose not only algorithms to protect visual privacy in image content but also design of architectures for privacy-preserving photo sharing. Beyond privacy, we also explore additional impacts and potentials of employing daily images in other three relevant applications. First, we propose and study two image encoding algorithms to protect visual content in image, within a Secure JPEG framework. The first method scrambles a JPEG image by randomly changing the signs of its DCT coefficients based on a secret key. The second method, named JPEG Transmorphing, allows one to protect arbitrary image regions with any obfuscation, while secretly preserving the original image regions in application segments of the obfuscated JPEG image. Performance evaluations reveal a good degree of storage overhead and privacy protection capability for both methods, and particularly a good level of pleasantness for JPEG Transmorphing, if proper manipulations are applied. Second, we investigate the design of two architectures for privacy-preserving photo sharing. The first architecture, named ProShare, is built on a public key infrastructure (PKI) integrated with a ciphertext-policy attribute-based encryption (CP-ABE), to enable the secure and efficient access to user-posted photos protected by Secure JPEG. The second architecture is named ProShare S, in which a photo sharing service provider helps users make photo sharing decisions automatically based on their past decisions using machine learning. The photo sharing service analyzes not only the content of a user's photo, but also context information about the image capture and a prospective requester, and finally makes decision whether or not to share a particular photo to the requester, and if yes, at which granularity. A user study along with extensive evaluations were performed to validate the proposed architecture. In the end, we research into three relevant topics in regard to daily photos captured or shared by people, but beyond their privacy implications. In the first study, inspired by JPEG Transmorphing, we propose an animated JPEG file format, named aJPEG. aJPEG preserves its animation frames as application markers in a JPEG image and provides smaller file size and better image quality than conventional GIF. In the second study, we attempt to understand the impact of popular image manipulations applied in online photo sharing on evoked emotions of observers. The study reveals that image manipulations indeed influence people's emotion, but such impact also depends on the image content. In the last study, we employ a deep convolutional neural network (CNN), the GoogLeNet model, to perform automatic food image detection and categorization. The promising results obtained provide meaningful insights in design of automatic dietary assessment system based on multimedia techniques, e.g. image analysis

Image Privacy Prediction Using Deep Features

Online image sharing in social media sites such as Facebook, Flickr, and Instagram can lead to unwanted disclosure and privacy violations, when privacy settings are used inappropriately. With the exponential increase in the number of images that are shared online, the development of effective and efficient prediction methods for image privacy settings are highly needed. In this study, we explore deep visual features and deep image tags for image privacy prediction. The results of our experiments show that models trained on deep visual features outperform those trained on SIFT and GIST. The results also show that deep image tags combined with user tags perform best among all tested features