Stealing Knowledge from Protected Deep Neural Networks Using Composite Unlabeled Data

As state-of-the-art deep neural networks are deployed at the core of more advanced Al-based products and services, the incentive for copying them (i.e., their intellectual properties) by rival adversaries is expected to increase considerably over time. The best way to extract or steal knowledge from such networks is by querying them using a large dataset of random samples and recording their output, followed by training a student network to mimic these outputs, without making any assumption about the original networks. The most effective way to protect against such a mimicking attack is to provide only the classification result, without confidence values associated with the softmax layer.In this paper, we present a novel method for generating composite images for attacking a mentor neural network using a student model. Our method assumes no information regarding the mentor's training dataset, architecture, or weights. Further assuming no information regarding the mentor's softmax output values, our method successfully mimics the given neural network and steals all of its knowledge. We also demonstrate that our student network (which copies the mentor) is impervious to watermarking protection methods, and thus would not be detected as a stolen model.Our results imply, essentially, that all current neural networks are vulnerable to mimicking attacks, even if they do not divulge anything but the most basic required output, and that the student model which mimics them cannot be easily detected and singled out as a stolen copy using currently available techniques

DeepMarks: A Digital Fingerprinting Framework for Deep Neural Networks

This paper proposes DeepMarks, a novel end-to-end framework for systematic fingerprinting in the context of Deep Learning (DL). Remarkable progress has been made in the area of deep learning. Sharing the trained DL models has become a trend that is ubiquitous in various fields ranging from biomedical diagnosis to stock prediction. As the availability and popularity of pre-trained models are increasing, it is critical to protect the Intellectual Property (IP) of the model owner. DeepMarks introduces the first fingerprinting methodology that enables the model owner to embed unique fingerprints within the parameters (weights) of her model and later identify undesired usages of her distributed models. The proposed framework embeds the fingerprints in the Probability Density Function (pdf) of trainable weights by leveraging the extra capacity available in contemporary DL models. DeepMarks is robust against fingerprints collusion as well as network transformation attacks, including model compression and model fine-tuning. Extensive proof-of-concept evaluations on MNIST and CIFAR10 datasets, as well as a wide variety of deep neural networks architectures such as Wide Residual Networks (WRNs) and Convolutional Neural Networks (CNNs), corroborate the effectiveness and robustness of DeepMarks framework

Revisiting the Information Capacity of Neural Network Watermarks: Upper Bound Estimation and Beyond

To trace the copyright of deep neural networks, an owner can embed its identity information into its model as a watermark. The capacity of the watermark quantify the maximal volume of information that can be verified from the watermarked model. Current studies on capacity focus on the ownership verification accuracy under ordinary removal attacks and fail to capture the relationship between robustness and fidelity. This paper studies the capacity of deep neural network watermarks from an information theoretical perspective. We propose a new definition of deep neural network watermark capacity analogous to channel capacity, analyze its properties, and design an algorithm that yields a tight estimation of its upper bound under adversarial overwriting. We also propose a universal non-invasive method to secure the transmission of the identity message beyond capacity by multiple rounds of ownership verification. Our observations provide evidence for neural network owners and defenders that are curious about the tradeoff between the integrity of their ownership and the performance degradation of their products.Comment: Accepted by AAAI 202

Coded DNN Watermark: Robustness against Pruning Models Using Constant Weight Code

Deep Neural Network (DNN) watermarking techniques are increasingly being used to protect the intellectual property of DNN models. Basically, DNN watermarking is a technique to insert side information into the DNN model without significantly degrading the performance of its original task. A pruning attack is a threat to DNN watermarking, wherein the less important neurons in the model are pruned to make it faster and more compact. As a result, removing the watermark from the DNN model is possible. This study investigates a channel coding approach to protect DNN watermarking against pruning attacks. The channel model differs completely from conventional models involving digital images. Determining the suitable encoding methods for DNN watermarking remains an open problem. Herein, we presented a novel encoding approach using constant weight codes to protect the DNN watermarking against pruning attacks. The experimental results confirmed that the robustness against pruning attacks could be controlled by carefully setting two thresholds for binary symbols in the codeword

Optimized DWT Based Digital Image Watermarking and Extraction Using RNN-LSTM

The rapid growth of Internet and the fast emergence of multi-media applications over the past decades have led to new problems such as illegal copying, digital plagiarism, distribution and use of copyrighted digital data. Watermarking digital data for copyright protection is a current need of the community. For embedding watermarks, robust algorithms in die media will resolve copyright infringements. Therefore, to enhance the robustness, optimization techniques and deep neural network concepts are utilized. In this paper, the optimized Discrete Wavelet Transform (DWT) is utilized for embedding the watermark. The optimization algorithm is a combination of Simulated Annealing (SA) and Tunicate Swarm Algorithm (TSA). After performing the embedding process, the extraction is processed by deep neural network concept of Recurrent Neural Network based Long Short-Term Memory (RNN-LSTM). From the extraction process, the original image is obtained by this RNN-LSTM method. The experimental set up is carried out in the MATLAB platform. The performance metrics of PSNR, NC and SSIM are determined and compared with existing optimization and machine learning approaches. The results are achieved under various attacks to show the robustness of the proposed work

Intellectual Property Protection for Deep Learning Models: Taxonomy, Methods, Attacks, and Evaluations

The training and creation of deep learning model is usually costly, thus it can be regarded as an intellectual property (IP) of the model creator. However, malicious users who obtain high-performance models may illegally copy, redistribute, or abuse the models without permission. To deal with such security threats, a few deep neural networks (DNN) IP protection methods have been proposed in recent years. This paper attempts to provide a review of the existing DNN IP protection works and also an outlook. First, we propose the first taxonomy for DNN IP protection methods in terms of six attributes: scenario, mechanism, capacity, type, function, and target models. Then, we present a survey on existing DNN IP protection works in terms of the above six attributes, especially focusing on the challenges these methods face, whether these methods can provide proactive protection, and their resistances to different levels of attacks. After that, we analyze the potential attacks on DNN IP protection methods from the aspects of model modifications, evasion attacks, and active attacks. Besides, a systematic evaluation method for DNN IP protection methods with respect to basic functional metrics, attack-resistance metrics, and customized metrics for different application scenarios is given. Lastly, future research opportunities and challenges on DNN IP protection are presented