Information embedding and retrieval in 3D printed objects

Deep learning and convolutional neural networks have become the main tools of computer vision. These techniques are good at using supervised learning to learn complex representations from data. In particular, under limited settings, the image recognition model now performs better than the human baseline. However, computer vision science aims to build machines that can see. It requires the model to be able to extract more valuable information from images and videos than recognition. Generally, it is much more challenging to apply these deep learning models from recognition to other problems in computer vision. This thesis presents end-to-end deep learning architectures for a new computer vision field: watermark retrieval from 3D printed objects. As it is a new area, there is no state-of-the-art on many challenging benchmarks. Hence, we first define the problems and introduce the traditional approach, Local Binary Pattern method, to set our baseline for further study. Our neural networks seem useful but straightfor- ward, which outperform traditional approaches. What is more, these networks have good generalization. However, because our research field is new, the problems we face are not only various unpredictable parameters but also limited and low-quality training data. To address this, we make two observations: (i) we do not need to learn everything from scratch, we know a lot about the image segmentation area, and (ii) we cannot know everything from data, our models should be aware what key features they should learn. This thesis explores these ideas and even explore more. We show how to use end-to-end deep learning models to learn to retrieve watermark bumps and tackle covariates from a few training images data. Secondly, we introduce ideas from synthetic image data and domain randomization to augment training data and understand various covariates that may affect retrieve real-world 3D watermark bumps. We also show how the illumination in synthetic images data to effect and even improve retrieval accuracy for real-world recognization applications

Data Hiding with Deep Learning: A Survey Unifying Digital Watermarking and Steganography

Data hiding is the process of embedding information into a noise-tolerant signal such as a piece of audio, video, or image. Digital watermarking is a form of data hiding where identifying data is robustly embedded so that it can resist tampering and be used to identify the original owners of the media. Steganography, another form of data hiding, embeds data for the purpose of secure and secret communication. This survey summarises recent developments in deep learning techniques for data hiding for the purposes of watermarking and steganography, categorising them based on model architectures and noise injection methods. The objective functions, evaluation metrics, and datasets used for training these data hiding models are comprehensively summarised. Finally, we propose and discuss possible future directions for research into deep data hiding techniques

Comparing small visual differences between conforming meshes

This paper gives a method of quantifying small visual differences between 3D mesh models with conforming topology, based on the theory of strain fields. Our experiments show that our difference estimates are well correlated with human perception of differences. This work has applications in the evaluation of 3D mesh watermarking, 3D mesh compression reconstruction, and 3D mesh filtering

DIGITAL SECURITY: 3D GEOMETRY PROTECTION OF THE AUTOMATICALLY RESTITUTED HISTORICAL BUILDINGS

This paper describes a novel method of data protection of the three-dimensional (3D) models that are obtained from automatic process of geometric restitution, using old two-dimensional (2D) architectural and artistic drawings. The first contribution of our research is the algorithm that includes several image processing steps, which are required in order to define walls, staircases and openings from the digitalized hand drawn architectural plans. The result of this step is detailed 3D model of the digitally processed historical building plans. The experimental confirmation of the algorithm accuracy is 3D model of the Chateau de Versailles, which is descripted by old hand drawings, dating between the end of the XVII and the XIX century. Next part of our research is theoretical and mathematical analysis of geometrical features of such 3D model that is a result of the image processing algorithm. The key-achievement of this part is new method of protecting the geometrical data using optimized adaptive Sparse Quantization Index Modulation (QIM) for embedding data bits into essential structure of the generated model. As a final result we present a secure authentication of the automatically restituted 3D model of the historically important artifact

Nouvelles méthodes de synchronisation de nuages de points 3D pour l'insertion de données cachées

This thesis addresses issues relating to the protection of 3D object meshes. For instance, these objects can be created using CAD tool developed by the company STRATEGIES. In an industrial context, 3D meshes creators need to have tools in order to verify meshes integrity, or check permission for 3D printing for example.In this context we study data hiding on 3D meshes. This approach allows us to insert information in a secure and imperceptible way in a mesh. This may be an identifier, a meta-information or a third-party content, for instance, in order to transmit secretly a texture. Data hiding can address these problems by adjusting the trade-off between capacity, imperceptibility and robustness. Generally, data hiding methods consist of two stages, the synchronization and the embedding. The synchronization stage consists of finding and ordering available components for insertion. One of the main challenges is to propose an effective synchronization method that defines an order on mesh components. In our work, we propose to use mesh vertices, specifically their geometric representation in space, as basic components for synchronization and embedding. We present three new synchronisation methods based on the construction of a Hamiltonian path in a vertex cloud. Two of these methods jointly perform the synchronization stage and the embedding stage. This is possible thanks to two new high-capacity embedding methods (from 3 to 24 bits per vertex) that rely on coordinates quantization. In this work we also highlight the constraints of this kind of synchronization. We analyze the different approaches proposed with several experimental studies. Our work is assessed on various criteria including the capacity and imperceptibility of the embedding method. We also pay attention to security aspects of the proposed methods.Cette thèse aborde les problèmes liés à la protection de maillages d'objets 3D. Ces objets peuvent, par exemple, être créés à l'aide d'outil de CAD développés par la société STRATEGIES. Dans un cadre industriel, les créateurs de maillages 3D ont besoin de disposer d'outils leur permettant de vérifier l'intégrité des maillages, ou de vérifier des autorisations pour l'impression 3D par exemple. Dans ce contexte nous étudions l'insertion de données cachées dans des maillages 3D. Cette approche permet d'insérer de façon imperceptible et sécurisée de l'information dans un maillage. Il peut s'agir d'un identifiant, de méta-informations ou d'un contenu tiers, par exemple, pour transmettre de façon secrète une texture. L'insertion de données cachées permet de répondre à ces problèmes en jouant sur le compromis entre la capacité, l'imperceptibilité et la robustesse. Généralement, les méthodes d'insertion de données cachées se composent de deux phases, la synchronisation et l'insertion. La synchronisation consiste à trouver et ordonner les éléments disponibles pour l'insertion. L'un des principaux challenges est de proposer une méthode de synchronisation 3D efficace qui définit un ordre sur les composants des maillages. Dans nos travaux, nous proposons d'utiliser les sommets du maillage, plus précisément leur représentation géométrique dans l'espace comme composants de base pour la synchronisation et l'insertion. Nous présentons donc trois nouvelles méthodes de synchronisation de la géométrie des maillages basées sur la construction d'un chemin hamiltonien dans un nuage de sommets. Deux de ces méthodes permettent de manière conjointe de synchroniser les sommets et de cacher un message. Cela est possible grâce à deux nouvelles méthodes d'insertion haute capacité (de 3 à 24 bits par sommet) qui s'appuient sur la quantification des coordonnées. Dans ces travaux nous mettons également en évidence les contraintes propres à ce type de synchronisation. Nous discutons des différentes approches proposées dans plusieurs études expérimentales. Nos travaux sont évalués sur différents critères dont la capacité et l'imperceptibilité de la méthode d'insertion. Nous portons également notre attention aux aspects sécurité des méthodes

A 3D object encryption scheme which maintains dimensional and spatial stability

Due to widespread applications of 3D vision technology, the research into 3D object protection is primarily important. To maintain confidentiality, encryption of 3D objects is essential. However, the requirements and limitations imposed by 3D objects indicate the impropriety of conventional cryptosystems for 3D object encryption. This suggests the necessity of designing new ciphers. In addition, the study of prior works indicates that the majority of problems encountered with encrypting 3D objects are about point cloud protection, dimensional and spatial stability, and robustness against surface reconstruction attacks. To address these problems, this paper proposes a 3D object encryption scheme, based on a series of random permutations and rotations, which deform the geometry of the point cloud. Since the inverse of a permutation and a rotation matrix is its transpose, the decryption implementation is very efficient. Our statistical analyses show that within the cipher point cloud, points are randomly distributed. Furthermore, the proposed cipher leaks no information regarding the geometric structure of the plain point cloud, and is also highly sensitive to the changes of the plaintext and secret key. The theoretical and experimental analyses demonstrate the security, effectiveness, and robustness of the proposed cipher against surface reconstruction attacks

A robust region-adaptive digital image watermarking system

Digital image watermarking techniques have drawn the attention of researchers and practitioners as a means of protecting copyright in digital images. The technique involves a subset of information-hiding technologies, which work by embedding information into a host image without perceptually altering the appearance of the host image. Despite progress in digital image watermarking technology, the main objectives of the majority of research in this area remain improvements in the imperceptibility and robustness of the watermark to attacks. Watermark attacks are often deliberately applied to a watermarked image in order to remove or destroy any watermark signals in the host data. The purpose of the attack is. aimed at disabling the copyright protection system offered by watermarking technology. Our research in the area of watermark attacks found a number of different types, which can be classified into a number of categories including removal attacks, geometry attacks, cryptographic attacks and protocol attacks. Our research also found that both pixel domain and transform domain watermarking techniques share similar levels of sensitivity to these attacks. The experiment conducted to analyse the effects of different attacks on watermarked data provided us with the conclusion that each attack affects the high and low frequency part of the watermarked image spectrum differently. Furthermore, the findings also showed that the effects of an attack can be alleviated by using a watermark image with a similar frequency spectrum to that of the host image. The results of this experiment led us to a hypothesis that would be proven by applying a watermark embedding technique which takes into account all of the above phenomena. We call this technique 'region-adaptive watermarking'. Region-adaptive watermarking is a novel embedding technique where the watermark data is embedded in different regions of the host image. The embedding algorithms use discrete wavelet transforms and a combination of discrete wavelet transforms and singular value decomposition, respectively. This technique is derived from the earlier hypothesis that the robustness of a watermarking process can be improved by using watermark data in the frequency spectrum that are not too dissimilar to that of the host data. To facilitate this, the technique utilises dual watermarking technologies and embeds parts of the watermark images into selected regions of the host image. Our experiment shows that our technique improves the robustness of the watermark data to image processing and geometric attacks, thus validating the earlier hypothesis. In addition to improving the robustness of the watermark to attacks, we can also show a novel use for the region-adaptive watermarking technique as a means of detecting whether certain types of attack have occurred. This is a unique feature of our watermarking algorithm, which separates it from other state-of-the-art techniques. The watermark detection process uses coefficients derived from the region-adaptive watermarking algorithm in a linear classifier. The experiment conducted to validate this feature shows that, on average, 94.5% of all watermark attacks can be correctly detected and identified

Application of Machine Learning within Visual Content Production

We are living in an era where digital content is being produced at a dazzling pace. The heterogeneity of contents and contexts is so varied that a numerous amount of applications have been created to respond to people and market demands. The visual content production pipeline is the generalisation of the process that allows a content editor to create and evaluate their product, such as a video, an image, a 3D model, etc. Such data is then displayed on one or more devices such as TVs, PC monitors, virtual reality head-mounted displays, tablets, mobiles, or even smartwatches. Content creation can be simple as clicking a button to film a video and then share it into a social network, or complex as managing a dense user interface full of parameters by using keyboard and mouse to generate a realistic 3D model for a VR game. In this second example, such sophistication results in a steep learning curve for beginner-level users. In contrast, expert users regularly need to refine their skills via expensive lessons, time-consuming tutorials, or experience. Thus, user interaction plays an essential role in the diffusion of content creation software, primarily when it is targeted to untrained people. In particular, with the fast spread of virtual reality devices into the consumer market, new opportunities for designing reliable and intuitive interfaces have been created. Such new interactions need to take a step beyond the point and click interaction typical of the 2D desktop environment. The interactions need to be smart, intuitive and reliable, to interpret 3D gestures and therefore, more accurate algorithms are needed to recognise patterns. In recent years, machine learning and in particular deep learning have achieved outstanding results in many branches of computer science, such as computer graphics and human-computer interface, outperforming algorithms that were considered state of the art, however, there are only fleeting efforts to translate this into virtual reality. In this thesis, we seek to apply and take advantage of deep learning models to two different content production pipeline areas embracing the following subjects of interest: advanced methods for user interaction and visual quality assessment. First, we focus on 3D sketching to retrieve models from an extensive database of complex geometries and textures, while the user is immersed in a virtual environment. We explore both 2D and 3D strokes as tools for model retrieval in VR. Therefore, we implement a novel system for improving accuracy in searching for a 3D model. We contribute an efficient method to describe models through 3D sketch via an iterative descriptor generation, focusing both on accuracy and user experience. To evaluate it, we design a user study to compare different interactions for sketch generation. Second, we explore the combination of sketch input and vocal description to correct and fine-tune the search for 3D models in a database containing fine-grained variation. We analyse sketch and speech queries, identifying a way to incorporate both of them into our system's interaction loop. Third, in the context of the visual content production pipeline, we present a detailed study of visual metrics. We propose a novel method for detecting rendering-based artefacts in images. It exploits analogous deep learning algorithms used when extracting features from sketches

Human-Centric Deep Generative Models: The Blessing and The Curse

Over the past years, deep neural networks have achieved significant progress in a wide range of real-world applications. In particular, my research puts a focused lens in deep generative models, a neural network solution that proves effective in visual (re)creation. But is generative modeling a niche topic that should be researched on its own? My answer is critically no. In the thesis, I present the two sides of deep generative models, their blessing and their curse to human beings. Regarding what can deep generative models do for us, I demonstrate the improvement in performance and steerability of visual (re)creation. Regarding what can we do for deep generative models, my answer is to mitigate the security concerns of DeepFakes and improve minority inclusion of deep generative models. For the performance of deep generative models, I probe on applying attention modules and dual contrastive loss to generative adversarial networks (GANs), which pushes photorealistic image generation to a new state of the art. For the steerability, I introduce Texture Mixer, a simple yet effective approach to achieve steerable texture synthesis and blending. For the security, my research spans over a series of GAN fingerprinting solutions that enable the detection and attribution of GAN-generated image misuse. For the inclusion, I investigate the biased misbehavior of generative models and present my solution in enhancing the minority inclusion of GAN models over underrepresented image attributes. All in all, I propose to project actionable insights to the applications of deep generative models, and finally contribute to human-generator interaction