CamPro: Camera-based Anti-Facial Recognition

The proliferation of images captured from millions of cameras and the advancement of facial recognition (FR) technology have made the abuse of FR a severe privacy threat. Existing works typically rely on obfuscation, synthesis, or adversarial examples to modify faces in images to achieve anti-facial recognition (AFR). However, the unmodified images captured by camera modules that contain sensitive personally identifiable information (PII) could still be leaked. In this paper, we propose a novel approach, CamPro, to capture inborn AFR images. CamPro enables well-packed commodity camera modules to produce images that contain little PII and yet still contain enough information to support other non-sensitive vision applications, such as person detection. Specifically, CamPro tunes the configuration setup inside the camera image signal processor (ISP), i.e., color correction matrix and gamma correction, to achieve AFR, and designs an image enhancer to keep the image quality for possible human viewers. We implemented and validated CamPro on a proof-of-concept camera, and our experiments demonstrate its effectiveness on ten state-of-the-art black-box FR models. The results show that CamPro images can significantly reduce face identification accuracy to 0.3\% while having little impact on the targeted non-sensitive vision application. Furthermore, we find that CamPro is resilient to adaptive attackers who have re-trained their FR models using images generated by CamPro, even with full knowledge of privacy-preserving ISP parameters.Comment: Accepted by NDSS Symposium 202

Kamerabasierte Egomotion-Bestimmung mit natürlichen Merkmalen zur Unterstützung von Augmented-Reality-Systemen

In dieser Arbeit werden Verfahren zur Eigenbewegungsschätzung mit Stereokamerasystemen und Tiefenbildkameras untersucht. Der erste Teil beschäftigt sich mit Merkmalsextraktion und -Verfolgung in Bildsequenzen zum Gebrauch in Augmented-Reality-Anwendungen. Im zweiten Teil werden Anwendungsgebiete und Verfahren aus dem Bereich der Stereo-Egomotion analysiert und ein eigener Ansatz, der sowohl mit Stereobildsequenzen als auch mit Tiefenbildsequenzen zurechtkommt, vorgestellt

Machine learning based digital image forensics and steganalysis

The security and trustworthiness of digital images have become crucial issues due to the simplicity of malicious processing. Therefore, the research on image steganalysis (determining if a given image has secret information hidden inside) and image forensics (determining the origin and authenticity of a given image and revealing the processing history the image has gone through) has become crucial to the digital society. In this dissertation, the steganalysis and forensics of digital images are treated as pattern classification problems so as to make advanced machine learning (ML) methods applicable. Three topics are covered: (1) architectural design of convolutional neural networks (CNNs) for steganalysis, (2) statistical feature extraction for camera model classification, and (3) real-world tampering detection and localization. For covert communications, steganography is used to embed secret messages into images by altering pixel values slightly. Since advanced steganography alters the pixel values in the image regions that are hard to be detected, the traditional ML-based steganalytic methods heavily relied on sophisticated manual feature design have been pushed to the limit. To overcome this difficulty, in-depth studies are conducted and reported in this dissertation so as to move the success achieved by the CNNs in computer vision to steganalysis. The outcomes achieved and reported in this dissertation are: (1) a proposed CNN architecture incorporating the domain knowledge of steganography and steganalysis, and (2) ensemble methods of the CNNs for steganalysis. The proposed CNN is currently one of the best classifiers against steganography. Camera model classification from images aims at assigning a given image to its source capturing camera model based on the statistics of image pixel values. For this, two types of statistical features are designed to capture the traces left by in-camera image processing algorithms. The first is Markov transition probabilities modeling block-DCT coefficients for JPEG images; the second is based on histograms of local binary patterns obtained in both the spatial and wavelet domains. The designed features serve as the input to train support vector machines, which have the best classification performance at the time the features are proposed. The last part of this dissertation documents the solutions delivered by the author’s team to The First Image Forensics Challenge organized by the Information Forensics and Security Technical Committee of the IEEE Signal Processing Society. In the competition, all the fake images involved were doctored by popular image-editing software to simulate the real-world scenario of tampering detection (determine if a given image has been tampered or not) and localization (determine which pixels have been tampered). In Phase-1 of the Challenge, advanced steganalysis features were successfully migrated to tampering detection. In Phase-2 of the Challenge, an efficient copy-move detector equipped with PatchMatch as a fast approximate nearest neighbor searching method were developed to identify duplicated regions within images. With these tools, the author’s team won the runner-up prizes in both the two phases of the Challenge

An innovative vision system for industrial applications

Tesis doctoral inédita leída en la Universidad Autónoma de Madrid, Escuela Politécnica Superior, Departamento de Tecnología Electrónica y de las Comunicaciones. Fecha de lectura: 20-11-2015A pesar de que los sistemas de visión por computadora ocupan un puesto predominante en nuestra sociedad, su estructura no sigue ningún estándar. La implementación de aplicaciones de visión requiere de plataformas de alto rendimiento tales como GPUs o FPGAs y el uso de sensores de imagen con características muy distintas a las de la electrónica de consumo. En la actualidad, cada fabricante y equipo de investigación desarrollan sus plataformas de visión de forma independiente y sin ningún tipo de intercompatibilidad. En esta tesis se presenta una nueva plataforma de visión por computador utilizable en un amplio espectro de aplicaciones. Las características de dicha plataforma se han definido tras la implementación de tres aplicaciones de visión, basadas en: SOC, FPGA y GPU, respectivamente. Como resultado, se ha definido una plataforma modular con los siguientes componentes intercambiables: Sensor, procesador de imágenes ”al vuelo”, unidad de procesado principal, acelerador hardware y pila de software. Asimismo, se presenta un algoritmo para realizar transformaciones geométricas, sintetizable en FPGA y con una latencia de tan solo 90 líneas horizontales. Todos los elementos software de esta plataforma están desarrollados con licencias de Software Libre; durante el trascurso de esta tesis se han contribuido y aceptado más de 200 cambios a distintos proyectos de Software Libre, tales como: Linux, YoctoProject y U-boot, entre otros, promoviendo el ecosistema necesario para la creación de una comunidad alrededor de esta tesis.Tras la implementación de la plataforma en un producto comercial, Qtechnology QT5022, y su uso en varias aplicaciones industriales se ha demostrado que es posible el uso de una plataforma genérica de visión que permita reutilizar elementos y comparar resultados objetivamenteDespite the fact that computer vision systems place an important role in our society, its structure does not follow any standard. The implementation of computer vision application require high performance platforms, such as GPUs or FPGAs, and very specialized image sensors. Nowadays, each manufacturer and research lab develops their own vision platform independently without considering any inter-compatibility. This Thesis introduces a new computer vision platform that can be used in a wide spectrum of applications. The characteristics of the platform has been defined after the implementation of three different computer vision applications, based on: SOC, FPGA and GPU respectively. As a result, a new modular platform has been defined with the following interchangeably elements: Sensor, Image Processing Pipeline, Processing Unit, Acceleration unit and Computer Vision Stack. This thesis also presents an FPGA synthetizable algorithm for performing geometric transformations on the fly, with a latency under 90 horizontal lines. All the software elements of this platform have an Open Source licence; over the course of this thesis, more than 200 patches have been contributed and accepted into different Open Source projects like the Linux Kernel, Yocto Project and U-boot, among others, promoting the required ecosystem for the creation of a community around this novel system. The platform has been validated in an industrial product, Qtechnology QT5022, used on diverse industrial applications; demonstrating the great advantages of a generic computer vision system as a platform for reusing elements and comparing results objectivel

Towards the development of flexible, reliable, reconfigurable, and high-performance imaging systems

Current FPGAs can implement large systems because of the high density of reconfigurable logic resources in a single chip. FPGAs are comprehensive devices that combine flexibility and high performance in the same platform compared to other platform such as General-Purpose Processors (GPPs) and Application Specific Integrated Circuits (ASICs). The flexibility of modern FPGAs is further enhanced by introducing Dynamic Partial Reconfiguration (DPR) feature, which allows for changing the functionality of part of the system while other parts are functioning. FPGAs became an important platform for digital image processing applications because of the aforementioned features. They can fulfil the need of efficient and flexible platforms that execute imaging tasks efficiently as well as the reliably with low power, high performance and high flexibility. The use of FPGAs as accelerators for image processing outperforms most of the current solutions. Current FPGA solutions can to load part of the imaging application that needs high computational power on dedicated reconfigurable hardware accelerators while other parts are working on the traditional solution to increase the system performance. Moreover, the use of the DPR feature enhances the flexibility of image processing further by swapping accelerators in and out at run-time. The use of fault mitigation techniques in FPGAs enables imaging applications to operate in harsh environments following the fact that FPGAs are sensitive to radiation and extreme conditions. The aim of this thesis is to present a platform for efficient implementations of imaging tasks. The research uses FPGAs as the key component of this platform and uses the concept of DPR to increase the performance, flexibility, to reduce the power dissipation and to expand the cycle of possible imaging applications. In this context, it proposes the use of FPGAs to accelerate the Image Processing Pipeline (IPP) stages, the core part of most imaging devices. The thesis has a number of novel concepts. The first novel concept is the use of FPGA hardware environment and DPR feature to increase the parallelism and achieve high flexibility. The concept also increases the performance and reduces the power consumption and area utilisation. Based on this concept, the following implementations are presented in this thesis: An implementation of Adams Hamilton Demosaicing algorithm for camera colour interpolation, which exploits the FPGA parallelism to outperform other equivalents. In addition, an implementation of Automatic White Balance (AWB), another IPP stage that employs DPR feature to prove the mentioned novelty aspects. Another novel concept in this thesis is presented in chapter 6, which uses DPR feature to develop a novel flexible imaging system that requires less logic and can be implemented in small FPGAs. The system can be employed as a template for any imaging application with no limitation. Moreover, discussed in this thesis is a novel reliable version of the imaging system that adopts novel techniques including scrubbing, Built-In Self Test (BIST), and Triple Modular Redundancy (TMR) to detect and correct errors using the Internal Configuration Access Port (ICAP) primitive. These techniques exploit the datapath-based nature of the implemented imaging system to improve the system's overall reliability. The thesis presents a proposal for integrating the imaging system with the Robust Reliable Reconfigurable Real-Time Heterogeneous Operating System (R4THOS) to get the best out of the system. The proposal shows the suitability of the proposed DPR imaging system to be used as part of the core system of autonomous cars because of its unbounded flexibility. These novel works are presented in a number of publications as shown in section 1.3 later in this thesis

Camera Pose Estimation from Aerial Images

This thesis demonstrates the applicability of the digital camera as an aerial po sitioning device. The necessary theory behind digital, optical imaging systems and geometrical image formation is presented. In addition, basic image distortions and camera calibration are introduced. However, the main emphasis is on the correspondence problem between two images and on camera pose estimation. The position and orientation of the camera can be estimated relatively to previous known coordinates or absolutely to some reference coordinate system. In relative camera pose estimation, the correspondences between two consecutive images can be recognized from image derivatives. In general, di erential methods are used for low resolution images with high frame rates. For high resolution images, feature-based methods are generally more appropriate. Image features are often detected with subpixel accuracy, and their surroundings are described with feature vectors. These feature vectors are matched between two images to and the pointwise correspondences. The relative translation and orientation of the camera can be estimated from the correspondences. However, the major problem in all relative positioning methods is the error accumulation, where errors from previous estimations are accumulated to further estimations. The error accumulation can be avoided by registering sensed aerial images to previously captured georeferenced images, which coordinates are known for every pixel. In this thesis, image registration between the reference image and an aerial image is implemented manually. Position and orientation of a camera are estimated absolutely to the reference coordinate system. This thesis presents algorithms to solve the correspondence problem and to estimate the relative and absolute position and orientation of an aerial camera. The presented algorithms are verified with virtual Google Earth images and real-lif eaerial images from the test ight. In addition, the performance of the algorithms is also analyzed in terms of noise resistance

Multiprocessor Image-Based Control: Model-Driven Optimisation

Over the last years, cameras have become an integral component of modern cyber-physical systems due to their versatility, relatively low cost and multi-functionality. Camera sensors form the backbone of modern applications like advanced driver assistance systems (ADASs), visual servoing, telerobotics, autonomous systems, electron microscopes, surveillance and augmented reality. Image-based control (IBC) systems refer to a class of data-intensive feedback control systems whose feedback is provided by the camera sensor(s). IBC systems have become popular with the advent of efficient image-processing algorithms, low-cost complementary metal–oxide semiconductor (CMOS) cameras with high resolution and embedded multiprocessor computing platforms with high performance. The combination of the camera sensor(s) and image-processing algorithms can detect a rich set of features in an image. These features help to compute the states of the IBC system, such as relative position, distance, or depth, and support tracking of the object-of-interest. Modern industrial compute platforms offer high performance by allowing parallel and pipelined execution of tasks on their multiprocessors.The challenge, however, is that the image-processing algorithms are compute-intensive and result in an inherent relatively long sensing delay. State-of-the-art design methods do not fully exploit the IBC system characteristics and advantages of the multiprocessor platforms for optimising the sensing delay. The sensing delay of an IBC system is moreover variable with a significant degree of variation between the best-case and worst-case delay due to application-specific image-processing workload variations and the impact of platform resources. A long variable sensing delay degrades system performance and stability. A tight predictable sensing delay is required to optimise the IBC system performance and to guarantee the stability of the IBC system. Analytical computation of sensing delay is often pessimistic due to image-dependent workload variations or challenging platform timing analysis. Therefore, this thesis explores techniques to cope with the long variable sensing delay by considering application-specific IBC system characteristics and exploiting the benefits of the multiprocessor platforms. Effectively handling the long variable sensing delay helps to optimise IBC system performance while guaranteeing IBC system stability

Nanophotonic filters for digital imaging

There has been an increasing demand for low cost, portable CMOS image sensors because of increased integration, and new applications in the automotive, mobile communication and medical industries, amongst others. Colour reproduction remains imperfect in conventional digital image sensors, due to the limitations of the dye-based filters. Further improvement is required if the full potential of digital imaging is to be realised. In alternative systems, where accurate colour reproduction is a priority, existing equipment is too bulky for anything but specialist use. In this work both these issues are addressed by exploiting nanophotonic techniques to create enhanced trichromatic filters, and multispectral filters, all of which can be fabricated on-chip, i.e. integrated into a conventional digital image sensor, to create compact, low cost, mass produceable imaging systems with accurate colour reproduction. The trichromatic filters are based on plasmonic structures. They exploit the excitation of surface plasmon resonances in arrays of subwavelength holes in metal films to filter light. The currently-known analytical expressions are inadequate for optimising all relevant parameters of a plasmonic structure. In order to obtain arbitrary filter characteristics, an automated design procedure was developed that integrated a genetic algorithm and 3D finite-difference time-domain tool. The optimisation procedure's efficacy is demonstrated by designing a set of plasmonic filters that replicate the CIE (1931) colour matching functions, which themselves mimic the human eye's daytime colour response. The best designs were fabricated and demonstrated a least-mean-square error, in comparison to the desired colour matching functions, of 6.37*10^3, 2.34*10^3 and 11.10*10^3 for the red, green, and blue filters respectively. Notably the spectrum for the red filter contained a double peak, as present in the corresponding colour matching function. Such dual peak behaviour cannot be achieved using a single current dye-based filter. The filters retain the same layer thickness for all structures so they can be defined in a single lithography step. A new approach to enable the fabrication of a multispectral filter array on a CMOS imager is also presented. This combines a Fabry-Perot filter with effective medium theory (EMT) to enable the fabrication of multiple filters in a single cavity length via lithographic tuning of the filter passband. Two approaches are proposed; air-filled nanostructures and dielectric backfilled nanostructures. The air-filled approach is demonstrated experimentally producing three filters with FWHM of 60 - 64 nm. Using the backfilled design, and incorporating a highindex cavity material, a set of twenty three narrowband filters, with a FWHM of 22 - 46nm is demonstrated. A virtual image reproduction process was developed to quantify the image reproduction performance of both the plasmonic and Fabry-Perot filter sets. A typical rgb dye-based filter set used in conventional imagers achieves a mean colour error of 2.711, whereas the experimental data from the plasmonic filters achieves an error of 2.222 which demonstrated a slight improvement in colour reproduction. The multispectral filter set developed in this work performed even better, with 4 filters giving an error of 0.906, 10 filters an error of 0.072 and continued improvement in the colour error reaching 0.047 for 23 filters. All the filter sets proposed are fully compatible with the CMOS process so as to enable direct integration onto CMOS image sensors in industrial foundries in future. The performance of the presented filters also suggest new compact applications in art reproduction, agricultural monitoring and medical imaging

Image Color Correction, Enhancement, and Editing

This thesis presents methods and approaches to image color correction, color enhancement, and color editing. To begin, we study the color correction problem from the standpoint of the camera's image signal processor (ISP). A camera's ISP is hardware that applies a series of in-camera image processing and color manipulation steps, many of which are nonlinear in nature, to render the initial sensor image to its final photo-finished representation saved in the 8-bit standard RGB (sRGB) color space. As white balance (WB) is one of the major procedures applied by the ISP for color correction, this thesis presents two different methods for ISP white balancing. Afterwards, we discuss another scenario of correcting and editing image colors, where we present a set of methods to correct and edit WB settings for images that have been improperly white-balanced by the ISP. Then, we explore another factor that has a significant impact on the quality of camera-rendered colors, in which we outline two different methods to correct exposure errors in camera-rendered images. Lastly, we discuss post-capture auto color editing and manipulation. In particular, we propose auto image recoloring methods to generate different realistic versions of the same camera-rendered image with new colors. Through extensive evaluations, we demonstrate that our methods provide superior solutions compared to existing alternatives targeting color correction, color enhancement, and color editing