Increasing temporal, structural, and spectral resolution in images using exemplar-based priors

In the past decade, camera manufacturers have offered smaller form factors, smaller pixel sizes (leading to higher resolution images), and faster processing chips to increase the performance of consumer cameras. However, these conventional approaches have failed to capitalize on the spatio-temporal redundancy inherent in images, nor have they adequately provided a solution for finding $3$D point correspondences for cameras sampling different bands of the visible spectrum. In this thesis, we pose the following question---given the repetitious nature of image patches, and appropriate camera architectures, can statistical models be used to increase temporal, structural, or spectral resolution? While many techniques have been suggested to tackle individual aspects of this question, the proposed solutions either require prohibitively expensive hardware modifications and/or require overly simplistic assumptions about the geometry of the scene. We propose a two-stage solution to facilitate image reconstruction; 1) design a linear camera system that optically encodes scene information and 2) recover full scene information using prior models learned from statistics of natural images. By leveraging the tendency of small regions to repeat throughout an image or video, we are able to learn prior models from patches pulled from exemplar images. The quality of this approach will be demonstrated for two application domains, using low-speed video cameras for high-speed video acquisition and multi-spectral fusion using an array of cameras. We also investigate a conventional approach for finding 3D correspondence that enables a generalized assorted array of cameras to operate in multiple modalities, including multi-spectral, high dynamic range, and polarization imaging of dynamic scenes

Multimedia Forensics

This book is open access. Media forensics has never been more relevant to societal life. Not only media content represents an ever-increasing share of the data traveling on the net and the preferred communications means for most users, it has also become integral part of most innovative applications in the digital information ecosystem that serves various sectors of society, from the entertainment, to journalism, to politics. Undoubtedly, the advances in deep learning and computational imaging contributed significantly to this outcome. The underlying technologies that drive this trend, however, also pose a profound challenge in establishing trust in what we see, hear, and read, and make media content the preferred target of malicious attacks. In this new threat landscape powered by innovative imaging technologies and sophisticated tools, based on autoencoders and generative adversarial networks, this book fills an important gap. It presents a comprehensive review of state-of-the-art forensics capabilities that relate to media attribution, integrity and authenticity verification, and counter forensics. Its content is developed to provide practitioners, researchers, photo and video enthusiasts, and students a holistic view of the field

Blickpunktabhängige Computergraphik

Contemporary digital displays feature multi-million pixels at ever-increasing refresh rates. Reality, on the other hand, provides us with a view of the world that is continuous in space and time. The discrepancy between viewing the physical world and its sampled depiction on digital displays gives rise to perceptual quality degradations. By measuring or estimating where we look, gaze-contingent algorithms aim at exploiting the way we visually perceive to remedy visible artifacts. This dissertation presents a variety of novel gaze-contingent algorithms and respective perceptual studies. Chapter 4 and 5 present methods to boost perceived visual quality of conventional video footage when viewed on commodity monitors or projectors. In Chapter 6 a novel head-mounted display with real-time gaze tracking is described. The device enables a large variety of applications in the context of Virtual Reality and Augmented Reality. Using the gaze-tracking VR headset, a novel gaze-contingent render method is described in Chapter 7. The gaze-aware approach greatly reduces computational efforts for shading virtual worlds. The described methods and studies show that gaze-contingent algorithms are able to improve the quality of displayed images and videos or reduce the computational effort for image generation, while display quality perceived by the user does not change.Moderne digitale Bildschirme ermöglichen immer höhere Auflösungen bei ebenfalls steigenden Bildwiederholraten. Die Realität hingegen ist in Raum und Zeit kontinuierlich. Diese Grundverschiedenheit führt beim Betrachter zu perzeptuellen Unterschieden. Die Verfolgung der Aug-Blickrichtung ermöglicht blickpunktabhängige Darstellungsmethoden, die sichtbare Artefakte verhindern können. Diese Dissertation trägt zu vier Bereichen blickpunktabhängiger und wahrnehmungstreuer Darstellungsmethoden bei. Die Verfahren in Kapitel 4 und 5 haben zum Ziel, die wahrgenommene visuelle Qualität von Videos für den Betrachter zu erhöhen, wobei die Videos auf gewöhnlicher Ausgabehardware wie z.B. einem Fernseher oder Projektor dargestellt werden. Kapitel 6 beschreibt die Entwicklung eines neuartigen Head-mounted Displays mit Unterstützung zur Erfassung der Blickrichtung in Echtzeit. Die Kombination der Funktionen ermöglicht eine Reihe interessanter Anwendungen in Bezug auf Virtuelle Realität (VR) und Erweiterte Realität (AR). Das vierte und abschließende Verfahren in Kapitel 7 dieser Dissertation beschreibt einen neuen Algorithmus, der das entwickelte Eye-Tracking Head-mounted Display zum blickpunktabhängigen Rendern nutzt. Die Qualität des Shadings wird hierbei auf Basis eines Wahrnehmungsmodells für jeden Bildpixel in Echtzeit analysiert und angepasst. Das Verfahren hat das Potenzial den Berechnungsaufwand für das Shading einer virtuellen Szene auf ein Bruchteil zu reduzieren. Die in dieser Dissertation beschriebenen Verfahren und Untersuchungen zeigen, dass blickpunktabhängige Algorithmen die Darstellungsqualität von Bildern und Videos wirksam verbessern können, beziehungsweise sich bei gleichbleibender Bildqualität der Berechnungsaufwand des bildgebenden Verfahrens erheblich verringern lässt

Development of novel multimodal light-sheet fluorescence microscopes for in-vivo imaging of vertebrate organisms

The observation of biological processes in their native environments is of critical importance for life science. While substantial information can be derived from the examination of in-vitro biological samples, in-vivo studies are necessary to reveal the complexity of the dynamics happening in real-time within a living organism. Between the possible biological model choices, vertebrates represent an important family due to the various characteristics they share with the human organism. The development of an embryo, the effect of a drug, the interaction between the immune system and pathogens, and the cellular machinery activities are all examples of highly-relevant applications requiring in-vivo observations on broadly used vertebrate models such as the zebrafish and the mouse. To perform such observations, appropriate devices have been devised. Fluorescence microscopy is one of the main approaches through which specific sample structures can be detected and registered in high-contrast images. Through micro-injections or transgenic lines, a living specimen can express fluorescence and can be imaged through such microscopes. Various fluorescence microscopy techniques have been developed, such as Widefield Microscopy (WM) and Laser Scanning Confocal Microscopy (LSCM). In WM the entire sample is visualized in a single 2D image, therefore losing the depth information, while LSCM can recover the 3D information of the sample but with inherent limitations, such as phototoxicity and limited imaging speed. In the last two decades, Light-Sheet Fluorescence Microscopy (LSFM) emerged as a technique providing fast and 3D imaging, while minimizing collateral damages to the specimen. However, due to the particular configuration of the microscope’s components, LSFM setups are normally optimized for a single application. Also, sample management is not trivial, as controlling the specimen positioning and keeping it alive for a long time within the microscope needs dedicated environmental conditioning. In this thesis, I aimed at advancing the imaging flexibility of LSFM, with particular attention to sample management. The conjugation of these aspects enabled novel observations and applications on living vertebrate samples. In Chapter 1, a brief review of the concepts employed within this thesis is presented, also pointing to the main challenges that the thesis aims to solve. In Chapter 2, a new design for multimodal LSFM is presented, which enables performing different experiments with the same instrument. Particularly, high-throughput studies would benefit from this imaging paradigm, conjugating the need for fast and reproducible mounting of multiple samples with the opportunity to image them in 3D. Additionally, from this design, a transportable setup has also been implemented. With these systems, I studied the dynamics of the yolk’s microtubule network of zebrafish embryos, describing novel features and underlining the importance of live imaging to have a whole view of the sample’s peculiarities. This is described in Chapter 3. Further applications on challenging live samples have been implemented, monitoring the macrophage recruitment in zebrafish and the development of mouse embryos. For these applications, described in Chapter 4, I devised specific mounting protocols for the samples, keeping them alive during the imaging sessions. In Chapter 5, an additional LSFM system is described, which allows for recording the sub-cellular machinery in a living vertebrate sample, while avoiding its damage thanks to the devised sample mounting. Through this, single-molecule microscopy (SMM) studies, normally performed on cultured cells, can be extended to the nuclei of living zebrafish embryos, which better recapitulate the native environment where biological processes take place. Finally, Chapter 6 recapitulates the conclusions, the impacts, future integrations, and experimental procedures that would be enabled by the work resumed in this thesis.La observación de los procesos biológicos en su entorno es de vital importancia para las ciencias de la vida. Si bien se puede derivar información sustancial desde muestras biológicas in-vitro, los estudios in-vivo son necesarios para revelar la complejidad de la dinámica que ocurre, en tiempo real, dentro de un organismo vivo. Entre las posibles elecciones de modelos biológicos, los vertebrados representan una familia importante debido a las diversas características que comparten con el organismo humano. El desarrollo de un embrión, la interacción entre el sistema inmunitario y los patógenos, el efecto de un fármaco y las actividades celulares son ejemplos de aplicaciones que requieren observaciones in-vivo en modelos de vertebrados, como el pez cebra y el ratón. La microscopía de fluorescencia es uno de los principales métodos mediante los cuales se pueden grabar imágenes, de alto contraste, de estructuras biológicas específicas. Utilizando microinyecciones o líneas transgénicas, es posible inducir una expresión de proteínas fluorescentes en la muestra y entonces puede ser observada a través de dichos microscopios. Existen varias técnicas de microscopía de fluorescencia, entre ellas las más utilizadas son la microscopía ¿widefield¿ (WM) y la microscopía ¿confocal¿ (LSCM). En WM, una sola imagen en 2D representa el volumen entero de la muestra, por lo cual la información de profundidad se pierde. Por otro lado, LSCM puede recuperar la información en 3D con algunas limitaciones como la fototoxicidad y una velocidad de generación de las imágenes limitada. En las últimas dos décadas, la microscopía de fluorescencia de hoja de luz (LSFM) surgió como técnica que ofrece imágenes de manera rápidas y en 3D, y que al mismo tiempo minimiza los daños colaterales de la muestra. Sin embargo, debido a la geometría de los componentes del microscopio, las configuraciones de LSFM normalmente se optimizan para una sola aplicación. Además, la gestión de las muestras no es trivial, ya que controlar su posición y mantenerlas vivas durante largos periodos de tiempo dentro del microscopio requiere una atención especifica. En esta tesis, me propuse mejorar la versatilidad que LSFM puede ofrecer, con especial atención a la gestión de muestras vivas. La conjugación de estos aspectos permitió nuevas observaciones y nuevas aplicaciones en vertebrados vivos. En el Capítulo 1, se presenta un breve resumen de los conceptos empleados dentro de esta tesis, señalando también los principales desafíos que la tesis pretende resolver. En el Capítulo 2, se presenta un nuevo diseño para un LSFM multimodal, que permite realizar diferentes experimentos con el mismo instrumento. Los estudios de High-Throughput se beneficiarían de este diseño, ya que conjuga la necesidad de un montaje rápido y reproducible de varias muestras con las ventajas de LSFM. Además, a partir de este diseño, también se ha desarrollado un otro microscopio LSFM transportable. Con estos sistemas, se estudió la dinámica de la red de microtúbulos en embriones de pez cebra, describiendo características nuevas y acentuando la importancia de los experimentos in-vivo para obtener una visión completa de la muestra. Esto se describe en el Capítulo 3. Para realizar otras aplicaciones, como la observación de la dinámica de macrófagos en el pez cebra y del desarrollo de embriones de ratón, descritas en el Capítulo 4, se establecieron protocolos de montaje específicos para las muestras, manteniéndolas vivas durante las sesiones experimentales. En el Capítulo 5, se describe otro sistema LSFM, que permite extender los estudios de microscopía de moléculas individuales (SMM), normalmente realizados en cultivos de células, a núcleos de embriones de pez cebra vivos, que recrean mejor el entorno natural de los procesos biológicos. Finalmente, el Capítulo 6 recapitula las conclusiones, los impactos, las integraciones futuras y los procedimientos experimentales que serían posibilitados por el trabajo resumido en esta tesis.Postprint (published version