Fusion of interpolated frames superresolution in the presence of atmospheric optical turbulence

An extension of the fusion of interpolated frames superresolution (FIF SR) method to perform SR in the presence of atmospheric optical turbulence is presented. The goal of such processing is to improve the performance of imaging systems impacted by turbulence. We provide an optical transfer function analysis that illustrates regimes where significant degradation from both aliasing and turbulence may be present in imaging systems. This analysis demonstrates the potential need for simultaneous SR and turbulence mitigation (TM). While the FIF SR method was not originally proposed to address this joint restoration problem, we believe it is well suited for this task. We propose a variation of the FIF SR method that has a fusion parameter that allows it to transition from traditional diffraction-limited SR to pure TM with no SR as well as a continuum in between. This fusion parameter balances subpixel resolution, needed for SR, with the amount of temporal averaging, needed for TM and noise reduction. In addition, we develop a model of the interpolation blurring that results from the fusion process, as a function of this tuning parameter. The blurring model is then incorporated into the overall degradation model that is addressed in the restoration step of the FIF SR method. This innovation benefits the FIF SR method in all applications. We present a number of experimental results to demonstrate the efficacy of the FIF SR method in different levels of turbulence. Simulated imagery with known ground truth is used for a detailed quantitative analysis. Three real infrared image sequences are also used. Two of these include bar targets that allow for a quantitative resolution enhancement assessment

Holoscopic 3D image depth estimation and segmentation techniques

This thesis was submitted for the award of Doctor of Philosophy and was awarded by Brunel University LondonToday’s 3D imaging techniques offer significant benefits over conventional 2D imaging techniques. The presence of natural depth information in the scene affords the observer an overall improved sense of reality and naturalness. A variety of systems attempting to reach this goal have been designed by many independent research groups, such as stereoscopic and auto-stereoscopic systems. Though the images displayed by such systems tend to cause eye strain, fatigue and headaches after prolonged viewing as users are required to focus on the screen plane/accommodation to converge their eyes to a point in space in a different plane/convergence. Holoscopy is a 3D technology that targets overcoming the above limitations of current 3D technology and was recently developed at Brunel University. This work is part W4.1 of the 3D VIVANT project that is funded by the EU under the ICT program and coordinated by Dr. Aman Aggoun at Brunel University, West London, UK. The objective of the work described in this thesis is to develop estimation and segmentation techniques that are capable of estimating precise 3D depth, and are applicable for holoscopic 3D imaging system. Particular emphasis is given to the task of automatic techniques i.e. favours algorithms with broad generalisation abilities, as no constraints are placed on the setting. Algorithms that provide invariance to most appearance based variation of objects in the scene (e.g. viewpoint changes, deformable objects, presence of noise and changes in lighting). Moreover, have the ability to estimate depth information from both types of holoscopic 3D images i.e. Unidirectional and Omni-directional which gives horizontal parallax and full parallax (vertical and horizontal), respectively. The main aim of this research is to develop 3D depth estimation and 3D image segmentation techniques with great precision. In particular, emphasis on automation of thresholding techniques and cues identifications for development of robust algorithms. A method for depth-through-disparity feature analysis has been built based on the existing correlation between the pixels at a one micro-lens pitch which has been exploited to extract the viewpoint images (VPIs). The corresponding displacement among the VPIs has been exploited to estimate the depth information map via setting and extracting reliable sets of local features. ii Feature-based-point and feature-based-edge are two novel automatic thresholding techniques for detecting and extracting features that have been used in this approach. These techniques offer a solution to the problem of setting and extracting reliable features automatically to improve the performance of the depth estimation related to the generalizations, speed and quality. Due to the resolution limitation of the extracted VPIs, obtaining an accurate 3D depth map is challenging. Therefore, sub-pixel shift and integration is a novel interpolation technique that has been used in this approach to generate super-resolution VPIs. By shift and integration of a set of up-sampled low resolution VPIs, the new information contained in each viewpoint is exploited to obtain a super resolution VPI. This produces a high resolution perspective VPI with wide Field Of View (FOV). This means that the holoscopic 3D image system can be converted into a multi-view 3D image pixel format. Both depth accuracy and a fast execution time have been achieved that improved the 3D depth map. For a 3D object to be recognized the related foreground regions and depth information map needs to be identified. Two novel unsupervised segmentation methods that generate interactive depth maps from single viewpoint segmentation were developed. Both techniques offer new improvements over the existing methods due to their simple use and being fully automatic; therefore, producing the 3D depth interactive map without human interaction. The final contribution is a performance evaluation, to provide an equitable measurement for the extent of the success of the proposed techniques for foreground object segmentation, 3D depth interactive map creation and the generation of 2D super-resolution viewpoint techniques. The no-reference image quality assessment metrics and their correlation with the human perception of quality are used with the help of human participants in a subjective manner

3D high resolution techniques applied on small and medium size objects: from the analysis of the process towards quality assessment

The need for metric data acquisition is an issue strictly related to the human capability of describing the world with rigorous and repeatable methods. From the invention of photography to the development of advanced computers, the metric data acquisition has been subjected to rapid mutation, and nowadays there exists a strict connection between metric data acquisition and image processing, Computer Vision and Artificial Intelligence. The sensor devices for the 3D model generation are various and characterized by different functioning principles. In this work, optical passive and active sensors are treated, focusing specifically on close-range photogrammetry, Time of Flight (ToF) sensors and Structured-light scanners (SLS). Starting from the functioning principles of the techniques and showing some issues related to them, the work highlights their potentialities, analyzing the fundamental and most critical steps of the process leading to the quality assessment of the data. Central themes are the instruments calibration, the acquisition plan and the interpretation of the final results. The capability of the acquisition techniques to satisfy unconventional requirements in the field of Cultural Heritage is also shown. The thesis starts with an overview about the history and developments of 3D metric data acquisition. Chapter 1 treats the Human Vision System and presents a complete overview of 3D sensing devices. Chapter 2 starts from the enunciation of the basic principle of close-range photogrammetry considering digital cameras functioning principles, calibration issues, and the process leading to the 3D mesh reconstruction. The case of multi-image acquisition is analyzed, deepening the quality assessment of the photogrammetric process through a case study. Chapter 3 is devoted to the range-based acquisition techniques, namely ToF laser scanners and SLSs. Lastly, Chapter 4 focuses on unconventional applications of the mentioned high-resolution acquisition techniques showing some examples of study cases in the field of Cultural Heritage

Super-resolution imaging of cell-surface Sonic hedgehog multimolecular signalling complexes

Sonic hedgehog is a fascinating protein with great responsibility over the formation and upkeep of our bodies. It is widely studied, not least because dysregulation of the Shh signalling pathway leads to repercussions on human health, such as contraction of cancer. Gaining an understanding of its signalling mechanism is central to inventing preventative measures and treatments against this disease. This thesis focuses on the study of the spatial organisation of Shh multimolecular signalling complexes on the surface of producing cells, and those dispatched in the vicinity of those cells, using high-resolution optical imaging beyond the diffraction limit. With un-precedented resolution, the differences in organisation of Shh pre- and post-release from the surface were characterised, and the influence of the lipid modifications of Shh, namely choles-terol and palmitate, investigated. The main findings were that both lipid adducts are necessary for large-scale multimerisation, but not for the formation of small, sub-diffraction limit oligomers. Together with data I collected about the profile of the clusters’ size distributions, I find that electrostatic interactions between the molecules may be the engine driving the multimerisation process. Furthermore, the role of lipid modifications may, at least in part, be to retain Shh on the surface while multimerisation proceeding according to the law of mass action builds upon the small oligomer nucleation sites prepared presumably by the electrostatic interactions in the first place. Other, more indirect lines of evidence again based on the profile of the multimer size distribution insinuated that Shh complexes may not undergo any proteolytic modifications prior to release – contrary to some reports in the literature. The results presented in this thesis are the fruits of a completely fresh and innovative approach to examining Shh, which for the first time delivers concrete dimensional details about the elusive structure of the Shh multimer.Open Acces

Treatise on Hearing: The Temporal Auditory Imaging Theory Inspired by Optics and Communication

A new theory of mammalian hearing is presented, which accounts for the auditory image in the midbrain (inferior colliculus) of objects in the acoustical environment of the listener. It is shown that the ear is a temporal imaging system that comprises three transformations of the envelope functions: cochlear group-delay dispersion, cochlear time lensing, and neural group-delay dispersion. These elements are analogous to the optical transformations in vision of diffraction between the object and the eye, spatial lensing by the lens, and second diffraction between the lens and the retina. Unlike the eye, it is established that the human auditory system is naturally defocused, so that coherent stimuli do not react to the defocus, whereas completely incoherent stimuli are impacted by it and may be blurred by design. It is argued that the auditory system can use this differential focusing to enhance or degrade the images of real-world acoustical objects that are partially coherent. The theory is founded on coherence and temporal imaging theories that were adopted from optics. In addition to the imaging transformations, the corresponding inverse-domain modulation transfer functions are derived and interpreted with consideration to the nonuniform neural sampling operation of the auditory nerve. These ideas are used to rigorously initiate the concepts of sharpness and blur in auditory imaging, auditory aberrations, and auditory depth of field. In parallel, ideas from communication theory are used to show that the organ of Corti functions as a multichannel phase-locked loop (PLL) that constitutes the point of entry for auditory phase locking and hence conserves the signal coherence. It provides an anchor for a dual coherent and noncoherent auditory detection in the auditory brain that culminates in auditory accommodation. Implications on hearing impairments are discussed as well.Comment: 603 pages, 131 figures, 13 tables, 1570 reference

Coupled Plasmonic Nanostructures Based on Core-Shell Particles

Plasmonic nanoparticles feature remarkable optical and electronic properties in consequence of the excitation of conduction band electrons by visible light, which leads to collective oscillations. This so called localized surface plasmon resonance (LSPR) is utilized in the fields of photovoltaics, sensing, catalysis and optoelectronics. Especially, the emergence of optical metasurfaces—subwavelength structured surfaces with properties typically not occurring for homogeneous materials—has attracted significant attention for the applications mentioned above. However, their fabrication is usually complex and the materials often lack in situ tunability. Here, a colloidal approach is demonstrated for the preparation of optical metasurfaces with tunable properties. They are based on plasmonic gold nanoparticles, which were coated with three different shell materials to provide three different functionalities when coupled to plasmonic mirrors: i) Dye-labeled silica coatings exhibit strong enhancement of their fluorescent properties, as shown in this extensive single particle study. ii) Hydrogel shells are applied to receive switchable electric and magnetic properties in response to swelling of the gel. iii) Electrochromic polymer coatings facilitate the preparation of anti-reflective metasurfaces that feature tunable efficiency by changing the pH or applying a voltage. In addition, mechano-tunable plasmonic lattices are demonstrated. The material is based on self-assembled gold nanoparticles, which are embedded in a transparent elastomer matrix and feature pronounced surface lattice resonances (SLR). These tunable resonances could be applied for lasing, strain sensing, or controlling catalytic reactions.Plasmonische Nanopartikel besitzen bemerkenswerte optische und elektronische Eigenschaften, die sie für Anwendungen in Bereichen der Katalyse, Sensorik, Optoelektronik, sowie der Nanooptik prädestinieren. Ihre Eigenschaften beruhen auf der Anregung von Leitungsbandelektronen zu kollektiven Oszillationen durch sichtbares Licht. Diese sogenannte Oberflächenplasmonenresonanz ist insbesondere für optische Metaoberflächen von Interesse, also Materialien mit strukturierten Oberflächen im Größenbereich unterhalb der sichtbaren Wellenlängen, welche Charakteristika aufweisen, die bei homogenen Materialien typischerweise nicht auftreten. Sie werden allerdings häufig mit aufwendigen Methoden hergestellt und sind in situ nicht justierbar. In dieser Arbeit werden kolloidale Ansätze zur Herstellung plasmonischer Metaoberflächen mit einstellbaren optischen und elektronischen Eigenschaften vorgestellt. Das Konzept basiert auf der Verwendung von plasmonischen Goldkernen, die mit drei unterschiedlichen funktionellen Schalen beschichtet und anschließend mit plasmonischen Spiegeln gekoppelt wurden: i) Farbstoffmarkierte Silicapartikel zeigen starke Fluoreszenz-verstärkung, wie in dieser ausführlichen Einzelpartikelstudie nachgewiesen wird. ii) Hydrogelbeschichtungen werden verwendet um schaltbare elektrische und magnetische Eigenschaften mittels Quellung zu erzeugen. iii) Elektrochrome Polymerhüllen fungieren als Antireflexschicht auf Goldoberflächen, deren Extinktion sich mittels Anlegen einer Spannung oder durch pH-Änderungen einstellen lässt. Neben diesen Ansätzen werden mechanisch einstellbare plasmonische Gitterstrukturen vorgestellt. Die selbstassemblierten und in transparentem Elastomer eingebetteten Goldnanopartikel weisen eine ausgeprägte Oberflächengitterresonanz auf. Diese kann für sensorische Zwecke in den Bereichen der Mikromechanik und der Katalyse, sowie für abstimmbare Laser verwendet werden

Realistic visualisation of cultural heritage objects

This research investigation used digital photography in a hemispherical dome, enabling a set of 64 photographic images of an object to be captured in perfect pixel register, with each image illuminated from a different direction. This representation turns out to be much richer than a single 2D image, because it contains information at each point about both the 3D shape of the surface (gradient and local curvature) and the directionality of reflectance (gloss and specularity). Thereby it enables not only interactive visualisation through viewer software, giving the illusion of 3D, but also the reconstruction of an actual 3D surface and highly realistic rendering of a wide range of materials. The following seven outcomes of the research are claimed as novel and therefore as representing contributions to knowledge in the field: A method for determining the geometry of an illumination dome; An adaptive method for finding surface normals by bounded regression; Generating 3D surfaces from photometric stereo; Relationship between surface normals and specular angles; Modelling surface specularity by a modified Lorentzian function; Determining the optimal wavelengths of colour laser scanners; Characterising colour devices by synthetic reflectance spectra

Non contact surface metrology in a hazardous environment

The EFDA-JET tokamak is an experimental fusion device researching fusion as a means of energy production. Inside the toroidal vessel, plasma with temperature in excess of 100 million degrees Celsius is generated and constrained by high power magnetic fields. Additional protection is provided by tiles which clad the inside of the machine. As part of a major upgrade existing heat protective tiles are to be replaced with an advanced design, and renewed interest has been shown in dimensional measurement of the surface. Measurement must occur during shutdown periods where temperature and pressure are at ambient levels. Manned entry is not permissible and all work should be performed remotely. To avoid contamination which could affect the fusion reaction and experimental results, contact with the measurement surface is not permitted. This work assesses non-contact surface measurement technologies, along with standards and guidelines for dimensional surface measurement. Existing measurement test artefacts do not offer the required surface finish and features, so specific test artefacts have been designed and produced. These artefacts are traceable to the national length standard, as traceability is a pre-requisite to evaluate accuracy. Exploratory tests highlighted two technologies for further investigation, laser triangulation and white light fringe projection. Two commercially available, state-ofthe- art examples of each technology have been evaluated using a processing method developed to highlight performance in key areas relevant to EFDA-JET. These areas include quantitative assessments of the effect of surface angle on measurement quality, the effect of depth of field for fringe projection systems and the ability of technologies to record gap and flush from tens of micrometres to millimetres. Tests enable a user to begin to assess the impact the measurement system has on the measurement result, how different technologies and systems used alone or in combination may resolve or compound erroneous results, clarifying or disrupting the meaning of results

Development of a Nano-Illumination Microscope

[eng] This doctoral thesis proposes and explores a new approach to lensless microscopy, focusing on making high resolution imaging ubiquitous and low cost. A short introduction to microscopy frames the state of current techniques: Abbe’s law limits the resolving power for visible light microscopes with lenses, techniques using UV, X-rays, or electrons are incompatible with live samples and all of them, including super-resolution microscopy methods, are complex devices not suitable for being used in the field as mobile devices. Some lensless microscopy methods try to solve these issues. The microscopy method is named Nano Illumination Microscopy (NIM) because it relies on using nanometric light sources in an ordered array to illuminate a sample placed in close proximity to them, and a photodetector at the other side to measure the amount of light arriving from each LED. In a setup like this, the resolving power is provided by the nano-LEDs and their distribution instead of the sensing devices, as is the case in the other methods. Since the resolving power depends on the pitch of the LED array, this method also opens a path to obtain super-resolution images, depending only on obtaining LED arrays with pitches smaller than Abbe’s limit for the wavelength. After the introduction to microscopy setting the context of the thesis, the thesis continues explaining the main components used to build the microscope: a SPAD camera, designed within the context of this work, and the electronics to control the nano-LED array. The third chapter of this thesis provides an overview of the microscopy method and its fundaments, exploring the requirements and capabilities. Image formation is first introduced with simulations, and this information is then used to build the very first prototype, a microscope capable of forming 8x8 pixel images -since that is the form factor of the LED array used, with LEDs of 5 μm in size (and 10 μm in pitch). The first results from this technique are presented and compared with the simulations, showing the agreement between both, validating the method, and offering insight on building the next prototypes, which will use smaller LEDs in an attempt to study the technological limits. The thesis continues with the work done in search of the limits of the technique, building and testing new improved versions of the microscope and confronting the limitations which arise. Some of those came from the structure of the LED arrays themselves: while nano-LEDs well below the sizes used have been reported, those have been isolated structures or non individually addressable. Selecting exactly which LED will emit is one of the main problems to solve since with increasingly large arrays, the connections required increase exponentially until routing is impossible. The thesis also studies this problem, as the LED arrays were changed in search of the proper solution. This implied moving from a direct addressing strategy, in which each LED was selected individually, towards a matrix-addressing format, in which the LEDs are selected by polarising the appropriate row and columns. The microscopy technique is validated and the more advanced prototypes presented. Images with a maximum resolving power of 800 nm are shown, and the results discussed, since the physical limitations on fabricating the chips limit the maximum resolving power below what was theoretically expected. The thesis also offers a short overview into the future of the Nano Illumination Microscopy technique.[cat] Aquesta tesi doctoral proposa i explora una nova aproximació a la microscopia sense lents, amb la intenció de facilitar l’obtenció d’imatges d’alta resolució amb baix cost i disponible arreu. S’ha batejat aquest mètode de microscòpia com a Microscopia de Nano-Il·luminació (MNI) perquè la imatge es construeix a partir de fonts de llum de mida nanomètrica distribuïdes en una matriu que il·luminen la mostra de forma consecutiva i ordenada. Un sensor a l’altre costat recull la intensitat de llum que arriba de cada LED, creant un mapa de l’objecte observat. Aquest mètode fa que la resolució de les imatges depengui de la mida i distribució dels LEDs, en comptes de la del sensor com és el cas convencionalment, obrint la porta a noves integracions. En la tesi s’ofereix una introducció general a la microscòpia abans d’entrar a detallar els components del microscopi i com s’integren per muntar-lo. A continuació es presenta i s’estudia el funcionament del mètode, començant amb simulacions i seguint amb la construcció del primer prototip de microscopi amb el que s’obtenen les primeres imatges. La tesi procedeix a continuació a investigar els límits actuals de la tècnica de microscòpia, utilitzant noves versions de la matriu de LEDs i estratègies alternatives per intentar superar-ne les complicacions tècniques. Així, s’obtenen imatges amb una resolució de 800 nm i es discuteix la problemàtica d’implementar dispositius que s’aproximin a les expectatives teòriques per la tècnica