Real-Time Computational Gigapixel Multi-Camera Systems

The standard cameras are designed to truthfully mimic the human eye and the visual system. In recent years, commercially available cameras are becoming more complex, and offer higher image resolutions than ever before. However, the quality of conventional imaging methods is limited by several parameters, such as the pixel size, lens system, the diffraction limit, etc. The rapid technological advancements, increase in the available computing power, and introduction of Graphics Processing Units (GPU) and Field-Programmable-Gate-Arrays (FPGA) open new possibilities in the computer vision and computer graphics communities. The researchers are now focusing on utilizing the immense computational power offered on the modern processing platforms, to create imaging systems with novel or significantly enhanced capabilities compared to the standard ones. One popular type of the computational imaging systems offering new possibilities is a multi-camera system. This thesis will focus on FPGA-based multi-camera systems that operate in real-time. The aim of themulti-camera systems presented in this thesis is to offer a wide field-of-view (FOV) video coverage at high frame rates. The wide FOV is achieved by constructing a panoramic image from the images acquired by the multi-camera system. Two new real-time computational imaging systems that provide new functionalities and better performance compared to conventional cameras are presented in this thesis. Each camera system design and implementation are analyzed in detail, built and tested in real-time conditions. Panoptic is a miniaturized low-cost multi-camera system that reconstructs a 360 degrees view in real-time. Since it is an easily portable system, it provides means to capture the complete surrounding light field in dynamic environment, such as when mounted on a vehicle or a flying drone. The second presented system, GigaEye II , is a modular high-resolution imaging system that introduces the concept of distributed image processing in the real-time camera systems. This thesis explains in detail howsuch concept can be efficiently used in real-time computational imaging systems. The purpose of computational imaging systems in the form of multi-camera systems does not end with real-time panoramas. The application scope of these cameras is vast. They can be used in 3D cinematography, for broadcasting live events, or for immersive telepresence experience. The final chapter of this thesis presents three potential applications of these systems: object detection and tracking, high dynamic range (HDR) imaging, and observation of multiple regions of interest. Object detection and tracking, and observation of multiple regions of interest are extremely useful and desired capabilities of surveillance systems, in security and defense industry, or in the fast-growing industry of autonomous vehicles. On the other hand, high dynamic range imaging is becoming a common option in the consumer market cameras, and the presented method allows instantaneous capture of HDR videos. Finally, this thesis concludes with the discussion of the real-time multi-camera systems, their advantages, their limitations, and the future predictions

Tomografia estendida : do básico até o mapeamento de cérebro de camundongos

Orientador: Mateus Borba CardosoTese (doutorado) - Universidade Estadual de Campinas, Instituto de Física Gleb WataghinResumo: Esta tese apresentará uma introdução a imagens de raios-x e como adquirir e processar imagens usando linhas de luz síncrotron. Apresentará os desafios matemáticos e técnicos para reconstruir amostras em três dimensões usando a reconstrução de Tomografia Computadorizada, uma técnica conhecida como CT. Esta técnica tem seu campo de visão limitado ao tamanho da câmera e ao tamanho da iluminação. Uma técnica para ampliar esse campo de visão vai ser apresentada e os desafios técnicos envolvidos para que isso aconteça. Um \textit{pipeline} é proposto e todos os algoritmos necessários foram empacotados em um pacote python chamado Tomosaic. A abordagem baseia-se em adquirir tomogramas parciais em posiçoes pré definidas e depois mesclar os dados em um novo conjunto de dados. Duas maneiras possíveis são apresentadas para essa mescla, uma no domínio das projeções e uma no domínio dos sinogramas. Experimentos iniciais serão então usadas para mostrar que o método proposto funciona com computadores normais. A técnica será aplicada mais tarde para pesquisar a anatomia de cérebros de camundongo completos. Um estudo será apresentado de como obter informação em diferentes escalas do cérebro completo do rato utilizando raios-xAbstract: This thesis will present an introduction to x-ray images and how to acquire and thread images using synchrotron beamlines. It will present the mathematical and technical challenges to reconstruct samples in three dimensions using Computed Tomography reconstruction, a technique known as CT. This technique has a field of view bounded to the camera size and the illumination size. A technique to extended this field of view is going to be presented and the technical challenges involved in order for that to happen will be described. A pipeline is proposed and all the necessary algorithms are contained into a python packaged called Tomosaic. The approach relies on acquired partial tomogram data in a defined grid and later merging the data into a new dataset. Two possible ways are presented in order to that: in the projection domain, and in the sinogram domain. Initial experiments will then be used to show that the pipeline works with normal computers. The technique will be later applied to survey the whole anatomy of whole mouse brains. A study will be shown of how to get the complete range of scales of the mouse brain using x-ray tomography at different resolutionsDoutoradoFísicaDoutor em Ciências163304/2013-01247445/2013, 1456912/2014CNPQCAPE

Real-time Panorama Stitching using a Single PTZ-Camera without using Image Feature Matching

In surveillance applications one thing to consider is how much of a scene one can cover with a camera. One way to augment this is to take images with overlap and blend them, creating a new image with bigger field of view and thereby increase the scene coverage. In this thesis work we have been looking at how one can create panorama images with a pan-tilt-camera and how fast it can be done. We chose a circular panorama representation for this. Our approach was that gathering enough metadata from the camera one can rectify the gathered images and blend them without matching feature-points or other computationally heavy operations. We show that this can be done. The images gathered was corrected for lens distortions and rolling shutter effects arising from rotating the camera. Attempts where made to find an optimal path for the camera to follow while capturing images. An algorithm to do intensity corrections of the images was also implemented. We find that one can rotate the camera at high speeds and still produce a good quality panorama image. The limiting factors are the precision of the meta data gathered, like motion data from the on-board gyro, and the lighting conditions, since a short shutter time is required to minimize motion blur. The quality varies depending on the time taken to capture the images needed to create the spherical projection. The fastest run was done in 1.6 seconds with some distortions. A run in around 4 seconds generally produce a good quality panorama image

Development Of A High Performance Mosaicing And Super-Resolution Algorithm

In this dissertation, a high-performance mosaicing and super-resolution algorithm is described. The scale invariant feature transform (SIFT)-based mosaicing algorithm builds an initial mosaic which is iteratively updated by the robust super resolution algorithm to achieve the final high-resolution mosaic. Two different types of datasets are used for testing: high altitude balloon data and unmanned aerial vehicle data. To evaluate our algorithm, five performance metrics are employed: mean square error, peak signal to noise ratio, singular value decomposition, slope of reciprocal singular value curve, and cumulative probability of blur detection. Extensive testing shows that the proposed algorithm is effective in improving the captured aerial data and the performance metrics are accurate in quantifying the evaluation of the algorithm

Computational Models of Perceptual Organization and Bottom-up Attention in Visual and Audio-Visual Environments

Figure Ground Organization (FGO) - inferring spatial depth ordering of objects in a visual scene - involves determining which side of an occlusion boundary (OB) is figure (closer to the observer) and which is ground (further away from the observer). Attention, the process that governs how only some part of sensory information is selected for further analysis based on behavioral relevance, can be exogenous, driven by stimulus properties such as an abrupt sound or a bright flash, the processing of which is purely bottom-up; or endogenous (goal-driven or voluntary), where top-down factors such as familiarity, aesthetic quality, etc., determine attentional selection. The two main objectives of this thesis are developing computational models of: (i) FGO in visual environments; (ii) bottom-up attention in audio-visual environments. In the visual domain, we first identify Spectral Anisotropy (SA), characterized by anisotropic distribution of oriented high frequency spectral power on the figure side and lack of it on the ground side, as a novel FGO cue, that can determine Figure/Ground (FG) relations at an OB with an accuracy exceeding 60%. Next, we show a non-linear Support Vector Machine based classifier trained on the SA features achieves an accuracy close to 70% in determining FG relations, the highest for a stand-alone local cue. We then show SA can be computed in a biologically plausible manner by pooling the Complex cell responses of different scales in a specific orientation, which also achieves an accuracy greater than or equal to 60% in determining FG relations. Next, we present a biologically motivated, feed forward model of FGO incorporating convexity, surroundedness, parallelism as global cues and SA, T-junctions as local cues, where SA is computed in a biologically plausible manner. Each local cue, when added alone, gives statistically significant improvement in the model's performance. The model with both local cues achieves higher accuracy than those of models with individual cues in determining FG relations, indicating SA and T-Junctions are not mutually contradictory. Compared to the model with no local cues, the model with both local cues achieves greater than or equal to 8.78% improvement in determining FG relations at every border location of images in the BSDS dataset. In the audio-visual domain, first we build a simple computational model to explain how visual search can be aided by providing concurrent, co-spatial auditory cues. Our model shows that adding a co-spatial, concurrent auditory cue can enhance the saliency of a weakly visible target among prominent visual distractors, the behavioral effect of which could be faster reaction time and/or better search accuracy. Lastly, a bottom-up, feed-forward, proto-object based audiovisual saliency map (AVSM) for the analysis of dynamic natural scenes is presented. We demonstrate that the performance of proto-object based AVSM in detecting and localizing salient objects/events is in agreement with human judgment. In addition, we show the AVSM computed as a linear combination of visual and auditory feature conspicuity maps captures a higher number of valid salient events compared to unisensory saliency maps

Photorealistic physically based render engines: a comparative study

Pérez Roig, F. (2012). Photorealistic physically based render engines: a comparative study. http://hdl.handle.net/10251/14797.Archivo delegad

A vision system for mobile maritime surveillance platforms

Mobile surveillance systems play an important role to minimise security and safety threats in high-risk or hazardous environments. Providing a mobile marine surveillance platform with situational awareness of its environment is important for mission success. An essential part of situational awareness is the ability to detect and subsequently track potential target objects.Typically, the exact type of target objects is unknown, hence detection is addressed as a problem of finding parts of an image that stand out in relation to their surrounding regions or are atypical to the domain. Contrary to existing saliency methods, this thesis proposes the use of a domain specific visual attention approach for detecting potential regions of interest in maritime imagery. For this, low-level features that are indicative of maritime targets are identified. These features are then evaluated with respect to their local, regional, and global significance. Together with a domain specific background segmentation technique, the features are combined in a Bayesian classifier to direct visual attention to potential target objects.The maritime environment introduces challenges to the camera system: gusts, wind, swell, or waves can cause the platform to move drastically and unpredictably. Pan-tilt-zoom cameras that are often utilised for surveillance tasks can adjusting their orientation to provide a stable view onto the target. However, in rough maritime environments this requires high-speed and precise inputs. In contrast, omnidirectional cameras provide a full spherical view, which allows the acquisition and tracking of multiple targets at the same time. However, the target itself only occupies a small fraction of the overall view. This thesis proposes a novel, target-centric approach for image stabilisation. A virtual camera is extracted from the omnidirectional view for each target and is adjusted based on the measurements of an inertial measurement unit and an image feature tracker. The combination of these two techniques in a probabilistic framework allows for stabilisation of rotational and translational ego-motion. Furthermore, it has the specific advantage of being robust to loosely calibrated and synchronised hardware since the fusion of tracking and stabilisation means that tracking uncertainty can be used to compensate for errors in calibration and synchronisation. This then completely eliminates the need for tedious calibration phases and the adverse effects of assembly slippage over time.Finally, this thesis combines the visual attention and omnidirectional stabilisation frameworks and proposes a multi view tracking system that is capable of detecting potential target objects in the maritime domain. Although the visual attention framework performed well on the benchmark datasets, the evaluation on real-world maritime imagery produced a high number of false positives. An investigation reveals that the problem is that benchmark data sets are unconsciously being influenced by human shot selection, which greatly simplifies the problem of visual attention. Despite the number of false positives, the tracking approach itself is robust even if a high number of false positives are tracked

Art as integral part of architectural space

Thesis (M.S.V.S.)--Massachusetts Institute of Technology, Dept. of Architecture, 1993.Includes bibliographical references (leaves 93-97).To integrate art with architecture is the intention of every architect. However, many times other requirements overwhelm artistic potential. There are numerous good, simple examples in the history of architecture where solutions to a variety of often functional requirements have produced exceptional artistic expressions, which in tum have inspired contemporary architectural practice. Pre-industrial architecture not only responded to natural conditions in the environment; it employed all the senses as well ·. in its design of living environments. Today, unfortunately, we rarely find that architects pay attention to sound, smell, water, natural cycles, or, almost unimaginable, to time. Our technology encourages us to separate ourselves from nature. However, this same technology can help us to reintegrate ourselves with nature by designing better living environments. This thesis is, therefore, my way of rethinking design principles that shape the contemporary urban environment and often give it such a cold, formal image. My own philosophy of design is given in the introduction. The rest of the thesis is basically the supportive material, which further illuminates the ideas presented in the introduction. The first part discusses some general trends in contemporary society in order to place my own view of design within a broader context, while the second part lists numerous examples from the history of architecture and art to illustrate and further my philosophy. At the end, in the appendix, I present one of my own projects, the Interactive Water Curtain, to concretely show some of the implications of my aesthetics. Through this work I hope to illustrate the richness of various traditional architectural practices that take advantage of sound, water, time (celestial movement), fragrances, and even living creatures in designing places. I hope this will stimulate creative thinking about using not just visual effects in the design of our living environment, but employing acoustic, olfactory, astronomical, ecological, and kinesthetic design in order to create sensually richer and more pleasant environments where people can live in harmony with nature and other living creatures.by Marta Vahtar.M.S.V.S

Pixel-level Image Fusion Algorithms for Multi-camera Imaging System

This thesis work is motivated by the potential and promise of image fusion technologies in the multi sensor image fusion system and applications. With specific focus on pixel level image fusion, the process after the image registration is processed, we develop graphic user interface for multi-sensor image fusion software using Microsoft visual studio and Microsoft Foundation Class library. In this thesis, we proposed and presented some image fusion algorithms with low computational cost, based upon spatial mixture analysis. The segment weighted average image fusion combines several low spatial resolution data source from different sensors to create high resolution and large size of fused image. This research includes developing a segment-based step, based upon stepwise divide and combine process. In the second stage of the process, the linear interpolation optimization is used to sharpen the image resolution. Implementation of these image fusion algorithms are completed based on the graphic user interface we developed. Multiple sensor image fusion is easily accommodated by the algorithm, and the results are demonstrated at multiple scales. By using quantitative estimation such as mutual information, we obtain the experiment quantifiable results. We also use the image morphing technique to generate fused image sequence, to simulate the results of image fusion. While deploying our pixel level image fusion algorithm approaches, we observe several challenges from the popular image fusion methods. While high computational cost and complex processing steps of image fusion algorithms provide accurate fused results, they also makes it hard to become deployed in system and applications that require real-time feedback, high flexibility and low computation abilit

Using reconstructed visual reality in ant navigation research

Insects have low resolution eyes and a tiny brain, yet they continuously solve very complex navigational problems; an ability that underpins fundamental biological processes such as pollination and parental care. Understanding the methods they employ would have profound impact on the fields of machine vision and robotics. As our knowledge on insect navigation grows, our physical, physiological and neural models get more complex and detailed. To test these models we need to perform increasingly sophisticated experiments. Evolution has optimised the animals to operate in their natural environment. To probe the fine details of the methods they utilise we need to use natural visual scenery which, for experimental purposes, we must be able to manipulate arbitrarily. Performing physiological experiments on insects outside the laboratory is not practical and our ability to modify the natural scenery for outdoor behavioural experiments is very limited. The solution is reconstructed visual reality, a projector that can present the visual aspect of the natural environment to the animal with high fidelity, taking the peculiarities of insect vision into account. While projectors have been used in insect research before, during my candidature I designed and built a projector specifically tuned to insect vision. To allow the ant to experience a full panoramic view, the projector completely surrounds her. The device (Antarium) is a polyhedral approximation of a sphere. It contains 20 thousand pixels made out of light emitting diodes (LEDs) that match the spectral sensitivity of Myrmecia. Insects have a much higher fusion frequency limit than humans, therefore the device has a very high flicker frequency (9kHz) and also a high frame rate (190fps). In the Antarium the animal is placed in the centre of the projector on a trackball. To test the trackball and to collect reference data, outdoor experiments were performed where ants were captured, tethered and placed on the trackball. The apparatus with the ant on it was then placed at certain locations relative to the nest and the foraging tree and the movements of the animal on the ball were recorded and analysed. The outdoor experiments proved that the trackball was well suited for our ants, and also provided the baseline behaviour reference for the subsequent Antarium experiments. To assess the Antarium, the natural habitat of the experimental animals was recreated as a 3-dimensional model. That model was then projected for the ants and their movements on the trackball was recorded, just like in the outdoor experiments Initial feasibility tests were performed by projecting a static image, which matches what the animals experienced during the outdoor experiments. To assess whether the ant was orienting herself relative to the scene we rotated the projected scene around her and her response monitored. Statistical methods were used to compare the outdoor and in-Antarium behaviour. The results proved that the concept was solid, but they also uncovered several shortcomings of the Antarium. Nevertheless, even with its limitations the Antarium was used to perform experiments that would be very hard to do in a real environment. In one experiment the foraging tree was repositioned in or deleted from the scene to see whether the animals go to where the tree is or where by their knowledge it should be. The results suggest the latter but the absence or altered location of the foraging tree certainly had a significant effect on the animals. In another experiment the scene, including the sky, were re-coloured to see whether colour plays a significant role in navigation. Results indicate that even very small amount of UV information statistically significantly improves the navigation of the animals. To rectify the device limitations discovered during the experiments a new, improved projector was designed and is currently being built