Optimization of video capturing and tone mapping in video camera systems

Image enhancement techniques are widely employed in many areas of professional and consumer imaging, machine vision and computational imaging. Image enhancement techniques used in surveillance video cameras are complex systems involving controllable lenses, sensors and advanced signal processing. In surveillance, a high output image quality with very robust and stable operation under difficult imaging conditions are essential, combined with automatic, intelligent camera behavior without user intervention. The key problem discussed in this thesis is to ensure this high quality under all conditions, which specifically addresses the discrepancy of the dynamic range of input scenes and displays. For example, typical challenges are High Dynamic Range (HDR) and low-dynamic range scenes with strong light-dark differences and overall poor visibility of details, respectively. The detailed problem statement is as follows: (1) performing correct and stable image acquisition for video cameras in variable dynamic range environments, and (2) finding the best image processing algorithms to maximize the visualization of all image details without introducing image distortions. Additionally, the solutions should satisfy complexity and cost requirements of typical video surveillance cameras. For image acquisition, we develop optimal image exposure algorithms that use a controlled lens, sensor integration time and camera gain, to maximize SNR. For faster and more stable control of the camera exposure system, we remove nonlinear tone-mapping steps from the level control loop and we derive a parallel control strategy that prevents control delays and compensates for the non-linearity and unknown transfer characteristics of the used lenses. For HDR imaging we adopt exposure bracketing that merges short and long exposed images. To solve the involved non-linear sensor distortions, we apply a non-linear correction function to the distorted sensor signal, implementing a second-order polynomial with coefficients adaptively estimated from the signal itself. The result is a good, dynamically controlled match between the long- and short-exposed image. The robustness of this technique is improved for fluorescent light conditions, preventing serious distortions by luminance flickering and color errors. To prevent image degradation we propose both fluorescent light detection and fluorescence locking, based on measurements of the sensor signal intensity and color errors in the short-exposed image. The use of various filtering steps increases the detector robustness and reliability for scenes with motion and the appearance of other light sources. In the alternative algorithm principle of fluorescence locking, we ensure that light integrated during the short exposure time has a correct intensity and color by synchronizing the exposure measurement to the mains frequency. The second area of research is to maximize visualization of all image details. This is achieved by both global and local tone mapping functions. The largest problem of Global Tone Mapping Functions (GTMF) is that they often significantly deteriorate the image contrast. We have developed a new GTMF and illustrate, both analytically and perceptually, that it exhibits only a limited amount of compression, compared to conventional solutions. Our algorithm splits GTMF into two tasks: (1) compressing HDR images (DRC transfer function) and (2) enhancing the (global) image contrast (CHRE transfer function). The DRC subsystem adapts the HDR video signal to the remainder of the system, which can handle only a fraction of the original dynamic range. Our main contribution is a novel DRC function shape which is adaptive to the image, so that details in the dark image parts are enhanced simultaneously while only moderately compressing details in the bright areas. Also, the DRC function shape is well matched with the sensor noise characteristics in order to limit the noise amplification. Furthermore, we show that the image quality can be significantly improved in DRC compression if a local contrast preservation step is included. The second part of GTMF is a CHRE subsystem that fine-tunes and redistributes the luminance (and color) signal in the image, to optimize global contrast of the scene. The contribution of the proposed CHRE processing is that unlike standard histogram equalization, it can preserve details in statistically unpopulated but visually relevant luminance regions. One of the important cornerstones of the GTMF is that both DRC and CHRE algorithms are performed in the perceptually uniform space and optimized for the salient regions obtained by the improved salient-region detector, to maximize the relevant information transfer to the HVS. The proposed GTMF solution offers a good processing quality, but cannot sufficiently preserve local contrast for extreme HDR signals and it gives limited improvement low-contrast scenes. The local contrast improvement is based on the Locally Adaptive Contrast Enhancement (LACE) algorithm. We contribute by using multi-band frequency decomposition, to set up the complete enhancement system. Four key problems occur with real-time LACE processing: (1) "halo" artifacts, (2) clipping of the enhancement signal, (3) noise degradation and (4) the overall system complexity. "Halo" artifacts are eliminated by a new contrast gain specification using local energy and contrast measurements. This solution has a low complexity and offers excellent performance in terms of higher contrast and visually appealing performance. Algorithms preventing clipping of the output signal and reducing noise amplification give a further enhancement. We have added a supplementary discussion on executing LACE in the logarithmic domain, where we have derived a new contrast gain function solving LACE problems efficiently. For the best results, we have found that LACE processing should be performed in the logarithmic domain for standard and HDR images, and in the linear domain for low-contrast images. Finally, the complexity of the contrast gain calculation is reduced by a new local energy metric, which can be calculated efficiently in a 2D-separable fashion. Besides the complexity benefit, the proposed energy metric gives better performance compared to the conventional metrics. The conclusions of our work are summarized as follows. For acquisition, we need to combine an optimal exposure algorithm, giving both improved dynamic performance and maximum image contrast/SNR, with robust exposure bracketing that can handle difficult conditions such as fluorescent lighting. For optimizing visibility of details in the scene, we have split the GTMF in two parts, DRC and CHRE, so that a controlled optimization can be performed offering less contrast compression and detail loss than in the conventional case. Local contrast is enhanced with the known LACE algorithm, but the performance is significantly improved by individually addressing "halo" artifacts, signal clipping and noise degradation. We provide artifact reduction by new contrast gain function based on local energy, contrast measurements and noise estimation. Besides the above arguments, we have contributed feasible performance metrics and listed ample practical evidence of the real-time implementation of our algorithms in FPGAs and ASICs, used in commercially available surveillance cameras, which obtained awards for their image quality

Space programs summary no. 37-64, volume 1 for the period 1 May to 30 June 1970. Flight projects

Mariner and Viking flight projects for Mars exploratio

Study of robotics systems applications to the space station program

Applications of robotics systems to potential uses of the Space Station as an assembly facility, and secondarily as a servicing facility, are considered. A typical robotics system mission is described along with the pertinent application guidelines and Space Station environmental assumptions utilized in developing the robotic task scenarios. A functional description of a supervised dual-robot space structure construction system is given, and four key areas of robotic technology are defined, described, and assessed. Alternate technologies for implementing the more routine space technology support subsystems that will be required to support the Space Station robotic systems in assembly and servicing tasks are briefly discussed. The environmental conditions impacting on the robotic configuration design and operation are reviewed

Development of an image converter of radical design

A long term investigation of thin film sensors, monolithic photo-field effect transistors, and epitaxially diffused phototransistors and photodiodes to meet requirements to produce acceptable all solid state, electronically scanned imaging system, led to the production of an advanced engineering model camera which employs a 200,000 element phototransistor array (organized in a matrix of 400 rows by 500 columns) to secure resolution comparable to commercial television. The full investigation is described for the period July 1962 through July 1972, and covers the following broad topics in detail: (1) sensor monoliths; (2) fabrication technology; (3) functional theory; (4) system methodology; and (5) deployment profile. A summary of the work and conclusions are given, along with extensive schematic diagrams of the final solid state imaging system product

High Resolution, High Frame Rate Video Technology

Papers and working group summaries presented at the High Resolution, High Frame Rate Video (HHV) Workshop are compiled. HHV system is intended for future use on the Space Shuttle and Space Station Freedom. The Workshop was held for the dual purpose of: (1) allowing potential scientific users to assess the utility of the proposed system for monitoring microgravity science experiments; and (2) letting technical experts from industry recommend improvements to the proposed near-term HHV system. The following topics are covered: (1) State of the art in the video system performance; (2) Development plan for the HHV system; (3) Advanced technology for image gathering, coding, and processing; (4) Data compression applied to HHV; (5) Data transmission networks; and (6) Results of the users' requirements survey conducted by NASA

Application of advanced technology to space automation

Automated operations in space provide the key to optimized mission design and data acquisition at minimum cost for the future. The results of this study strongly accentuate this statement and should provide further incentive for immediate development of specific automtion technology as defined herein. Essential automation technology requirements were identified for future programs. The study was undertaken to address the future role of automation in the space program, the potential benefits to be derived, and the technology efforts that should be directed toward obtaining these benefits

Evaluation of the color image and video processing chain and visual quality management for consumer systems

With the advent of novel digital display technologies, color processing is increasingly becoming a key aspect in consumer video applications. Today’s state-of-the-art displays require sophisticated color and image reproduction techniques in order to achieve larger screen size, higher luminance and higher resolution than ever before. However, from color science perspective, there are clearly opportunities for improvement in the color reproduction capabilities of various emerging and conventional display technologies. This research seeks to identify potential areas for improvement in color processing in a video processing chain. As part of this research, various processes involved in a typical video processing chain in consumer video applications were reviewed. Several published color and contrast enhancement algorithms were evaluated, and a novel algorithm was developed to enhance color and contrast in images and videos in an effective and coordinated manner. Further, a psychophysical technique was developed and implemented for performing visual evaluation of color image and consumer video quality. Based on the performance analysis and visual experiments involving various algorithms, guidelines were proposed for the development of an effective color and contrast enhancement method for images and video applications. It is hoped that the knowledge gained from this research will help build a better understanding of color processing and color quality management methods in consumer video

Contours and contrast

Contrast in photographic and computer-generated imagery communicates colour and lightness differences that would be perceived when viewing the represented scene. Due to depiction constraints, the amount of displayable contrast is limited, reducing the image's ability to accurately represent the scene. A local contrast enhancement technique called unsharp masking can overcome these constraints by adding high-frequency contours to an image that increase its apparent contrast. In three novel algorithms inspired by unsharp masking, specialized local contrast enhancements are shown to overcome constraints of a limited dynamic range, overcome an achromatic palette, and to improve the rendering of 3D shapes and scenes. The Beyond Tone Mapping approach restores original HDR contrast to its tone mapped LDR counterpart by adding highfrequency colour contours to the LDR image while preserving its luminance. Apparent Greyscale is a multi-scale two-step technique that first converts colour images and video to greyscale according to their chromatic lightness, then restores diminished colour contrast with high-frequency luminance contours. Finally, 3D Unsharp Masking performs scene coherent enhancement by introducing 3D high-frequency luminance contours to emphasize the details, shapes, tonal range and spatial organization of a 3D scene within the rendering pipeline. As a perceptual justification, it is argued that a local contrast enhancement made with unsharp masking is related to the Cornsweet illusion, and that this may explain its effect on apparent contrast.Seit vielen Jahren ist die realistische Erzeugung von virtuellen Charakteren ein zentraler Teil der Computergraphikforschung. Dennoch blieben bisher einige Probleme ungelöst. Dazu zählt unter anderem die Erzeugung von Charakteranimationen, welche unter der Benutzung der traditionellen, skelettbasierten Ansätze immer noch zeitaufwändig sind. Eine weitere Herausforderung stellt auch die passive Erfassung von Schauspielern in alltäglicher Kleidung dar. Darüber hinaus existieren im Gegensatz zu den zahlreichen skelettbasierten Ansätzen nur wenige Methoden zur Verarbeitung und Veränderung von Netzanimationen. In dieser Arbeit präsentieren wir Algorithmen zur Lösung jeder dieser Aufgaben. Unser erster Ansatz besteht aus zwei Netz-basierten Verfahren zur Vereinfachung von Charakteranimationen. Obwohl das kinematische Skelett beiseite gelegt wird, können beide Verfahren direkt in die traditionelle Pipeline integriert werden, wobei die Erstellung von Animationen mit wirklichkeitsgetreuen Körperverformungen ermöglicht wird. Im Anschluss präsentieren wir drei passive Aufnahmemethoden für Körperbewegung und Schauspiel, die ein deformierbares 3D-Modell zur Repräsentation der Szene benutzen. Diese Methoden können zur gemeinsamen Rekonstruktion von zeit- und raummässig kohärenter Geometrie, Bewegung und Oberflächentexturen benutzt werden, die auch zeitlich veränderlich sein dürfen. Aufnahmen von lockerer und alltäglicher Kleidung sind dabei problemlos möglich. Darüber hinaus ermöglichen die qualitativ hochwertigen Rekonstruktionen die realistische Darstellung von 3D Video-Sequenzen. Schließlich werden zwei neuartige Algorithmen zur Verarbeitung von Netz-Animationen beschrieben. Während der erste Algorithmus die vollautomatische Umwandlung von Netz-Animationen in skelettbasierte Animationen ermöglicht, erlaubt der zweite die automatische Konvertierung von Netz-Animationen in so genannte Animations-Collagen, einem neuen Kunst-Stil zur Animationsdarstellung. Die in dieser Dissertation beschriebenen Methoden können als Lösungen spezieller Probleme, aber auch als wichtige Bausteine größerer Anwendungen betrachtet werden. Zusammengenommen bilden sie ein leistungsfähiges System zur akkuraten Erfassung, zur Manipulation und zum realistischen Rendern von künstlerischen Aufführungen, dessen Fähigkeiten über diejenigen vieler verwandter Capture-Techniken hinausgehen. Auf diese Weise können wir die Bewegung, die im Zeitverlauf variierenden Details und die Textur-Informationen eines Schauspielers erfassen und sie in eine mit vollständiger Information versehene Charakter-Animation umwandeln, die unmittelbar weiterverwendet werden kann, sich aber auch zur realistischen Darstellung des Schauspielers aus beliebigen Blickrichtungen eignet