In this paper we present a method capable of perceiving high dynamic range scene. The special feature of the method is that it changes the integration time of the imager on the pixel level. Using CNN-UM we can calculate the integration time for the pixels, and hence low dynamic range integration type CMOS sensors will be able to perceive high dynamic range scenes. The method yields high contrast without introducing non-existing edges

Roska, Tamás

Wagner, R.

Zarándy, Ákos

English

Repository of the Academy's Library

High Dynamic Range Perception with Spatially Variant Exposure R. Wagner1, Á. Zarándy2, T. Roska,1,2 1Faculty of Information Technology and the Jedlik Laboratories, Pázmány University, Budapest  2Computer and Automation Research Institute of the Hungarian Academy of Sciences, Budapest wagnerr@sztaki.hu ABSTRACT: In this paper we present a method capable of perceiving high dynamic range scene. The special feature of the method is that it changes the integration time of the imager on the pixel level. Using CNN-UM we can calculate the integration time for the pixels, and hence low dynamic range integration type CMOS sensors will be able to perceive high dynamic range scenes. The method yields high contrast without introducing non-existing edges. 1. Introduction Equally illuminated scenes have 2-3 log orders (8-10 bits) dynamics only, while unequally illuminated scenes might easily bear with 5 decades (17 bits) of dynamics. Perception of these high dynamic range scenes is an easy job for our eyes, however it is a great challenge for traditional image sensors. If we want to capture these high dynamic range scenes, an obvious idea is the selection of a logarithmic imager. Though it can cover the high dynamic range, but as an exchange, the image will be noisy and low contrast. To obtain better quality images we may use less noisy integrating type sensors [8], but these can cover only 8-10 bits of dynamic range. In order to overcome this problem, images of different exposure parameters can be combined or very expensive large dynamic range linear cameras applied [8]. In [12,13] the time interval during which the sensor integrates until a certain level is measured. With this method the dynamic range is represented in the temporal domain. These results need a post-processing step after which the large dynamic range image can be visualized. We propose a dynamic, local tuning of the sensor elements, so that the large intra-scene illumination differences can be reduced by spatially variant exposure parameters. We perceive the high dynamic range, and retrieve a human perceivable image representing the uniformly illuminated scene. This approach is very similar to the operation method of our retina. In this paper we present an integration time adjustment algorithm for a locally controllable photo-sensor matrix. The research of the underlying hardware is done in some research laboratories (e.g. IMSE-CNM Seville Spain, MTA-SzTAKI, Budapest, Hungary, [1-3]). We improved our previous work [14] with some extensions of the method. 2. Integration time Adjustment Algorithm and Extensions of the Method Our integration time adjustment method is based on inter-frame processing. It is an iterative method, where the iteration is done through the captured frames. Similarly to our retina the adaptation is a dynamic process it eliminates the spatial-temporal low-pass component ([4-6]) and thus reduces the high dynamic range. The detailed description can be found in the following subsections. The presented pictures are results of simulations. The local varying integration time pictures are an interpolation between a picture series of different integration time (see [14] for details). 2.1 Dynamic Adjustment In this subsection, we present the inter-frame dynamic adjustment of the integration time. In Fig. 1 we can see a general flowchart. As a first step, we define an initial integration time map T0. Then, we repeat the followings in each iterative step: At the n-th iteration we capture an image with the integration time map Tn and get the result image Vn. Then, we perform diffusion operation on both Vn and Tn. Based on the diffused images we compute the integration time map Tn+1 for the next snapshot. This is the adaptation phase where we change the variable parameter Tn in order to adapt to the scene.   Figure. 1. Flowchart of the integration times dynamic adjustment. The DC component can be computed based on the integration time values, and it is added to the resulting image. In [14] we showed some computation of the integration time map. The basic feature of these methods is, that they all alter the integration map Tn so that the local average of Vn (spatial low-pass component) becomes the half of the maximal response (Vmax/2). Hence at adapted state the low-pass component will be eliminated and there will not be saturated areas. The computation of the spatial low-pass component is feasible on CNN-UM architecture with diffusion operator [9], and the further operation multiplication, addition of two images (see [14]) as well. On Fig. 2. and 3. we can see the adjustment results. In this paper in the followings we present some improvements of the dynamic adjustment. 2.2 Restoring DC level The methods presented so far enhance the high-pass component of the images: edges, local differences. We eliminate the effect of the differences in the local average by driving all the averages to the half of the maximum response (Vmax/2). This is very advantageous in some applications, but other observers may be interested in a reduced DC component, which does not saturate the image. This DC component can be computed from the integration time, because the integration time is adjusted according to it. We normalize the reduced DC component to [-Vmax/2, Vmax/2]; and multiply it with a parameter: cDC.. More details can be read in [14]. Fig. 2. shows pure dynamic adjustment result and result where the DC component was added.  a.)  b.) Figure 2. Result images after the 8th adjustment.. (a) without DC component, (b) DC component added. cDC=0.5 2.3 Dynamic Results We have applied our methods to dynamic image sequence. In this case, the input scene -captured in each iteration step- changes in time. We investigated how the method adapts to spatial and temporal luminance changes.  The complete image flow can be viewed in [11]. Having a change in the illumination the algorithm needs an adaptation time, to calculate the new integration time map. Before having adapted to the new conditions we get dark values where we diminished the illumination and vice versa. Thus the integration time can be viewed as a state variable of the system, similar to the adaptation state of the retinal cones ([4]). This means, that we have a temporal high-pass filtering effect as well. This is similar to the retinal processing, where the spatial-temporal low-pass component is suppressed and the high-pass components enhanced (see section II). 2.4 Introducing anisotropy Though our algorithm was able to adapt to the changed conditions, some parts of the computation may be improved, in order to obtain nicer results. The basic idea of our adjustment method is, that we alter the integration time so, that the spatial low-pass component of the intensity becomes Vmax/2. This is advantageous, because we eliminate the effect of the intra-scene illumination differences (Fig. 2.a.). On the other hand the illumination changes at some region crisply (e.g. edge of the book). Along the edge of the book the local average is smaller because of the neighboring dark areas, thus the integration time is set higher, and we get a white strip (see Fig.2 a.). This is the “halo” effect [7]. To solve this problem in [7] an anisotropic diffusion- like method (low-pass component computing) is introduced. In [7] the author simulates a resistive grid, where the resistance between points i and j increases with the contrast between them. Hence, for computing the low pass component along the brighter side of the edges, only the brighter areas are taken into account, the effect of dark areas is stopped by the large resistances (and vice versa). In our simulations we computed contrast from the integration time, which represents the local average. A great difference in the integration time means that there are neighboring regions of different brightness, and the diffusion should be weaker. The resistance between two points i and j was computed as followings: =ijjianisji TTTTcR ,max,         (1) where canis is a scaling factor of the anisotropy On Fig.3 we can see the effect of using anisotropic diffusion. We can use the difference of the integration time as a constraint for the diffusion. On the other hand the method still needs parameter tuning and a consideration of the computation, since division is difficult to implement on the future chip.  a.)  b.) Figure 3. a.) shows adjustment result using low pass filtering after the 5th iteration. b.) shows  results simulating anisotropic resistive grid. It is observable that along the edges of bright areas the white stripes are disappeared. 2.5 Post-processing The method so far assumes, that we alter the integration time unrestrictedly until it fits to the local average. In some cases we do not want to (or we cannot) increase the integration time anymore. In case of night scenes, the counter - which specifies the integration time - may reach its maximal value and we cannot raise it anymore. Having a moving scene the increase of integration time may evoke strong motion blur effect. In this cases it is worth not to increase the integration time, but post-process (enhance) the captured image [10,12].  a.)  b.) Figure 4.  a.) shows image captured with the highest integration time. b.) shows the result after adaptation. One can clearly see how b.) is enhanced by the post-processing. Tmax was global for all pixels, so far did not take into account location dependent motion blur. In our algorithm we propose the followings: Compute the integration time T unrestrictedly as in the previous sections (Fig.1.), but this will be only a logical value. We specify a maximal capturing integration time Tmax, which is depending on the maximal possible time, and the motion of the scene. At the pixels where T>Tmax we capture an image with Tmax and multiply the result image with maxTT . If this post-processing gain is high enough (local average of the result image became Vmax/2) T should not be enhanced more. Thus the computation of the new integration time map is based on this post-processed image. Some results can be seen on Fig.4 3. Conclusions In this paper we presented our work on a local-adjustment algorithm for the future locally adjustable sensor chip. This retina inspired algorithm enhances the spatial-temporal high-pass component of the high dynamic range scene. Beside that it suppresses the low-pass component thus reduces the dynamic range of the scene. We have also presented some experimental results showing the perception of static and time variant scenes. As other high dynamic range sensing systems our method is capable of perceiving high dynamic range scenes. Using our method, each kind of photodetectors dynamic range can be extended. Acknowledgement This research was founded by the Grant of the National Science Fund of Hungary (OTKA), the multidisciplinary doctoral school at the Faculty of Information Technology of the Pázmány P. Catholic University and the Office of Naval Research (ONR) Grant No. N0001-4021-0884 and the European Community (Grant No. IST-2001-38097 LOCUST). References [1] T. Roska, ”Computer-Sensors: Spatial-Temporal Computers for Analog Array Signals, Dynamically Integrated with Sensors”, Journal of VLSI Signal Processing, Vol.23, pp.221-237, 1999 [2] T. Roska and Á. Zarándy: ”Proactive Adaptive Cellular Sensory-Computer Architecture via extending the CNN Universal Machine”, to be published on the ECCTD ‘03 European Conference on Circuit Theory and Design, 1 - 4 September 2003, Kraków, Poland [3] G. Linan: “ACE16K: an Advanced Focal-Plane Analog Programmable Array Processor”, ESSCIRC 2001 Presentations 27th European Solid-State Circuits Conference, Villach, Austria, 18-20 September 2001 [4] R. A. Norman and F. S. Werblin: “Control of Retinal Sensitivity: I. Light and Dark Adaptation of Vertebrate Rods and Cones”, The Journal of General Physiology, Vol.63, pp.37-61, 197 [5] F. S. Werblin: “Control of Retinal Sensitivity: II. Lateral Interactions at the Outer Plexiform Layer”, The Journal of General Physiology, Vol.63, pp.62-87, 1974 [6] Carver A. Mead and M. A. Mahowald: ”A silicon model of early visual processing”, Neural Networks, Vol. 1, Issue 1, pp. 91-97 (1988) [7] V. Brajovic: ”A model for reflectance perception in vision”, Bioengineered and Bioinspired Systems, Proceedings of SPIE, Vol. 5119 (2003) [8] A. El Gamal: “High Dynamic Range Image Sensors”, Tutorial at International Solid-State Circuits Conference, February 2002, Available: http://www-isl.stanford.edu/~abbas/group/papers_and_pub/isscc02_tutorial.pdf [9] T. Roska, L. Kék, L. Nemes, Á. Zarándy and P. Szolgay (ed), “CNN Software Library (Templates and Algorithms), Version 7.3”, Analogical and Neural Computing Laboratory, Computer and Automation Research Institute, Hungarian Academy of Sciences (MTA SzTAKI), DNS-CADET-15, Budapest, 1999 [10] D. Yang, A. El Gamal, B. Fowler, and H. Tian, “A 640x512 CMOS image sensor with ultra-wide dynamic range floating-point pixel-level ADC”, IEEE. J. Solid-State Circuits, vol. 34., no. 12, pp 1821-1834, Dec. 1999. [11] Image flows of the dynamic adjustment are available at the following site: http://digitus.itk.ppke.hu/~wagner/dynres.html [12] T. Hamamoto and K. Aizawa: “A Computational Image Sensor with Adaptive Pixel-Based Integration Time”, IEEE Journal of Solid State Circuits, Vol. 36. no. 4 April. 2001 [13] E. Culurciello, R. Etienne-Cummings, and K. Boahen: “A Biomorphic Digital Image Sensor”, IEEE Journal of Solid-StateCircuits, Vol. 38., No. 2. February 2003 [14] R. Wagner, Á. Zarándy and T. Roska: “Adaptive Perception with Locally-Adaptable Sensor Array”, IEEE Transactions on Circuits and Systems I. CNN Special Issue (accepted). 

A 640x512 CMOS image sensor with ultra-wide dynamic range floating-point pixel-level ADC”,

A Biomorphic Digital Image Sensor”,

A Computational Image Sensor with Adaptive PixelBased Integration Time”,

A model for reflectance perception in vision”, Bioengineered and Bioinspired Systems,

A silicon model of early visual processing”,

ACE16K: an Advanced Focal-Plane Analog Programmable Array Processor”,

Adaptive Perception with Locally-Adaptable Sensor Array”,

Computer-Sensors: Spatial-Temporal Computers for Analog Array Signals, Dynamically Integrated with Sensors”,

Gamal: “High Dynamic Range Image Sensors”,

Image flows of the dynamic adjustment are available at the following site: http://digitus.itk.ppke.hu/~wagner/dynres.html

Proactive Adaptive Cellular Sensory-Computer Architecture via extending the CNN Universal Machine”, to be published on

Werblin: “Control of Retinal Sensitivity: I. Light and Dark Adaptation of Vertebrate Rods and Cones”,

Werblin: “Control of Retinal Sensitivity: II. Lateral Interactions at the Outer Plexiform Layer”,

High dynamic range perception with spatially variant exposure

SZTAKI Publication Repository

High dynamic range perception with spatially variant exposure

Abstract

Similar works

Full text

Available Versions

Repository of the Academy's Library

SZTAKI Publication Repository