Bull, DR

Canagarajah, CN

Clewer, DR

Luo, L

English

Crossref

A hierarchical genetic disparity estimation algorithm for multiview image synthesis

A hierarchical genetic algorithm for disparity estimation is presented. The goal, to estimate reliable disparity fields with low computational cost, is reached using a hierarchical genetic matching procedure. Firstly, each hierarchical image of the stereo pair is divided into sets of feature points and non-feature points. The image morphological gradient for feature points and the disparity Laplacian function for non-feature points are incorporated into the matching function to serve as an adaptive smoothness term. Meanwhile, the vertical-discontinuity constraint and the ordering constraint are also proposed to smooth out vertical disparity discontinuities and to obtain a more reliable disparity estimation. In the hierarchical genetic matching procedure, previously estimated vectors at the former image hierarchy are used to predict the corresponding searching space of chromosomes, and to correct each newly calculated set of disparity vectors. This significantly increases the accuracy of disparity estimatio

Explore Bristol Research

                          Luo, L., Clewer, D. R., Bull, D. R., & Canagarajah, C. N. (2000). Ahierarchical genetic disparity estimation algorithm for multiview imagesynthesis. In IEEE Int. Conf. on Image Processing (ICIP), Vancouver. (Vol.2, pp. 768 - 771). Institute of Electrical and Electronics Engineers (IEEE).10.1109/ICIP.2000.899822Link to published version (if available):10.1109/ICIP.2000.899822Link to publication record in Explore Bristol ResearchPDF-documentUniversity of Bristol - Explore Bristol ResearchGeneral rightsThis document is made available in accordance with publisher policies. Please cite only the publishedversion using the reference above. Full terms of use are available:http://www.bristol.ac.uk/pure/about/ebr-terms.htmlTake down policyExplore Bristol Research is a digital archive and the intention is that deposited content should not beremoved. However, if you believe that this version of the work breaches copyright law please contactopen-access@bristol.ac.uk and include the following information in your message:• Your contact details• Bibliographic details for the item, including a URL• An outline of the nature of the complaintOn receipt of your message the Open Access Team will immediately investigate your claim, make aninitial judgement of the validity of the claim and, where appropriate, withdraw the item in questionfrom public view.A HIERARCHICAL GENETIC DISPARITY ESTIMATION ALGORITHM FOR MULTIVIEW IMAGE SYNTHESIS L. J. Luo, D. R. Clewer, D. R. Bull, C. N. Canagarajah Centre for Communications Research, Univ. of Bristol, Bristol BS8 lUB, UK E-mail: {Lijun.Luo, David.Clewer, Dave.Bull, Nishan.Canagarajah}O bristol.ac.uk ABSTRACT In this paper, a hierarchical genetic algorithm for disparity estimation is presented. The goal, to estimate reliable disparity fields with low computational cost, is reached using a hierarchical genetic matching procedure. Firstly, each hierarchical image of the stereo pair is divided into sets of feature points and non-feature points. The image morphological gradient for feature points and the disparity Laplacian function for non-feature points are incorporated into the matching function to serve as an adaptive smoothness term. Meanwhile, the vertical- discontinuity constraint and the ordering constraint are also proposed to smooth out vertical disparity discontinuities and to obtain a more reliable disparity estimation. In the hierarchical genetic matching procedure, previously estimated vectors at the former image hierarchy are used to predict the corresponding searching , space of chromosomes, and to correct each newly calculated set of disparity vectors. This significantly increases the accuracy of disparity estimation. 1. INTRODUCTION Multiple views of a scene, obtained from cameras positioned at distinct viewpoints, can provide the viewer with the benefits of added realism, selective viewing, and improved scene understanding. In a multiview video display system, the viewer would expect that the view angle could be altered with the movement of the head. In order to achieve this, the viewpoint adaptation must be performed according to the viewer’s head movements. In many situations it may be too impractical and costly to set up a multi-camera rig for image capture. The bandwidth requirements for transmission and storage may be prohibitive. A cheaper, lower bit-rate approach is to use a wide baseline stereoscopic camera set-up to capture and transmit left and right views of a scene. Intermediate image pairs are then synthesised in the decoder at virtual camera positions anywhere between the original left and right cameras. The virtual camera spacing should correspond to the average distance between a pair of human eyes. The intermediate-view synthesis can be achieved using disparity directed interpolation from the left- and right- view images. Disparity estimation is the most time- consuming task of the whole process. During the last few years, many algorithms for disparity estimation have been suggested. Matching approaches can be classified as areahlock-based schemes, objectlpixel-based schemes, and feature-based schemes. In this paper, a hierarchical genetic disparity estimation algorithm for multiview image synthesis is proposed. It is shown by simulation results with various real stereo images that the computational complexity is largely reduced compared to the above methods, which makes it possible to be used in a real-time system. The paper is organised as follows. Firstly, a new cost function for stereo. matching is formulated. Stereo matching using a hierarchical genetic algorithm is then introduced. Lastly some selected results of computer simulations and conclusions are given. 2. THE COST FUNCTION FOR STEREO MATCHING The epipolar lines in an image pair coincide with the horizontal scanlines if the camera axes are parallel. In this case, the corresponding points in both images must lie on the same horizontal scanline. Such stereo configurations reduce the search for correspondences from two- dimensions to one-dimension. In fact, a close look at the intensity profiles from the corresponding row of this class of the image pair reveals that the two intensity profiles differ only by a horizontal shift and a local foreshortening. The similarity between the one-dimensional intensity profiles of the two images suggests an optimization process would be suitable in which the only constraint being used is a minimum change of disparity values. This constraint is commonly known as the continuity constraint. Indeed, Barn [6] in 1987 attempted matching the parallel stereo images using simulated annealing. Robert [7] later incorporated the use of a multiresolution scheme together with a smoothness constraint similar to that of Barn into the constrained optimization process. In addition to the horizontal shift of corresponding pixels, they also allowed the corresponding pixels to undergo vertical shift (i.e. disparity in the vertical direction), so their matching Q-78Q3-6297-71Q01$1O.Q0 0 2000 IEEE 768 Authorized licensed use limited to: UNIVERSITY OF BRISTOL. Downloaded on January 22, 2009 at 11:00 from IEEE Xplore.  Restrictions apply.method is not restricted to only parallel stereo images. The energy function to be minimized, as expected, is more complicated than the one used by Barn. Unfortunately, like all constrained optimization problems, whether the system would converge to the global minimum is still an open problem. Dynamic programming [8] is also used for stereo matching. Since dynamic programming could only be practically applied along each epipolar line independently, the estimated disparities often exhibit obvious vertical discontinuities. Wei [4] proposed a stereo correspondence method by minimizing intensity and gradient errors simultaneously. In order to avoid local minima and slow convergence in the function minimisation, hierarchical Gaussian basis functions with very high computational complexity are used to parameterize the disparity function. Kim [9] proposed a genetic algorithm (block-based) optimization technique for stereo matching. The cost function is composed of the intensity-difference between two images and a simple smoothness of disparity. However, the vertical continuity can not be assured. The starting-point of the work presented here is the method introduced in [4]. The goal of our matching method is to overcome the difficulties mentioned above by combining different strategies and exploiting redundancy so as to increase the matching accuracy and to decrease the computational complexity. The employed strategies include the following: (1) Hat and non-flat regions are detected by using a morphological edge detector [5] ;  (2) A new cost function combining area and feature matching is proposed; (3) A hierarchical genetic matching procedure is used. Suppose that I,@, y) and Z,.(x, y) are the intensities (luminance and colour) of point (x ,  y) in the left and right image. Denote the disparity at position (x, y )  of the left image by dX,,. From the intensity preservation principle, it follows that Zl(x, y)=Z,.(x+d,,, y). However, since intensity measurements are not exact and all hypotheses are not absolute, the following cost function, including a smoothness constraint, a vertical-continuity constraint, and an ordering constraint, is minimized. It is given as follows where x,x 2 0 A*)’{o,x<o (3) F = { (XtY )I(X>Y) E D, q (XJ) > Threshold 1 (4) (5 )  N, = I(i,j- 2),(Lj - 1),(i,j),O,j+ 1),G9j + 2)) A reference window D of size p X q is placed around point ( i ,  j), and it is then compared with a search window in a given horizontal interval. It is very useful to define a disparity-offset do@,,, and the matching is performed within the search interval (dofie,, dranse+ dofie,). A feature set F consists of points in the reference window D whose output m(x, y) of the morphological edge detector (open-closing by reconstruction with a structuring element of 3x3 elements) is bigger than a given threshold and a, p, y,i l  are constants. In a process similar with that in [4], constraints on the disparity gradient are used to minimize either the morphological gradient of an image for feature points or the disparity Laplacian function for non- feature points in flat regions so as to serve as an intensity- adaptive smoothness constraint. Constraints of the local shape of the disparity surface at point (x, y )  aIso include the vertical-continuity constraint and the ordering constraint. When a step edge is caused by vertical discontinuity, the vertical-continuity constraint will smooth out the vertical disparity edge. The ordering-constraint check has been widely used in [2, 41. The ordering constraint is violated if di-i,,-di, j>1 (6) The ordering constraint used in equation (1) will add a penalty term to the cost function such that a global ordering is achieved. 3. A HIERARCHICAL GENETIC DISPARITY ESTIMATION ALGORITHM When the GA is applied to refine the displacement field, a population P is maintained which consists of N search points along the epipolar Iine, where N is the population size. The population P will evolve into another population P‘ by performing some genetic operations. Chromosomes with higher fitness values will have higher probabilities to be kept in the population of the next generation, and to propagate their offspring. When the iteration process converges, the improved disparity vector is expected to be contained in the mature population. Let the maximum search range dranSe be 2‘ and the disparity offset be . In order to reduce noise sensitivity and simultaneously reach higher efficiency, both the left image and right image are low-pass filtered and subsampled by a factor of two for (V-2) times in the horizontal and vertical direction. At the first hierarchy, the search range is set to 2v-2 769 Authorized licensed use limited to: UNIVERSITY OF BRISTOL. Downloaded on January 22, 2009 at 11:00 from IEEE Xplore.  Restrictions apply.be (-1, 3) for the left image and (-3, +1) for the right image, and the initial disparity vectors are all set to be zero. At the h-th hierarchy, the initial disparity field is upsampled by a factor of two in both directions from that at the (h-1)-th hierarchy. The above procedure repeats until the highest hierarchy is reached. The incremental scheme produces significant computational savings. Our proposed hierarchical genetic algorithm (HGA) consists of the following steps: (1). Initialization: At the first hierarchy an initial population of N=4 chromosomes (matching points) at point (i ,  J)  is randomly generated, biased at the central region of the search window. At the h-th hierarchy, an initial population of N chromosomes at point ( i ,  J)  is selected randomly from the region G=[u,b] as follows (7) a = min{d(i,j- 2),d(i,j- l),d(i,j),d(i,j+ l),d(i,j+ 2)] b = max{d(i,j - 2),d(i,j- l),d(i,j),d(i,j + l),d(i,j+ 2)} (2). Evaluation: The fitness function of each chromosome within a population is evaluated. The fitness functionfit is chosen to be lieu, eij > 0 -,e, = O  fifit(eii) = where ei,j is the matching criteria defined in equation (1). Hence the smaller the matching errors, the larger the fitness values. (3). Selection, crossover and mutation: Based on the fitness values of chromosomes in the current population, pairs of selected parents undergo a subsequent genetic operation, namely crossover, to produce pairs of children chromosomes that form a new pool of population as the next generation. During this process, the one having the larger fitness value is more likely to be selected. A uniform crossover is used here that allows production of offspring through combining and exchanging genes of the parent chromosomes. Mutation is used to generate N new mutated chromosomes from the seed through mutation operators. In order to reduce the computational complexity, this step might be omitted here. If so, there are N chromosomes in the pool of population. In order to avoid premature convergence, new N original chromosomes at point (i, j) are selected randomly from the current search range. (4). Survival competition: N chromosomes are selected from these 2N chromosomes according to their fitness values in the survival competition stage. Each chromosome can only be selected once. Chromosomes with larger fitness values will be picked up as the members of the population in the next generation, that is, we sort the 2N chromosomes and select the best N ones. The chromosome with the maximum fitness value is selected from the current population as the disparity vector of the current point. (5). Repetition: At each hierarchy, the HGA algorithm progresses point by point from the top left to the bottom right in the hierarchical left and right images and then the whole process for a whole hierarchical image repeats until convergence or a predefined number of iterations has been reached. 4. EXPERIEMTNAL RESULTS The robustness of our proposed approach is tested on real stereoscopic pairs. At the h-th hierarchy, the constraint parameters in equation (1) are selected empirically as follows a = 3q;p = 577; y = 5q;A = q (9) 77= " ,h  = 0, ..., V -2 Pv-2.9v-2 Based on experience, the size p X q  of the reference window D at the h-th hierarchy is also chosen to be ph = 4h = V - 2 - h . The threshold value defining the feature points is selected empirically to be the mean of the morphological gradient. Some results illustrating the performance of the HGA method and the hierarchical blockFeature-based method (HBM) [2-31 for AQUA are given in Figs. 1-2. Intermediate views have been constructed using the algorithm given in [2]. The synthesized central viewpoints of AQUA are presented in Fig. 1 for the HGA and the HBM method. In Fig. 2, images representing the horizontal component of dense L- R disparity fields for AQUA are shown. Darker gray levels mean larger negative horizontal vectors, whereas brighter gray levels mean larger positive horizontal vectors. For our method, the matching produces encouraging results. The matching tests of sparse lines in the left image and the corresponding deformed lines in the right image for all test sequences conform very well with the human perception of depth. Except for the possibility of blurring of the disparity edges, our proposed HGA method produces very accurate disparity estimation for most flat or non-flat regions. 5. CONCLUSIONS In this paper, a hierarchical genetic algorithm for disparity estimation has been presented. Each hierarchical image is divided into set of feature points and non-feature points. The image morphological gradient for feature points and the disparity Laplacian function for non-feature points are incorporated into the matching function to serve as an adaptive smoothness term. Meanwhile, a vertical- discontinuity constraint and an ordering constraint are also proposed to smooth out vertical disparity discontinuities 770 Authorized licensed use limited to: UNIVERSITY OF BRISTOL. Downloaded on January 22, 2009 at 11:00 from IEEE Xplore.  Restrictions apply.and to obtain more reliable disparity estimation. This offers a new scale-space solution to stereo matching when compared with multiresolutional block-based or feature- based approaches. The performance of the present method has been tested by computer experiments using natural stereoscopic sequences, and is shown to give image quality superior to conventional methods, while offering the reduction of computational complexity. ACKNOWLEDGEMENT The authors acknowledge to the support of the Virtual Centre of Excellence in Digital Broadcasting and Multimedia Technology (Theme 1) for the financial support of the work. REFERENCES [ 13 S .  Sethuraman, M.W. Siegel, A.G. Jordan, “Segmentation Based Coding of Stereoscopic Image Sequences,” Proc. SHE. Digital Video Compression: Algorithms and Technologies, vol. 2668,1996. [2] J. R. Ohm, and E Izquierdo, “An Object-Based System for Stereoscopic Viewpoint Synthesis, ” IEEE Trans. on Circuits and Systems for Video Technology, pp. 801-811, vol. 7, no. 5, October 1997. [3]W. Hoff and N. Ahuja, “Surface from stereo: integrating feature matching, disparity estimation and contour detection, ” IEEE Trans. Pattem Analysis and Machine Intelligence, vol. 1 1, [4] G. Wei, and W. Brauer, “Intensity- and Gradient-based stereo matching using hierarchical Gaussian basis functions”, IEEE Trans. Pattem Analysis and Machine Intelligence, vol. 20, no. 11, pp. 1143-1160, Nov. 1998. [5] R. Klette and P. Zamperoni, Handbook of image processing operators, John Wiley & Sons Ltd., ISBN 0471956422,1996. [6] Barnard, S. T. & Thompson, W. B. (1980), ‘Disparity Analysis of Images‘, IEEE Trans. ‘on Pattem Analysis and Machine Intelligence 2(4), 333-340. [7] Robert, L., Deriche, R. & Faugeras, 0. D. (1992), Dense Depth Map Reconstruction Using Multiscale Regularization, in Proc. Singapore International Conference on Image Processing, Singapore, pp. 123-127. [81 Y. Ohta and T. Kanade, “Stereo by inter- and inter-scanline search using dynamic programming,” IEEE Trans. Pattem Analysis and Machine Intelligence, vol. I, no. 2, pp. 139-154, March 1995. [9] S. H. Tan, K. K. Pang, and K. N. Ngan, “Classified Perceptual Coding with Adaptive Quantization,“ IEEE Trans. on Circuits and Systems for Video Technology, vol. 6, No. 4, 1996, pp. 121-136, 1989. pp.375-388. (a) The HBM method for  AQUA: (a)  The HBM method for  AQUA; (b) The HGA method for  AQUA Fig. I The synthesized central viewpoints of AQUA (b) The HGA method for  AQUA. Fig. 2 The lef-right disparity field for  AQUA. 77 1 Authorized licensed use limited to: UNIVERSITY OF BRISTOL. Downloaded on January 22, 2009 at 11:00 from IEEE Xplore.  Restrictions apply.

An Object-Based System for Stereoscopic Viewpoint Synthesis,

Classified Perceptual Coding with Adaptive Quantization,“

Dense Depth Map Reconstruction Using Multiscale Regularization, in

Disparity Analysis of Images‘,

Handbook of image processing operators,

Intensity- and Gradient-based stereo matching using hierarchical Gaussian basis functions”,

Segmentation Based Coding of Stereoscopic Image Sequences,”

Surface from stereo: integrating feature matching, disparity estimation and contour detection,

https://research-information.bris.ac.uk/files/3012259/luo_IEEE_image processing_2000.pdf

A hierarchical genetic disparity estimation algorithm for multiview image synthesis

Abstract

Similar works

Full text

Available Versions

Crossref

Explore Bristol Research