In two-view stereo matching, the disparity of occluded pixels cannot accurately be estimated directly: it needs to be inferred through, e.g., regularisation. When capturing scenes using a plenoptic camera or a camera dolly on a track, more than two input images are available, and - contrary to the two-view case -pixels in the central view will only very rarely be occluded in all of the other views. By explicitly handling occlusions, we can limit the depth estimation of pixel (P) over right arrow to only use those cameras that actually observe (p) over right arrow. We do this by extending variational stereo matching to multiple views, and by explicitly handling occlusion on a view-by-view basis. Resulting depth maps are illustrated to be sharper and less noisy than typical recent techniques working on light fields

Aelterman, Jan

Donné, Simon

Goossens, Bart

Philips, Wilfried

English

Ghent University Academic Bibliography

VARIATIONAL MULTI-IMAGE STEREO MATCHINGSimon Donne´, Bart Goossens, Jan Aelterman, Wilfried PhilipsGhent UniversityiMinds - IPI - UGent{Simon.Donne, Bart.Goossens, Jan.Aelterman, philips}@telin.ugent.beABSTRACTIn two-view stereo matching, the disparity of occluded pixelscannot accurately be estimated directly: it needs to be inferredthrough, e.g., regularisation. When capturing scenes usinga plenoptic camera or a camera dolly on a track, more thantwo input images are available, and – contrary to the two-view case – pixels in the central view will only very rarelybe occluded in all of the other views. By explicitly handlingocclusions, we can limit the depth estimation of pixel ~p toonly use those cameras that actually observe ~p. We do thisby extending variational stereo matching to multiple views,and by explicitly handling occlusion on a view-by-view basis.Resulting depth maps are illustrated to be sharper and lessnoisy than typical recent techniques working on light fields.Index Terms— stereo, light field, depth estimation1. INTRODUCTIONDepth maps for scenes can be acquired in several ways. Anactive measurement of scene depth is possible using time-of-flight cameras; yet these are still limited in spatial resolution.In stereo matching techniques, the disparity u(x, y) links pix-els on row y and column x in image IC to pixels in image IRrepresenting the same scene point. When the cameras sharetheir viewing plane and are connected by a line segment par-allel to their scan-line direction, the images are said to be rec-tified and the correspondences lie on the same scan line:IC(x, y) = IR(x+ u(x, y), y) (1)The actual depth of a given pixel is inversely proportional toits disparity (from trigonometry, e.g. as in figure 1).As technology advances, it becomes more and more feasi-ble to capture scenes with more than two cameras, for exam-ple with a plenoptic camera [1], which results in a light fieldrepresentation of the scene [2]. As more cameras observe thescene, all this information can improve the disparity/depth es-timation for the scene.This work was performed within the iMinds ASPRO+ project and”Multi-camera human behavior monitoring and unusual event detection”(G0398111N). Simon Donne´ is funded by BOF grant 01D21213 and BartGoossens is a postdoctoral researcher for FWO.Pbaseline widthx1 x1'x2disparityI ICRFig. 1: The central camera and the reference camera observethe same point P at x1 and x2. The orthogonal distance fromthe baseline to point P can be calculated when the baselinewidth, disparity and focal distance of the cameras are known.In this paper we evaluate the extension of the variationalstereo matching [3] approach to a light field set-up as in fig-ure 1. We start with an overview of existing work. In sec-tion 3 we formulate the problem mathematically and describeour proposed method. Finally, results are presented and theconclusion is drawn.2. EXISTING WORKTraditional approaches to (two-image) stereo matching con-sist of computing large cost fields which are aggregated overneighbourhood windows [5, 6] to achieve spatial regularisa-tion. For each pixel, a cost per candidate disparity value iscomputed. These cost fields are then aggregated over spa-tial neighbourhoods after which the best disparity for eachpixel is chosen. Extensions include the use of image trans-forms, such as the census transform [7, 3] or a normalizedcross-correlation [8], to make the cost field more robust to il-lumination changes and other influences, e.g. vignetting. Ourproposed method will allow for multiple image transforms tobe used simultaneously, in order to allow us to complementthe strengths of one transform with those of another as in [9].Fig. 2: Pixels are unlikely to be occluded in all views, because the occlusion occurs in opposite directions in case of a centralview. From left to right: the central view, the pixels not visible in the right view and the pixel invisible in the left one. The inputis a detail from the dolls sequence of Middlebury [4].Fig. 3: An example slice through a light field showing thesame scan line in all of the views. From top to bottom in theimage the camera is moving from left to right. The closer tovertical its trajectory is, the closer a pixel is to the baseline.Borrowing from optical-flow techniques such as [8], a re-formulation of stereo matching results in a more direct opti-misation of the disparity map [3] in a coarse-to-fine approach.The authors of [10] adopt a similar approach in order to accu-rately estimate disparity maps from light fields. Our proposedmethod will explicitly take occlusions into account.An alternative approach to computing depth from lightfields is presented in [11]: in a light field representation,the depth of a pixel is equivalent to the slope of the line ittraces in the light field, illustrated in figure 3. Through visualevaluation we illustrate that our proposed method results insmoother depth maps while respecting image boundaries.This results in more visually pleasing interpolated views:when generating new views, one of the most important fac-tors is the adherence to depth discontinuities in the scene.While plenoptic cameras generally have a two-dimensionalgrid of cameras (due to the microlenses), the focus of our re-search lies on sequences recorded with a camera mounted ona track. In such cases, the camera trajectory is restricted to asingle dimension. The extension of the work presented hereto a two-dimensional grid is relatively straightforward.3. MULTI-VIEW STEREOWe wish to estimate the disparity map of a central viewIC(x, y) based on a one-dimensional light field comprisingthe central camera and K additional views Ik(x, y) (see fig-ure 1). Trigonometry shows that the ratio of disparities fortwo different cameras relative to the central camera is directlyproportional to the ratio of these cameras’ (signed) distancesto the central camera.This means that, when denoting the location of the kthcamera by θk, the relation is:IC(x, y) = Ik(x+ θku(x, y), y), ∀k ∈ [1,K] . (2)This formulation assumes a one-dimensional light field. Inthe case of a two-dimensional light field this model shouldsimilarly include a vertical shift in correspondence matches.In this paper we will restrict ourselves to a single dimension.Ideally, the estimate of the disparity map fulfils equa-tion (2) in all pixels of the image. However, this is not thecase in areas of occlusion (where two pixels in Ik(x, y) aremapped onto the same pixel in IC(x, y)) or simply becausethe pixel lies outside of the bounds of Ik(x, y). Two-viewmethods generally use regularisation to estimate the disparityof such pixels [3].In the case of a one-dimensional light field, e.g. from acamera dolly on a track, it is unlikely for a pixel to be oc-cluded in all of the views (both to the left and to the right ofthe central view) as illustrated in figure 2. Generally, eachpixel is visible in at least one of the other views. Pixels whichare occluded in all views are only rarely caused solely by oc-clusions within the scene – they are most often caused bypixel correspondences being occluded in part of the views andlying outside of the image bounds in the others.3.1. Cost FunctionIdeally, equation (2) is fulfilled in all of the pixels in the im-age. To express the quality of an estimate u, we first warp theimage with the estimated disparity map and call it I˜k:I˜k(x, y, uˆ) = Ik(x+ θkuˆ(x, y), y). (3)Per equation (2) this warped version should resemble the cen-tral view as much as possible, which we evaluate with theEuclidean norm:d(IC , Ik, uˆ) =∑(x,y)∈Ω‖IC(x, y)− I˜k(x, y, uˆ)‖2, (4)where Ω is the set of all of pixels in the image plane.Fig. 4: Examples of neighbourhood transforms for an inputneighbourhood (left). In the middle, the census transform re-sults in either -1,0 or 1 depending on whether it has a lowervalue, a similar value or a higher value than the central pixel.To the right, the normalization transform subtracts the meanfrom each pixel and divides by the neighbourhood variance.In [10] the authors note that the ℓ2 norm can be sensitiveto outliers, for example from occlusions, and instead use theℓ1 norm. Because of our explicit occlusion handling, we canuse the ℓ2 norm which has the benefit of being differentiable.As the authors of [10] state, the comparison between theground truth IC(x, y) and I˜k (the current modelled view forthe central camera) is susceptible to illumination changes (e.g.due to vignetting or dynamic scenes). They use a structure-texture decomposition in order to circumvent this. We willinstead use various image transforms to achieve illumination-invariance, as was discussed in [9] for optical flow. Examplesof such transforms include the census transform [7] and thenormalization transform [8] (illustrated in figure 4). The lat-ter results in the normalized cross-correlation when using theEuclidean norm in equation (4), as shown in [8].Transforming the images in T ways, we denote the tthtransformed version of input image k by FtIk. Then the dataerror term for camera k and transformFt is d(FtIC ,FtIk, uˆ).The complete data fidelity term is:Ed(uˆ) =1KT∑t=1λt|Nt|K∑k=1d(FtIC ,FtI˜k, uˆ). (5)The divisions by K (the number of cameras) and |Nt| (theneighbourhood size for the tth transform) imply that theweighting between data fidelity and regularisation (see later)does not change as the number of input images or imagetransforms increases.The λt are weighting factors for the various transforms,which serves two purposes. First of all, the weighting fac-tors equalize the range of the transforms so that one trans-form does not have much more influence than another simplybecause the transformed values are larger by an order of mag-nitude. Secondly, the weighting factors also allows to tradeoff the data fidelity against the regularisation introduced later.As mentioned earlier, we explicitly take occlusions intoaccount. With the current formulation of the cost function,this is done by limiting the summation in equation (4) to thenon-occluded pixels. Additionally, in line with the earliernote on a constant weighting between data fidelity and reg-ularisation, the division by K in equation (5) is revised on apixel-by-pixel basis, now replaced by the number of views inwhich each pixel is visible. To decide whether a pixel (x, y)from the central view is visible in a given view, we use a z-buffer based warping: (x, y) is visible in the generated view ifand only if it is the pixel closest to the camera that is warped toits corresponding pixel. This number will generally be largerthan zero, but on image boundaries it may well be some pix-els are visible in no other view. In this case, we disregard datafidelity information for this pixel and infer the disparity solelyfrom regularisation.We adopt the same regularisation term as [8]. Based onthe bilateral filtering, it allows depth discontinuities only inthose locations where color discontinuities are present. Thisassumes that the foreground and background have distinctcolours. Denoting the bilateral filter coefficient between pix-els (x, y) and (p, q) as b(x,y),(p,q), the regularisation term isdefined over the neighbourhoods N (x, y):Es(uˆ) =∑(x,y)∈Ω∑(p,q)∈N (x,y)b(x,y),(p,q)‖uˆ(x, y)− uˆ(p, q)‖1.(6)3.2. OptimisationThe optimisation of the cost function results in the followingminimization problem:uˆ = argminuEd(u) + Es(u) (7)Our minimization approach for equation (7) is basedon [8], adjusted to our cost function. Non-differentiabilityof the smoothness term is resolved by defining the linearoperator K: Ku((x, y), n) = u(N(x,y)(n)) − u(x, y),where n is a linear index to the regularisation neighbour-hood. Using F (~y) = ‖~b · ~y‖1, the smoothness term becomesEs(u) = F (Ku). Finally, the problem is reformulated usingprimal-dual techniques [13]:uˆ = argminumaxqEd(u) + 〈Ku, q〉 − F∗(q). (8)In this expression, 〈·, ·〉 is the inner product in R|Ω|×|N|, andF ∗ is the convex conjugate of F :F ∗(q) ={0 if q ∈ Q∞ elsewhereQ = {q ∈ R|Ω|×|N| |∀~i ∈ Ω, ~s ∈ N~i : ‖q(~i, ~s)‖1 ≤ b~i,~s}(9)Now we optimise the problem by alternating gradient de-scent (respectively gradient ascent) between the variables uand q. In order to make the data term easily differentiable,we use the the first-order Taylor approximation in terms ofchanges to the disparity map in equation (3):I˜k(x, y, u+∆u) ≈ I˜k(x, y) + θk∆u(x, y)∂∂xI˜k(x, y),(10)Fig. 5: Depth estimation for the couch dataset from [11] with 10 input frames. From left to right: the input image for the centralcamera, our depth estimate, a detail of the depth estimate from [11] and our estimate for that same detail.Fig. 6: Depth estimation for the mona dataset from [12]. From left to right: the input image for the central camera, our depthestimate, the depth estimate from [10] for a detail and our estimate for that same detail. Note that we only used the central linein the light field for each pixel, while [10] exploited the entire light field.Fig. 7: Illustration for the use of multiple input images: theleft image shows estimation using only two images (4 and 5),while the right shows the result using all five. Input imagescourtesy of the Middlebury 2003 dataset[15].optimising∆u in each iteration. Because the solution methodis based on a first-order Taylor approximation the step ∆ushould not be too large. In order to enforce this, proximalpoint terms are added to equation (8) as in [14] (τ and η aretunable step sizes):12τ‖u− u(i)‖2 −12η‖q − q(i)‖2. (11)4. RESULTSIn figure 7 we illustrate that our estimate from more than two(five in this example) images is more accurate than the esti-mation based on two images, as one would expect. Secondly,we compare our depth estimate to the results given by [11]and [10] as seen in figures 5 and 6 (best viewed in colour).The proposed method results in less noisy estimations andour estimate follows the edges better. Remaining problemsare structures smaller than their own disparity (the small barin figure 5) and parts of the image where foreground and back-ground have very similar colours (the painting and the plantin figure 6).5. CONCLUSIONHere we have presented an approach to depth estimation inmulti-camera set-ups inspired by two-view stereo matching.Through explicit occlusion handling we are able to exploitthe characteristic of light fields that a pixel is only rarely oc-cluded in all of the other views (and then usually just becausein some views it lies outside of the image). We use multipleimage transforms, complementing the strong suits of one bythose of another. When comparing the output to that of ex-isting techniques, we illustrate that our proposed method re-sults in smoother disparity estimates adhering better to imagebounds: an important characteristic when using the estimateddisparities in view interpolation.Applying the proposed method to two-dimensional lightfields rather than one-dimensional ones is straightforward andwill allow even better occlusion handling. Yet, for the pro-posed method to be applicable in more practical applications,requiring all camera locations to be known accurately shouldbe less stringent: jointly estimating their locations and the dis-parity may provide the solution. This will be the topic of ourfuture work.6. REFERENCES[1] E.H. Adelson and J.Y.A. Wang, “Single lens stereo witha plenoptic camera,” Pattern Analysis and Machine In-telligence, IEEE Transactions on, vol. 14, no. 2, pp. 99–106, Feb 1992.[2] Marc Levoy and Pat Hanrahan, “Light field rendering,”in Proceedings of the 23rd Annual Conference on Com-puter Graphics and Interactive Techniques, New York,NY, USA, 1996, SIGGRAPH ’96, pp. 31–42, ACM.[3] R. Ranftl, S. Gehrig, T. Pock, and H. Bischof, “Pushingthe limits of stereo using variational stereo estimation,”in Intelligent Vehicles Symposium (IV), 2012 IEEE, June2012, pp. 401–407.[4] D. Scharstein and Chris Pal, “Learning conditional ran-dom fields for stereo,” in Computer Vision and Pat-tern Recognition, 2007. CVPR ’07. IEEE Conferenceon, June 2007, pp. 1–8.[5] Daniel Scharstein and Richard Szeliski, “A taxonomyand evaluation of dense two-frame stereo correspon-dence algorithms,” International Journal of ComputerVision, vol. 47, no. 1-3, pp. 7–42, 2002.[6] J. Braux-Zin, R. Dupont, and A. Bartoli, “A generaldense image matching framework combining direct andfeature-based costs,” in Computer Vision (ICCV), 2013IEEE International Conference on, Dec 2013, pp. 185–192.[7] Ramin Zabih and John Woodfill, “Non-parametric lo-cal transforms for computing visual correspondence,”in Computer Vision ECCV ’94, Jan-Olof Eklundh, Ed.,vol. 801 of Lecture Notes in Computer Science, pp. 151–158. Springer Berlin Heidelberg, 1994.[8] Marius Drulea and Sergiu Nedevschi, “Motion estima-tion using the correlation transform,” Image Processing,IEEE Transactions on, vol. 22, no. 8, pp. 3260–3270,2013.[9] Li Xu, Jiaya Jia, and Y. Matsushita, “Motion detail pre-serving optical flow estimation,” Pattern Analysis andMachine Intelligence, IEEE Transactions on, 2012.[10] Stefan Heber, Rene Ranftl, and Thomas Pock, “Varia-tional shape from light field,” in Energy MinimizationMethods in Computer Vision and Pattern Recognition,Anders Heyden, Fredrik Kahl, Carl Olsson, Magnus Os-karsson, and Xue-Cheng Tai, Eds., vol. 8081 of LectureNotes in Computer Science, pp. 66–79. Springer BerlinHeidelberg, 2013.[11] Changil Kim, Henning Zimmer, Yael Pritch, Alexan-der Sorkine-Hornung, and Markus Gross, “Scene re-construction from high spatio-angular resolution lightfields,” ACM Trans. Graph., vol. 32, no. 4, pp. 73:1–73:12, July 2013.[12] SvenWanner, Stephan Meister, and Bastian Goldluecke,“Datasets and benchmarks for densely sampled 4d lightfields,” in VMV’13, 2013, pp. 225–226.[13] Antonin Chambolle and Thomas Pock, “A first-orderprimal-dual algorithm for convex problems withappli-cations to imaging,” Journal of Mathematical Imagingand Vision, vol. 40, no. 1, pp. 120–145, 2011.[14] R. Rockafellar, “Monotone operators and the proximalpoint algorithm,” SIAM Journal on Control and Opti-mization, vol. 14, no. 5, pp. 877–898, 1976.[15] D. Scharstein and R. Szeliski, “High-accuracy stereodepth maps using structured light,” in Computer Visionand Pattern Recognition, 2003. Proceedings. 2003 IEEEComputer Society Conference on, June 2003, vol. 1, pp.I–195–I–202 vol.1.

Variational multi-image stereo matching

Abstract

Similar works

Full text

Available Versions

Ghent University Academic Bibliography

Crossref