Typical problems for creation of consistent underwater mosaic are misalignment and inhomogeneous illumination of the image frames, which causes visible seams and consequently complicates post-processing of the mosaics such as object recognition and shape extraction. Two recently developed image blending methods were explored in the literature:  gradient domain stitching  and  graph-cut  method, and they allow for improvement of illumination inconsistency and  ghosting  effects, respectively. However, due to the specifics of underwater imagery, these two methods cannot be used within a straightforward manner. In this paper, a new improved blending algorithm is proposed based on these two methods. By comparing with the previous methods from a perceptual point of view and as a potential input for pattern recognition algorithms, our results show an improvement in decreasing the mosaic degradation due to feature doubling and rapid illumination change

Gu, Fan

Rzhanov, Yuri

English

Crossref

Optimal Image Blending for Underwater Mosaics

UNH Scholars' Repository

University of New HampshireUniversity of New Hampshire Scholars' RepositoryCenter for Coastal and Ocean Mapping Center for Coastal and Ocean Mapping2006Optimal Image Blending for Underwater MosaicsFan GuUniversity of New Hampshire, DurhamYuri RzhanovUniversity of New Hampshire, Durham, Yuri.Rzhanov@unh.eduFollow this and additional works at: https://scholars.unh.edu/ccomPart of the Oceanography and Atmospheric Sciences and Meteorology CommonsThis Poster is brought to you for free and open access by the Center for Coastal and Ocean Mapping at University of New Hampshire Scholars'Repository. It has been accepted for inclusion in Center for Coastal and Ocean Mapping by an authorized administrator of University of NewHampshire Scholars' Repository. For more information, please contact nicole.hentz@unh.edu.Recommended CitationGu, Fan and Rzhanov, Yuri, "Optimal Image Blending for Underwater Mosaics" (2006). Center for Coastal and Ocean Mapping. 1159.https://scholars.unh.edu/ccom/1159

University of New HampshireUniversity of New Hampshire Scholars' RepositoryCenter for Coastal and Ocean Mapping Center for Coastal and Ocean Mapping9-2006Optical Image Blending for Underwater MosaicsFan GuUniversity of New Hampshire, DurhamYuri RzhanovUniversity of New Hampshire, Durham, Yuri.Rzhanov@unh.eduFollow this and additional works at: https://scholars.unh.edu/ccomPart of the Computer Sciences Commons, and the Oceanography and Atmospheric Sciences andMeteorology CommonsThis Conference Proceeding is brought to you for free and open access by the Center for Coastal and Ocean Mapping at University of New HampshireScholars' Repository. It has been accepted for inclusion in Center for Coastal and Ocean Mapping by an authorized administrator of University of NewHampshire Scholars' Repository. For more information, please contact nicole.hentz@unh.edu.Recommended CitationGu, Fan and Rzhanov, Yuri, "Optical Image Blending for Underwater Mosaics" (2006). IEEE Oceans. 372.https://scholars.unh.edu/ccom/372Optimal Image Blending forUnderwater MosaicsFan Gu, Yuri RzhanovCenter for Coastal and Ocean Mapping, University ofNew Hampshire24 Colovos Rd,Durham, NH, 03824,USAEmail: {fan, yuri}gccom.unh.eduAbstract- Typical problems for creation of consistentunderwater mosaic are misalignment and inhomogeneousillumination of the image frames, which causes visible seams andconsequently complicates post-processing of the mosaics such asobject recognition and shape extraction. Two recently developedimage blending methods were explored in the literature:"gradient domain stitching" and "graph-cut" method, and theyallow for improvement of illumination inconsistency and"ghosting" effects, respectively. However, due to the specifics ofunderwater imagery, these two methods cannot be used within astraightforward manner. In this paper, a new improved blendingalgorithm is proposed based on these two methods. By comparingwith the previous methods from a perceptual point of view and asa potential input for pattern recognition algorithms, our resultsshow an improvement in decreasing the mosaic degradation dueto feature doubling and rapid illumination change.I. INTRODUCTIONIn the recent years, mosaics created from individual imagesacquired underwater are attracting more and more attentionfrom marine geologists and biologists. Applications can beclearly divided into two categories: those targeting extractionof quantitative information (distances, sizes, shapes, etc.), andthose attempting to create a consistent continuous image,possibly at the expense of minor local distortions. (A specialcategory, aiming at accurate recovery of three-dimensionalinformation about the seafloor, is capable of achieving bothgoals, but requires principally different approach, and hassubstantially higher level of complexity.)In reality, due to limited visibility underwater, artificial and,as a consequence, spatially inhomogeneous illumination, andthe parallax issues, most underwater images are fuzzy anddifficult to process. In this paper, we are not concerned withthe ability to measure distances and sizes as accurately aspossible. Algorithms for object recognition and shapeextraction are typically tolerant to scaling and insignificantdistortions, but can be easily confused by feature doubling andrapid changes in illumination. Our goal is to diminish theeffects of inhomogeneous illumination, which are almostalways present in the case of artificial lighting, and to combineindividual image frames into a single mosaic in some optimalway. Note that "optimal" may have different meaningsdepending on intended consumer: scientist, trying to deducelarge-scale interrelationships; computer program, extractingshapes according to some specific rule; or a high-schoolstudent learning about a deep-sea environment.Current blending techniques can be divided in two maincategories, assuming that the images have already been aligned:One approach is an optimal seam algorithm [1-3] that searchesfor a curve in the overlap region on which the differencesbetween two overlapping images are minimal, and then eachimage is copied to the corresponding side of the seam. Onesimple and commonly used method is the minimum cutmethod which employs dynamic programming [1], but it workswell only when two images are involved. As opposed to this"memoryless" approach, the graph-cut method [4, 5] wasproposed that can be applied when more than two images areneeded to be mosaiced. However, the seam may still be visiblewhere brightness of neighboring original images differsdramatically.Another category is aiming at smoothing the transitionbetween two images. Most common blending techniquesemploy simple averaging of images in the overlapping regions.This results in ghosting artifacts, blurring, and visible seamsthat degrade the mosaic. Some improvement of this methodwere proposed, such as feathering or alpha blending [6] whichemploys the special weighting functions, multi-resolutionblending [7-9] which takes advantage of the characteristics ofdifferent sub-bands, and gradient domain stitching [10-12,16],which is designed to remove sharp changes of brightnessacross the frame boundaries. However, blurring and ghostingeffects could not be avoided due to misalignment of theunderwater imagery.In our paper, the methods mentioned above are explored inapplication to underwater images. Due to the complexity ofunderwater imagery, the defects of these methods are moreapparent, and thus other practice should be considered in orderto get higher quality mosaics for either post-processing orsimple viewing. Our proposed blending method is usingadvantages of the graph-cut technique and gradient domainstitching method, and has achieved a significant improvementover the existing algorithms.II. THEORETICAL BACKGROUNDIn this section, methods of gradient domain stitching andgraph-cut are highlighted and application details are introduced.1-4244-01 15-1/06/$20.00 ©2006 IEEEA. Gradient Domain StitchingComputation in the gradient domain was recently used incompression of dynamic range [12], image editing [11], imageinpainting [13] and separation of images to layers [14]. In [10],two approaches were proposed for image stitching in thegradient domain, and the previous spatial methods (such asfeathering, pyramid blending and optimal seam) performed ingradient domain of the images were compared with theiroriginal methods. Results show an improvement in overcomingthe photometric inconsistencies and small geometricmisalignment between the stitched images. Performance here issimilar to image editing [11], which suggests editing images bymanipulating their gradients. One of the editing applicationsconcerned is the object insertion, where an object is selectedand cut from an image, and inserted into a new backgroundimage. The insertion process is done by solving the Poissonequation in the gradient field of the inserted patch, withboundary conditions defined by the background image.Mathematically, the gradient of a two-variable function(here the image intensity function 1) is at each image point a2D vector with the components given by the derivatives in thehorizontal and vertical directions, that is:VI= (aI/ax3aI/ay). (1)With some additional assumptions, the derivative of thecontinuous intensity function can be calculated as a function ofthe sampled intensity function, i.e. the digital image. Forexample, the gradient for digital images approximated by theforward difference:VI(x, y) = (I(x + 1, y) - I(x, y), I(x,y + 1) - I(x, y)) (2)In order to reconstruct the pixel values, integration should beperformed, however, the conservativeness can rarely beachieved in this case [11]. Other methods were proposed tosolve this problem such as Fourier basis function algorithm [15]which orthogonally projects the gradient values onto a finiteset of ortho-normal basis functions spanning the set ofintegrable vector fields; another method is to search thefunction over the space of all 2D potential functions whosegradient is closest in the sense of least-squares. As proved in[11], the second method is equivalent to solving of thefollowing Poisson equation:V2I = divG,where Laplacian values ofI is expressed as:v21 = 2 +ax 2 ay 2and the divergence of gradient vector G is:divG= X+k OyApproximating them with the standard finite differencesyields a linear system of equations, where the Laplacian of I isexpressed as:V21 = I(x + 1, y) + I(x -1,y) + I(x,y + 1) + I(x,y -1) - 41(x,y)(6)and the divergence ofG is:divG = G, (x, y) - G (x -1, y) + Gy(x, y +1) -Gy(x, y -1)(7)In solving the Poisson equation, boundary conditions werereported to be chosen differently according to the applications.B. Graph-cut methodThe graph-cut method [5] is designed to find a boundarybetween two images in such a way that the seam is the leastnoticeable. This search is formulated in terms of finding theminimum of a certain energy function. The graph-cutalgorithm is based on the principles of combinatorialoptimization, and has attracted a lot of attention recently due toits ability to solve problems of this type extremely effective.Principally similar, dynamic programming method was firstproposed in [1], which also incorporate seams finding process.However, specifics of implementation impose restrictions onthe ways the seam is allowed to follow. This may lead tomissing of good seams that cannot be modeled within theimposed structure. In addition, dynamic programming is"memoryless" and cannot explicitly improve existing seams.This gives limitations when appending new images to theexisting images. Graph-cut technique overcomes thesedisadvantages by treating each pixel uniformly and is also ableto place patches over the existing images.Specifically, let x andy be two adjacent pixel positions in theoverlap region. Let A(x) and B(y) be the pixel values in thesame color channel coming from original and new images,respectively. The matching quality cost E can be definedbetween the two adjacent pixels x and y that are copied frompatches A and B to be:E(x, y, A, B)= IIA(x) -B(x)ll + IIA(y) -B(y)ll, (8)where * denotes an appropriate norm.II. PROPOSED METHOD(3) The method proposed in this paper is overcoming the defectsof the single graph-cut technique, which would have apparentseam when two images have inhomogeneous illumination, andthe single gradient domain stitching, which can still cause(4) bluffing in a misaligned case. As mentioned in the previoussections, our method is based on the graph-cut in the gradientdomain. Different from the method of optimal seam in gradientdomain [10], which is the dynamic programming based method,(5) graph-cut here is performed on the overall image values, and ismore flexible in defining the "cut" area.The procedure is as follows, assuming that two images ID, IBhave already been aligned and we take only one color channelfor illustration:1) Following the formula of (4), we calculate the gradientvalues oftwo images, ID, IB, obtaining GD, GB.2) According to the overlapping area (which in general isan irregular polygon), a boundary box is obtained,which is composed of three parts: overlapping area,and parts that have contributions from only one of theimages.3) Within the boundary box, execute the graph-cuttechnique and get the graph-cut mask, using weightingfunction to smooth the boundary cut and obtaining thefinal mask.4) Fill in the boundary box with gradient values accordingto the mask matrix, and use it as a source term of thePoisson equation. Boundary values of the boundarybox are from the original pixel values of two imagesgiven the boundary of the mask.5) Reconstruct the spatial values of the boundary box bysolving the Poisson equation with Dirichlet boundaryconditions.6) Put the corresponding reconstructed values back in thefinal mosaic.In practice, the images are part of a sequence, for example,captured from a video tape. Transformations relatingconsecutive images are either deduced from the navigation data,or estimated from the imagery. Frames are added sequentiallyto already existing mosaics.IV. EXPERIMENTSIn this section, the proposed method is applied and resultsare shown, then experiments using different methods andresults are compared from the perceptual point of view.In our case, size of the mosaic can rapidly grow so that atypical desktop computer cannot handle it. Addition of a newindividual image frame to an existing mosaic is described interms of interaction between two images: one, represented by arectangular footprint, and another, bounded by a complexpolygon, which in general may consist of several disjointedparts, be concave, and have holes. For simplicity ofcomparison, our experiments are performed on twooverlapping gray-level frames the video footage courtesy of Dr.R. Vrijenhoek, MBARI.A. Results oftheproposedmethodBelow are the results following the performance proceduresintroduced in the previous section: first, the original imagesoverlapping ID and IB are given in Fig. 1, then the position anddimension of boundary box is illustrated on the image ofmosaics in Fig. 2. Graph-cut matrix with the narrow weightingfunction (Here, we give the result of three pixel wide band) isshown in Fig. 3. The mask, which is filled by the sourcegradient values and boundary conditions are in Fig. 4. In thiscase, the light gray stands for the gradient values from GD, andthe darker gray stands for the values from GB, while the blackvalues in the bottom are the gradient values from GB , but notout of the overlapping area. Efficient solution of the Poissonequation can be achieved by a variety of methods. We havechosen the direct solver from the INTEL Math Kernel Library,v.8.01.B. Resultsfrom other methodsFor comparison, we performed the following blendingmethods on the same images, and their results were given inFig. 5-10:1) the direct averaging blending method,2) feathering method,3) feathering in gradient domain,4) direct graph-cut method,5) graph-cut method in gradient domain without weightingfunction methods.Fig. 1. Original Images ID and IB.1BFig. 2. The position of the boundary box.Fig. 3. Graph-cut mask of the boundary boxFig. 4. Mask for gradient values filling.Fig. 5. Graph-cut in gradient domain with a weighting function.Fig. 7. Feathering blending method.Fig. 8. Direct graph-cut method.Fig. 9. Feathering in gradient domain method.Fig. 6. Averaging blending method.Fig. 10. Graph-cut in gradient domain without weighting function.From Fig. 6, it can be observed that direct averaging methodgive rise to apparent bluffing and doubling, in addition, it doesnot improve the illumination difference in two images. In Fig.7, the weighted averaging seems to be improved in sense ofboth of these two disadvantages above, still, the seam due tothe illumination difference is apparent on the right of themosaic. In terms of decreasing the bluffing and ghostingeffects, graph-cut technique in Fig. 8 gives a good result,however, the seam is more apparent because, in this case, thedifference of illumination is large and the seam cannot behidden among the complex texture of the images. Fig. 9, whichperforms in the gradient domain with weighting function makethe whole mosaic more homogeneous in illumination, however,comparing with the original images, details are bluffed and notas distinctive as in the direct graph-cut method. Fig. 10 whichemploys graph-cut mask in the gradient domain, the mosaicillumination is more homogeneous comparing with directgraph-cut method, while the details are clearer compared to thegradient domain feathering. But there are some blocks of whitealong the region of cut, which are apparent artifacts. It mightbe due to the inconsistency of the source term along the seam.In our experiment, when using the weighting function alongthe cut, the effect is less apparent, as shown in Fig. 5.V. CONCLUSIONSDue to the artificial lighting and 3D content of imagedterrain, imagery taken underwater almost always suffers frominhomogeneous illumination and feature misalignment, whenmosaiced. This causes degradation of the final product andmakes it more difficult to post-process. Often used mean valueaveraging blending technique can hardly satisfy the demand ofpost processing such as feature extraction or human viewleisure.These days, a lot of blending techniques were explored inthe area of image processing. Most of them fail when it comesto the underwater images, which have different specifics. Wereviewed the existing popular methods and combined them in away to facilitate in post-processing of underwater mosaics.Specifically, we have combined the graph-cut method designedto improve on image bluffing and ghosting, caused by localmisalignments, and the gradient domain stitching technique,which helps with lighting inhomogenities and exposureartifacts. Employing the graph-cut in the gradient domaineliminates their defects with the weighting band. Experimentalresults show the effectiveness of the proposed methods,comparing with other existing methods such as specialaveraging, feathering, graph-cut, and feathering, graph-cut ingradient domain. The reason for artifacts, occasionallyoccurring in the reconstructed process, requires furtherinvestigation.ACKNOWLEDGMENTThe authors are grateful to Dr. R. Vrijenhoek from MBARIfor the data.REFERENCES[1] A. Efros and W. Freeman, "Image Quilting for texture synthesis andtransfer", SIGGRAPH2001, Proceedings of Computer Vision, pp. 1033-1038, 2001.[2] D. Milgram, "Computer methods for creating photomosaics", IEEETrans. Computer 23, pp. 1113-1119, 1975.[3] J. Davis, "Mosaics of scenes with moving objects", CVPR , pp. 354-360,1998.[4] Y. Boykov, 0. Veksler and R. Zabih, "Fast Approximate EnergyMinimization via Graph Cuts", ICCV(1), 1999.[5] V. Kwatra, A. Schodl, 1. Essa, G. Turk and A. Bobick, "GraphcutTextures: Image and Video Synthesis Using Graph Cuts", ACMTransaction On Graphics, pp. 277-286, 2003.[6] M. Uyttendaele, A. Eden and R. Szeliski, "Eliminating ghosting andexposure artifacts in image mosaics", IEEE Computer SocietyConference on Computer Vision and Pattern Recognition, pp. 509-516,2001.[7] P. Burt and E. Adelson, "A Multiresolution Spline with Application toImage Mosaics", ACM Transactions on Graphics, V2., pp. 217-236,1983.[8] S. Peleg, "Elimination of Seams from photomosaics", ComputerGraphics and Image Processing, 16(1): pp. 90-94, 1981.[9] Z. Hu and S. T. Acton, "Morphological Pyramid Image Registration",Proc. IEEE Southwest Symposium on Image Analysis and Interpretation,2000[10] A. Levin, A. Zomet, S. Peleg and Y. Weiss, "Seamless image stitching inthe gradient domain". ECCV'04.[11] P. Perez, M. Gangnet and A. Blake, "Poisson Image Editing",Proceeding ofSIGGRAPH 2003, pp. 313-318.[12] R. Fattal, D. Lischinski and M. Werman, "Gradient domain high dynamicrange compression", Proceeding ofSIGGRAPH 2001, pp. 249-3 56.[13] M. Bertalmio, G. Sapiro, V. Caselles and C. Ballester, "ImageInpainting", Proceedings ofACMSIGGRAPH 2000, pp. 417-424.[14] Y. Weiss, "Deriving intrinsic images from image sequences", Int. confon Computer Vision, pp. 68-75, 2001.[15] R. Fankot and R. Chellappa, "A method for enforcing integrability inshape from shading algorithms", IEEE Trans. On Patten Analysis andMachine Intelligence, 10(4): pp. 439-451, 1988.[16] A. Agarwala, M. Dontcheva, M. Agrawala, S. Drucker, A. Colburn, B.Curless, D. Salesin and M. Cohen, "Interactive Digital Photomontage",ACMSIGGRAPH 2004.