We propose in this paper a new formulation of the equation of the optical flow enabling to compute global and local motions of multi-structure objects (flowers and petals, trees and leaves,...). The displacement fields are computed using a Markovian Random Field (MRF) model. Local and global components of the vector flow are both explicitly retrieved. The minimization of the Gibbs energy is achieved with a down-scaling approach, in which we first analyze the motion of the compact object, the sub-structures ’ movement being retrieved in a second stage. We validate and demonstrate the efficiency of our approach on synthetic and real images for various applications. Index Terms — Image motion analysis, stochastic fields 1

C. Cassisa

Ff Tang

V. Prinet

Crossref

MRF Modeling for Optical Flow Computation from Multi-Structure Objects

We propose in this paper a new formulation of the equation of the optical flow enabling to compute global and local motions ofmulti-structure objects (flowers and petals, trees and leaves, ...). The displacement fields are computed using a Markovian Random Field (MRF) model. Local and global components of the vector flow are both explicitly retrieved. The minimization of the Gibbs energy is achieved with a down-scaling approach, in which we first analyze the motion of the compact object, the sub-structures' movement being retrieved in a second stage. We validate and demonstrate the efficiency of our approach on synthetic and real images for various applications

Prinet, Veronique

Cassisa, C.

Tang, F.F.

HAL-CIRAD

MRF Modeling for Optical Flow Computation FromMultistructure ObjectsVeronique Prinet, C. Cassisa, F.F. TangTo cite this version:Veronique Prinet, C. Cassisa, F.F. Tang. MRF Modeling for Optical Flow Computation FromMultistructure Objects. IEEE International Conference on Image Processing (ICIP), Oct 2006,Atlanta/USA, IEEE, pp.1093-1096, 2006, International Conference on Image Processing (ICIP2006),Atlanta, GA, Oct 08-11, 2006, IEEE. <inria-00124141>HAL Id: inria-00124141https://hal.inria.fr/inria-00124141Submitted on 12 Jan 2007HAL is a multi-disciplinary open accessarchive for the deposit and dissemination of sci-entific research documents, whether they are pub-lished or not. The documents may come fromteaching and research institutions in France orabroad, or from public or private research centers.L’archive ouverte pluridisciplinaire HAL, estdestine´e au de´poˆt et a` la diffusion de documentsscientifiques de niveau recherche, publie´s ou non,e´manant des e´tablissements d’enseignement et derecherche franc¸ais ou e´trangers, des laboratoirespublics ou prive´s.MRF MODELING FOR OPTICAL FLOW COMPUTATIONFROM MULTI-STRUCTURE OBJECTSV. Prinet C. Cassisa FF TangNational Lab. of Pattern Recognition (NLPR) /Sino-French Lab. in Computer Science, Automation and Applied Math. (LIAMA)Institute of Automation, Chinese Academy of Sciences (CASIA).{fftang,prinet}@nlpr.ia.ac.cn cassisa@liama.ia.ac.cnABSTRACTWe propose in this paper a new formulation of the equation ofthe optical flow enabling to compute global and local motionsof multi-structure objects (flowers and petals, trees and leaves,...). The displacement fields are computed using a MarkovianRandom Field (MRF) model. Local and global componentsof the vector flow are both explicitly retrieved. The mini-mization of the Gibbs energy is achieved with a down-scalingapproach, in which we first analyze the motion of the com-pact object, the sub-structures’ movement being retrieved ina second stage. We validate and demonstrate the efficiency ofour approach on synthetic and real images for various appli-cations.Index Terms— Image motion analysis, stochastic fields1. INTRODUCTIONThe objective of this paper is to propose an efficient formula-tion and computational framework to estimate local and globalmotions of non-compact non-rigid objects. Modeling rigidand non-rigid objects movement using optical flow (OF) hasbeen widely studied for various applications including track-ing, surveillance [1, 2]. Analysis of deformation of non-compactmaterial such as water or smoke has also recently raised muchattention [3]. However, objects which are composed of sub-structures (leaves of trees, feathers of birds, petals of flowers,etc) do not fit into these categories. Existing approaches areunable to tackle efficiently the problem because the deforma-tion of this type of object is both global (at the level of the en-tire object) and local (at the level of the sub-structure, usuallyof a few pixels size). Most recent works aim at tackling theproblem of discontinuities [4], of noise [5], to develop hierar-chical or multi-resolution computational approaches [6, 7].We propose to generalize the constant intensity constraintof the optical flow, and to retrieve the optimal solution viaa Markov Random Field (MRF) framework. The problemwe address focuses on the multi-motion analysis of objectscomposed of sub-structures, for which both local and globalcomponents of the displacement vectors are modeled.The rest of the paper is organized as follows: in section 2recalling briefly the basic mathematical foundations of OFand MRF, we define the generalized constant intensity con-straint (GCIC) and describe the energy function to be mi-nimised. We propose a down-scaling approach, thus allowingto process the image at two different scales and details levels.Illustrations and analysis of the results on various syntheticand real data set are given in section 3. Finally, section 4presents the conclusions.2. THE APPROACHWhile extremely simple, the two key points of the approachare (i) the formulation of a generalized constant intensity con-straint (GCIC) and (ii) the down-scaling optimization schemein a MRF framework.2.1. General concept and definitionsWe tackle the problem of one or several object(s) that arecomposed of multi-structures (trees, birds, flowers,..). Eachobject is animated by a global movement. Each object is it-self composed of sub-objects or structures (such as leaves,feathers, petals, ...) which have a local motion independentfrom the global displacement of the object.In order to differentiate the notion of global object andlocal structure, we define two graphs associated to differentlevels of resolution G :Gl = (Sl, Ll) Gg = (Sg, Lg)where Sl = {sl} represents the local nodes (indice l) de-fined either by the substructures of the objects or simply bythe pixels of the image –i.e. card(Sl) = dim(I)–, whileSg = {sg} is represents global nodes (indice g) defined by thecenter of each object in motion in the image –i.e. card(Sg) =number of objects. Ll and Lg are two distinct levels of ver-tices, linking independently the neighborhood nodes sl andsg respectively. Note that this definition imposes Sl ⊃ Sg .2.2. Modeling the GCICConsidering objects animated of a double movement –localand global ones–, the velocity vector v(i) at each point of theimage i is expressed by:v(i) = vg(i) + vl(i) vl(i) << vg(i) (1)where vg(i) is the global displacement of the object O at pointi and vl(i) is the local displacement of the substructure com-posing O at point i. Since all the substructures belonging to agiven object O should possess the same global movement wehave:vg(i) = constant = vg ∀i ∈ O (2)If the substructures are of one pixel size and have randomindependent movements, then local variations can be assimi-lated to white noise of zero mean and:∑i∈Ov(i) = N vg (3)where N is the number of points within the object.From equation (1), the first order generalized optical flowequation (so-called Generalised Constant Intensity Constraint)can be written:∇I. (vg + vl) +δIδt= 0 vl << vg (4)By averaging equation (4) over all the points of the objectO and knowing from equation (2) and (3) that vg + vl =vg + vl = vg = vg, thus, for a given object and as a firstapproximation, we have:(∇I)T .vg + It ≈ 0 (5)with a = 1N∑Ni=1 ai.2.3. Energy terms of the MRFThe displacement field in the image is given by the v that sa-tisfy equation (4). Knowing that the solution is undefined atpoints where v is orthogonal to∇I -so called aperture effect-,some additional constraints –or regularization factor– shouldbe added.Using an MRF framework, the solution v will be given bythe configuration of our random variable x, that maximizes ajoint probability function taking the form of a Gibbs distribu-tion:P (x,y) =1Ze−E(x,y) (6)where Z is the normalization constant andE(x,y) = −(∑s∈C1αdVd(ys,xs)+∑s,s′∈C2αpVp(xs,xs′)))(7)where y is a function of the observed data. From a generalpoint of view, C1 and C2 are single-site and pair-site cliquesrespectively; the Gibbs energy E is thus the summation of adata term Ed =∑αdVd, and a prior term Ep =∑αpVp in-dependent of the data but dependent of its neighborhood Ns;αd = αd(s) and αp = αp(s, s′) are two weighting coeffi-cients.For our problem, the data term is naturally given from theGCIC (equation (4)). It should be satisfied at each site of thegraph (Sl, Ll):|Ed| =∑s∈C1αd(∇I(s). (vg(s) + vl(s)) +δI(s)δt)2(8)with C1 = Sl.On the other hand, the prior term is defined such as toconstrain the displacement flow to be continuous: amplitudeand angular variations between neighborhood sites should beminimized, both at global and local levels. We have therefore:|Ep| =∑s,s′∈C2βp (||vl(s)− vl(s′)||)2 (9)+∑s,s′∈C3γp (||vg(s)− vg(s′)||)2+∑s,s′∈C3λp(vg(s).vg(s′)||vg(s)||||vg(s′)||− 1)2where C2 = {sl, s′l|sl, s′l ∈ Sl, s′l ∈ Nl} and C3 = {sg, s′g|sg, s′g ∈ Sg, s′g ∈ Ng}. The neighborhoods Nl and Ng aredefined within a 4-connectivity by the connected sites belong-ing to Gl and Gg respectively. The first two terms of equa-tion (9) impose a smooth variation of the amplitude at localscale (first term) and global scale (second term). The lastterm constrains the motion’s direction to change slowly be-tween neighboring objects. The site dependent coefficientsαd, βp, γp, λp are set such that : αd 6= 0 if s ∈ O, γp, λp 6= 0everywhere, andβp = 0 if s ∈ O, s′ ∈ O′ (10)βp 6= 0 otherwise.In other words, the smoothing between local vectorial com-ponents is to be applied only for intra-object and not inter-object neighborhood.It is worth noticing that, in the case of several objects inthe image, the model is properly defined only if the globalmovement between two neighborhood objects varies slowly.However, we will see that in practice we do not encounter thislimitation, because the local term will serve to compensate thepossible loss generated by the global smoothing.2.4. ImplementationThe optimal solution is retrieved by computing the v that mi-nimizes E(x, y). It is achieved without major difficulty witha ICM (iterated conditional mode) algorithm. However, sincethe random value x is a four dimensional vector which hascomponents that can take any real value within a given range[vminl , vmaxl , vming , vmaxg ], the direct optimization of equation(7) using (8) would be extremely time consuming. In addi-tion, the initialization of ICM algorithm would be twice ascritical as traditional approaches (i.e. approaches that do notsearch for a local and a global component).For these reasons, we choose to compute the displacementfield via a down-scaling approach, following two steps :• Initialization : compute the optimal solution of equa-tion (7), using (5) and (9) as data term and prior term re-spectively. This enables to find a realisation vˆ0g , whichwill serve as a first approximation. The process is ex-tremely fast and enables to give a rough picture of thedisplacement flow.• Estimation of vˆg, vˆl, by substituting equation (8) into(7) and using vˆ0g as initialization.Note that the approach is based on the knowledge of the graph(Sg, Lg), which means that, theoretically, one should be ableto localize the objects within the image, before running theoptical flow. When feasible, a pre-processing segmentationstep can be carried out for this purpose. In most cases how-ever, we can perform arbitary segmentation using a regulargrid, without affecting the quality of the results. We will il-lustrate several cases in the next section.3. RESULTS ANALYSISResults analysis is performed on a set of very different syn-thetic and real images, thus illustrating various possible ap-plications. All results presented in this section have beenobtained with the values (αd, βp, γp, λp) = (1, 10, 50, 100)for forest sequence and (1, 10, 10, 10) for the others. Videosequences have been previously smoothed in space and timewith Gaussian filter of variance 1.5; the synthetic forest ima-ges, have been spatially smoothed only. The derivation maskwe used is similar to [5, 8].3.1. Synthetic data : forestThe initial motivation of this work was to analise the Earthground displacement observed from forest trees aerial ima-ges. This phenomena can be observed on terrains of steepslopes and are characteristic of forthcoming landslides. Herewe make use of synthetic images simulating a plantation oftrees on a steep terrain, generated by the AMAP software[9]. Two sequences of two images each have been gene-rated.The first sequence shows a unidirectional displacement alongthe slope in the vertical direction, the second is a radial dis-placement. The position of the camera is unchanged betweenthe two “acquisitions”. One of the simulated image and theknown displacements are illustrated on figure 1.The objects we consider are the trees, the local structuresbeing defined by the leaves. The images are particularly diffi-cult to process because the intensity constancy is hardly veri-fied at pixel level.Figure 2 illustrates the results obtained from our approach(top), compared to those from Horn and Shunk (H&S) imple-mented by Barron [8, 10] (bottom). The computed displace-ment field, for transverse and radial motions, are both con-sistant with the ground truth. A contrario, H&S algorithmappears to be more sensitive to the texture effect and non uni-form brightness variation of the leaves. Table 1 compares theaverage angle error between H &S and our approach.In these images in which segmentation is rather easy toperform, we have analysed the impact of the pre-propressingstep on the results. In figure 3, we compare the vector flow ob-tained with and without presegmentation. In case of preseg-mentation, each tree is treated as an individual object. Other-wise, the image is organised into a matrix of squared paches,each patch being associate to an object. Table 1 shows thatthe effect of the square size is negligeable, as expected.(a) Entire image(b) Detail of (a).(c) Transverse displacement (d) Radial displacement.Fig. 1. Simulated forest trees and ground truth motion.Transverse motion Radial motionFig. 2. Results of GCIC (top) - Horn & Shunk (bottom).vg & segm. (vg + vl) & segm. vg & grid 40x40Fig. 3. Local vs global motion, with and without segmentationDisplacement Tranversal RadialH & S 32.1 32.0GCIC & seg 14.4 16.6GCIC & grid 30x30 15.1 17.5GCIC & grid 40x40 15.0 17.8GCIC & grid 50x50 15.7 17.7Table 1. Average angle error in degrees with presegmenta-tion (seg) and with arbitrary patchs (grid size NxN).3.2. Reference data set : synthetic and real dataResults on synthetic and real video sequences –data are down-loaded from ftp.csd.uwo.ca/pub/vision– have been computed.For the synthetic sequences, we obtained the same range ofaverage angle error than shown by [5].We illustrate here onlyone of them, i.e. the Magic Cube, given in figure 4. Thecomputed displacement (v = vg + vl) shows a clear smoothmotion tracking of the cube and of the bottom part of the tray.The motion of the white part however has not been detectedat all, due to the perfect uniformity of the gray level (constantwhite). Barron’s result with Horn and Shunk approach is alsoillustrated for comparison.(a) Original image (b) GCIC (vg + vl) (c) H & SFig. 4. Results on the Magic Cube sequence3.3. Flower videoThe last illustration is on a flower video sequence (down-loaded data). The sequence visualizes the blooming of theflower. Each petal can be considered as an individual object.The global displacement is first computed in each patch gridof the image. Edge points detection is performed by a Cannyfilter for each frame of the sequence. The displayed resultin figure 5 represents the vg component of the displacementtaken at each edge point. Result on the entire video providesclear cue of the flower blooming.4. CONCLUSIONWe proposed in this work an approach dedicated at first to themovement analysis of objects composed by sub-structures,Fig. 5. Two flower frames and the associated computed vgi.e. animated by global and local motions. We illustratedthat the introduction of a local displacement vector enables tohandle efficiently the problem of discontinuity. With regard toforestry application, ongoing works are dedicated to texturalmovement analysis.Acknowledgment The authors wish to thank Thierry Fourcaudfor stimulating discussions and Guillaume Perrin, for the creation ofthe forest images.5. REFERENCES[1] F. Ranchin and F. Dibos, “Moving objects segmentation usingoptical flow estimation,” Conf. on Mathematics and Image Anal-ysis (MIA), 2004.[2] M. Ye and R. Haralick, “Two stages robust optical flow estima-tion,” Conf. on Computer Vision and Pattern Recognition, vol. 2,pp. 623–628, 2000.[3] E. Menin and P. Perez, “Hierarchical estimation and segmenta-tion of dense motion field,” Int. J. Comput. Vision, vol. 42, pp.129–155, 2002.[4] F. Heits and P. Bouthemy, “Multimodal estimation of discon-tinous optical flow using markov random fields,” IEEE Trans.Pattern Anal. and Mach. Intell., vol. 15, no. 12, pp. 1217–1232,1993.[5] S. Roy and V. Govindu, “Mrf solution for probabilistic opticalflow formulation,” Int. Conf. on Comp. Vision, 2000.[6] E. Adachi and S. Horiuchi, “Multi-resolution optical flow esti-mation with local approximation,” Int. Conf. on Image Process-ing, 2002.[7] J.N. Provost, C. Collet, P. Rostaing, P. Perez, P. Bouthemy, “Hi-erarchical markovian segmentation of multispectral images forthe reconstruction of water depth maps,” Int. J. on ComputerVision and Vision Understanding, 2004.[8] J.L. Barron, D.J.Fleet, S.S. Beauchemin, “System and exper-iment performance of optical flow techniques,” Int. J. Comput.Vision, vol. 12, pp. 43–77, 1994.[9] P. De Reffye,M. Jaeger,C. Edelin,J. Franon,C. Puech, “Plantmodels faithful to botanical structure and development,” Int.Conf. Computer Graphics, vol. 22, pp. 151–158, 1988, Siggraph.[10] B.Horn, B Shunck, “Determining optical flow,” Artif. Intell,vol. 17, pp. 185–203, 1981.

MRF Modeling for Optical Flow Computation From Multistructure Objects

HAL Descartes

https://hal.inria.fr/inria-00124141/document

MRF MODELING FOR OPTICAL FLOW COMPUTATION FROM MULTI-STRUCTURE OBJECTS

Abstract

Similar works

Full text

Available Versions

Crossref

HAL-CIRAD

HAL Descartes