This paper proposes and discusses the use of motion-oriented connected operators for sprite creation. Motion-oriented connected operators are tools allowing the simplification of frames by removing objects that do not follow a given motion. They combine features of filtering and segmentation tools. They are, however, less computationally expensive than most motion-oriented segmentation algorithms. In this paper, we show how they can be used to efficiently remove outliers with respect to the dominant motion and to create layered representation of sequences.Peer ReviewedPostprint (published version

Garrido Ostermann, Luis

Pujol, O

Salembier Clairon, Philippe Jean

English

This paper proposes and discusses the use of motion-oriented connected operators for sprite creation. Motion-oriented connected operators are tools allowing the simplification of frames by removing objects that do not follow a given motion. They combine features of filtering and segmentation tools. They are, however, less computationally expensive than most motion-oriented segmentation algorithms. In this paper, we show how they can be used to efficiently remove outliers with respect to the dominant motion and to create layered representation of sequences.Peer Reviewe

UPCommons

Connected operators for sprite creation and layered representation of image sequences

UPCommons. Portal del coneixement obert de la UPC

CONNECTED OPERATORS FOR SPRITE CREATIONAND LAYERED REPRESENTATION OF IMAGESEQUENCESP Salembier O Pujol and L GarridoDept of Signal Theory and CommunicationsUniversitat Politecnica de Catalunya Barcelona SpainTel 	    Fax 	   Email philippegpstscupces URL httpgpstscupcesimatgeABSTRACTThis paper proposes and discusses the use of motionoriented connected operators for sprite creationMotionoriented connected operators are tools allowingthe simplication of frames by removing objects that donot follow a given motion They combine features of ltering and segmentation tools They are however lesscomputationally expensive than most motionorientedsegmentation algorithms In this paper we show howthey can be used to eciently remove outliers with respect to the dominant motion and to create layered representation of sequencesKeywords Sprite representation Mosaicing Connected operators Robust Motion estimation INTRODUCTIONSprite or mosaic representations of video sequences isbecoming popular for coding applications  	  Inparticular the MPEG standard contemplates a prediction tool which is based on sprite representations Most of the time these sprites are used to representthe background and concentrate the information thathas been or will be visible in the sequence They areused to improve the prediction of the background Thesprite itself can be used in dierent modes it can becomputed oline and sent to the receiver at the beginning of the transmission or it can be progressively estimated on both the transmitted and the receiver sides ina causal fashion from the transmitted frames Of coursethe use of sprites is not limited to the background information and the approach can be used to deal withany components of the scene that has an homogeneousmotion This approach leads to layered representationsof sequences  Furthermore sprite representation ofobjects is also investigated for contentbased retrievalapplications  MPEG for example In this framework sprite representation can be an attractive solutionfor representing a large number of frames in a compactand synthetic visual object that can be used for browsing and navigation purposesMost of the time the sprite creation is achieved either by very simple techniques or by very complex onesThe simplest approach consists in rst computing thedominant motion second compensating or wrapping allframes towards a single reference and third averagingthe gray level or color values that are compensated atthe same position If foreground objects do not occludethe background in a large number of frames this approach gives satisfactory results In order to improvethe robustness of the scheme the averaging operationmay be replaced by a temporal median lter On theother side of the complexity scale motion segmentationalgorithms can be used to actually segment the videosequence into a set of regions that are homogeneous inmotion The knowledge of the shape information of thevisible part of the background in each frame allows usto improve the sprite creation by removing the outliersThe goal of this paper is to propose and study theuse of motionoriented connected operators  to create sprite representation Connected operators are usedto remove the image components that do not follow themotion of the sprite under consideration without requiring an actual motion segmentation The use of this approach leads to a low complexity scheme compared toreal motion segmentation algorithms The organizationof this paper is as follows section  reviews the basisand main features of connected operators The application to sprite creation is presented in section  Finallysection  is devoted to the conclusions CONNECTED OPERATORS Image processing with connected operatorsConnected operators    are ltering techniquesderived from mathematical morphology that eliminatepart of the image content while preserving the contourinformation of the remaining parts of the image Theyinteract with the signal by means of  at zones  A connected operator is an operator that can only merge  atzones of the image As a result it can simplify an imagebut without introducing any new contour Flat zones are the connected components where the image isconstant Note that a at zone can be reduced to a single pointOriginal image Filtered imageTree representationMax-tree constructionImage constructionTree pruningFiltered treeFigure  Connected operator ltering strategy in threesteps  Maxtree creation  pruning of the tree ltered image reconstructionAn e	cient way of creating and implementing connected operators relies on MaxTree representations The ltering strategy is illustrated by Fig  The image is considered as a D relief and the rst step is toconstruct a MaxTree representation of the image Thenodes of the tree represent the binary connected components resulting from the thresholding of the originalimage at all possible gray level values The links betweenthe nodes describe how the at zones may be mergedNote that there exist fast algorithms to construct thetree see  Then starting from the leaves of the treeeach node is studied and a particular criterion is assessedfor each node If the criterion value is above below agiven threshold the node is preserved removed If thecriterion is increasing that is if the criterion value ofa node is always smaller or equal to the criterion valueof its father then the algorithm denes a tree pruning strategy and at the end of the pruning the lteredimage is reconstructed by stacking the connected components corresponding to the remaining nodes If thecriterion is not increasing the denition of the pruningstrategy is less straightforward As discussed in  thenonincreasingness of a criterion is most of the time adrawback that implies a lack of robustness of the operator similar images may give dierent results In a solution relying on dynamic programming techniqueViterbi algorithm was proposed In the sequel sincethe motion criterion is not increasing this solution isassumed to be used and the interested reader is referredto  for more details on this issueThe operator is said to be antiextensive because theltered image is for each pixel smaller than the originalimage In practice this means that the operator simplies the image by removing its bright components thatab cFigure  Example of simplication with motionoriented connected operator The operator eliminatesthe image components that do not follow the global motion a Original frame b Elimination of bright components with motionoriented connected operators celimination of dark components with the dual operatordo not fulll a given criterion To simplify dark components the dual operator should be used If  Ipis a connected operator applied on image Ip its dualoperator is   Ip     Ip   Motionoriented connected operatorsRecently a motionoriented connected operator has beendened   This operator allows the elimination of theimage components that do not undergo a given motionThe ltering parameter is the motion eld itself Letus denote bydp the motion eld corresponding to animage Itp at time t The basic ltering strategy is tomeasure for each node Zfof the Maxtree of image Itthe opposite of the energy of the mean displaced framedierence DFDCIt  ItZf   XpZfItp  Itp dp N where N is the number of pixels of Zf If the criterionvalue is high the connected component correspondingto Zfactually follows the motiond and should be preserved However if the criterion value is low the connected component can be considered as an outlier andshould be removed As explained in   the criterion robustness can be improved if a memory term is the opposite of the DFD is used in order to preserve nodesthat correspond to high values of the criterionGlobal motionestimationConnectedDual connect.operatoroperatorWarpingestimationGlobal motionOutliersmaskFigure  Diagram ow for sprite creationintroducedCZf  CIt  ItZf  	 C It  ItZf This equation states that the criterion is a weightedcombination of 	 the DFD between the original framesat time t and t	 and  the DFD between the originalframe at time t and the ltered frame at time t 	Let us illustrate the eect of the operator Fig ashows an original frame Assume that the dominantmotion has been estimated This motion is forwardedto the operator in order to eliminate the image components that do not follow this motion Fig b shows theoutput of the operator As can be seen bright objectsthat do not follow the background motion have been removed However the remaining components have beenwell preserved Fig c shows the output of the dual operator applied on the original frame Here dark objectsnot following the global motion have been removed  SPRITE CREATIONThe proposed scheme for sprite creation is illustrated inFig  Let us assume that we deal with the backgroundhowever the approach can be generalized to any groupof objects that undergo a similar motion The rst stepis to estimate the dominant motion This can viewedas the estimation of the dense motion elddp thatminimizes the DFD DR Xp RItp Itpdpwhere R is the region on which the estimation is performedFor the estimation of the dominant motion R is theentire frame In our experiment a ane model of thedense motion eld has been assumed This dense motion eld is forwarded to a motionoriented connectedoperator that removes all bright components that donot follow the dominant motion In parallel the dualoperator is applied to remove dark components Examples of ltered frames is shown in Fig a and bthe original frame is the one of Fig a As can beseen the background objects have been preserved anda bc dFigure  Example of sprite creation Processing of therst frame original frame can be seen in Fig a aoutput of the connected operator b output of the dualconnected operator c outliers mask white and blackpixels are outliers respectively removed by the connectedoperator and its dual d rst image warped in the spritethe two boats have been removed By simple comparison between the ltered and the original frames theoutliers mask can be extracted In Fig c the pixels inwhite dark correspond to locations where the outputvalues of the connected operators dual connected operators signicantly dier from the original image Thisoutliers mask identies the pixel that have to be warpedtowards the sprite see Fig dBefore actually warping the frame the motion can bereestimated Indeed the presence of outliers may havecorrupted the initial estimation Once detected theseoutliers can be removed from the frame and the motion can be reestimated on a region of support R thatcontains no outliers As usual the entire sprite is progressively constructed by averaging the pixels that arecompensated at the same position Fig  shows the progressive construction of the sprite Finally the interestof outliers removal is illustrated in Fig  In this casethe sprite has been created without removing the outliers As can be seen the presence of these outliers hasa strong negative inuence in the nal result CONCLUSIONSIn this paper the application of motionoriented connected operators to the problem of sprite generation hasbeen proposed Connected operators are ecient toolsto dene and to extract outliers with respect to a givenmotion eld They are particularly ecient is terms ofFigure  Progressive construction of the sprite Top frames Center  frames Bottom  framesFigure 	 Sprite without outliers elimination framescomputational complexity and avoid the use of a complex motion segmentation algorithm Furthermore theyare independent from the motion estimation and fromthe warping stages As a result they can be introducedjust before the warping in any sprite creation schemeReferences J Crespo Morphological Connected Filters andIntraregion Smoothing for Image SegmentationPhD thesis Georgia Institute of Technology  J L Dugelay and H Sanson Dierential methodsfor the identication of D and D motion models in image sequences Signal Processing ImageCommunication   L Garrido A Oliveras and P Salembier Motionanalysis of image sequences using connected operators In SPIE Visual Communications and ImageProcessing VCIP volume  pages 	San Jose CA USA February  MPEGN	 ISOIEC JTCSCWGCore experiment on sprites and global motion compensation April  K Jinzenji S Ishibashi and H Kotera Algorithmfor automatically producing layered sprites by detecting camera movement In IEEE InternationalConference on Image Processing ICIP pagesI	 Santa Barbara USA October 	 MC Lee W Chen CB Lin C Gu T MarkocSI Zabinsky and R Szeliski A layered video object coding system using sprite and ane motionmodel IEEE Transactions on circuits and systemsfor video technology  February  P Salembier A Oliveras and L Garrido Antiextensive connected operators for image and sequence processing IEEE Transactions on ImageProcessing  April  P Salembier and J Serra Flat zones ltering connected operators and lters by reconstruction IEEE Transactions on Image Processing	 August  JYA Wang and EH Adelson Representing moving images with layers IEEE Transactions on Image Processing 		 September  H Zhang JYA Wang and Y AltunbasakContentbased video retrieval and compression Aunied solution In IEEE International Conferenceon Image Processing ICIP pages I	 SantaBarbara USA October 

https://upcommons.upc.edu/bitstream/2117/103622/1/07089633.pdf

Connected operators for sprite creation and layered representation of image sequences

Abstract

Similar works

Full text

Available Versions

UPCommons

UPCommons. Portal del coneixement obert de la UPC