Centre for Multimedia Signal Processing, Department of Electronic and Information Engineering2001-2002 > Academic research: refereed > Publication in refereed journalVersion of RecordPublishe

Chan, YH

Siu, M

Siu, WC

IEEE Transactions on Circuits and Systems for Video Technology

PolyU Institutional Repository

1188 IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, VOL. 11, NO. 11, NOVEMBER 2001A Robust Model Generation Technique for Model-Based Video CodingManson Siu, Yuk-Hee Chan, and Wan-Chi SiuAbstract—In conventional model-based coding schemes, prede-fined static models are generally used. These models cannot adaptto new situations, and hence, they have to be very specific andcannot be generated from a single generic model even though theyare very similar. In this letter, we present a model-generation tech-nique that can gradually build a model and dynamically modify itaccording to new video frames scanned. The proposed techniqueis robust to the object’s orientation in the view and can be effi-ciently implemented with a parallel processing technique. As a re-sult, the proposed technique is more attractive to the practical useof model-based coding techniques in real applications.Index Terms—Model-based coding, model synthesis.I. INTRODUCTIONTHE MODEL-BASED coding system is the hottest memberin the object-based coding system. A vast amount of re-search has been carried out on relevant topics such as feature ex-traction [1], motion tracking [2], and model-synthesizing tech-niques [3], [4], but model generation is still a time-consumingprocedure that is not attractive for real-time applications.In conventional model-based coding schemes, pre-definedmodels are generally used. These models are static and canneither be adaptive to fit the real features of the object of in-terest nor be updated dynamically in the course of application.Accordingly, a very specific model is necessary for a particularobject, and one cannot use a generic model to generate modelsfor similar objects whenever it is necessary so as to savememory space. Therefore, for applications involving real-timevideo coding, a simple model-generation method that caneasily construct a model or modify it dynamically according tothe scenes processed is necessary.There are methods for generating an object model with stereographics [5] and laser scanning. Laser scanning can provide avery accurate model for an object, but the size and the cost of theequipment involved makes it infeasible for the aforementionedapplications. As for methods based on stereo graphics, they gen-erally work only on specific views. Because of these reasons,practical model-synthesis technique becomes an important in-vestigation in our model-based coding research.In this letter, we propose a new technique to generate amodel for a particular object with a generic model. The pro-posed approach has several advantages over the conventionalmethods using stereo graphics. First, no specific view withManuscript received July 21, 1999; revised September 30, 2000. Thiswork was supported by The Hong Kong Polytechnic University under GrantA/C V592 and Center of Multimedia Signal Processing. This paper wasrecommended by Associate Editor H. J. Zhang.The authors are with the Center of Multimedia Signal Processing, Depart-ment of Electronic and Information Engineering, The Hong Kong PolytechnicUniversity, Hong Kong (e-mail: mansonsiu@hotmail.com; enyhchan@hotmail.com; enwcsiu@polyu.edu.hk).Publisher Item Identifier S 1051-8215(01)10157-6.known orientation of the object is required. Second, updatingthe generic model with information extracted from multipleviews is supported. Third, the feature-extraction processesfor different views can be done independently, and henceparallel processing is allowed. Accordingly, our method iscomparatively more efficient and practical for real applications.II. ROBUST MODEL GENERATIONIn the proposed approach, a generic model for objects ofsimilar shapes is used in the model-based encoding and de-coding processes. The usage of the generic model can achievevery low bit-rate coding, which benefits communication appli-cations. The idea of using the generic model is to use a stan-dard collection of graphics models to represent standard objects.However, a single object may have various shapes that a stan-dard model may not represent well. Our proposed algorithm cansolve this problem easily as it can dynamically adapt a genericmodel to the shape of the object of interest whenever new infor-mation is available.Note that the algorithm presented in this letter can actuallyoperate in two modes for different applications. In its firstoperation mode, a target model can be generated as a pre-de-fined model for later applications based on different views ofthe object of interest. In its second operation mode, the targetmodel can be gradually built or updated with new informationextracted from new video frames being processed. Parallelprocessing is allowed in both circumstances.Fig. 1 shows the process flow of the proposed model-gen-eration scheme. In the following part of this section, we shalldiscuss the processes involved in details.A. Orientation ExtractionIn order to extract matching information for generating theobject’s model with the generic model on hand, the orientationof the object appeared in the captured image or video framesmust be known. This piece of information can be obtained via ananalysis of the image context. Various research for this purposehave been done. Typical methods include those use face featuresto estimate a human head’s orientation such as feature-locationtracking [1] and point-matching [2]. In our approach, we adoptthe point-matching technique [2] to determine the object’s ori-entation in a captured image.B. Visible Surface DeterminationAfter we have identified the orientation of the object, we ro-tate the generic model accordingly to match their orientations.The visible surface of the model is then determined with a mod-ified Z-buffering algorithm. In order to increase the efficiencyand reduce the computation effort, we project the vertices onto1051–8215/01$10.00 © 2001 IEEEAuthorized licensed use limited to: Hong Kong Polytechnic University. Downloaded on July 27, 2009 at 22:46 from IEEE Xplore.  Restrictions apply. IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, VOL. 11, NO. 11, NOVEMBER 2001 1189Fig. 1. Flow of the model synthesis process.(a)(b)Fig. 2. Visible surface determination. (a) Vertex projection. (b) Visiblesurfaces.a projection plane that passes the center of the model and is par-allel to the view plane. Fig. 2 shows the visible face of a headmodel and how it is obtained. The left portion of Fig. 2(b) showsthe surface closer while the right one shows the surface fartherthan the projection plane to the viewpoint.C. Matching a Generic Model to an Object’s ImageThe processed model will then undergo a model-matchingprocedure. First, a projected view of the processed model is ob-tained by projecting the model onto the view plane with ortho-graphic projection. Then, the boundary information of this pro-jected view and the object’s image is extracted. After translatingthe generic model to make the center of its projection coincidentwith the center of the object’s image, the model is adjusted suchthat the boundary of the projected view of the model matchesthat of the object’s image.Besides the boundary vertices, remarkable feature pointslike eye and mouth corners are also modified in this process.The corresponding feature points on the generic model willbe matched to that of the object’s image by making use offeature matching techniques. The feature-matching techniqueproposed by Tang and Thomas in [3] is adopted in our systemto match face features. Fig. 3 shows the intermediate result ofthe process.D. Modifying Generic ModelModifying the position of a vertex in an object model will re-sult in a change of the positions of its connected vertices. Theirchange is not uniform. The closer a vertex from the adjustedvertex, the more significant the change should be. The modifi-cation algorithm we adopted here is an iterative algorithm.Fig. 4 shows a particular vertex of the model and its con-nected neighbors. In general, the resultant displacement dueto its neighboring vertices’ displacements, say , is given as(1)where is the radial displacement of vertex , and isthe set of ’s connected neighboring vertices. The weightingparameters and are, respectively, defined as(2)and(3)where is the total number of elements in and is thedistance between vertices and .The vertices are updated one by one until all of them are pro-cessed. These procedures are then repeated again and again untilthere is no more change of the resultant displacement for everyvertex in the model.E. Updating Resultant ModelFor a particular view of the object, a new model can be ob-tained by adjusting the generic model with the aforementionedprocedures. For the sake of reference, we refer to the model as-sociated with view orientation as . This intermediate modelwill then be used to update the target model in order to pro-duce a natural model.Authorized licensed use limited to: Hong Kong Polytechnic University. Downloaded on July 27, 2009 at 22:46 from IEEE Xplore.  Restrictions apply. 1190 IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, VOL. 11, NO. 11, NOVEMBER 2001Fig. 3. Model matching: (a) projected view of a model; (b) corresponding positions of the vertices appeared at the boundary of the model’s projected view; and(c) adjusted model.Fig. 4. Corresponding resultant displacement and its connected neighbors.Let the vertices of model be : , whereis the total number of vertices of the model. The line con-necting the origin to a particular vertex defines a vector( , , ), as shown in Fig. 5. The componentis used as an input parameter to calculate , the confi-dence level of with respect to its counterparts ’s, where. Specifically, is given byif is visible in viewelse. (4)Hereafter, this equation is referred to as significance equation,as it tells the significance of a particular vertex of in up-dating .The coordinates of the corresponding vertex in , , arethen updated with the following rule accordingly:ifelse. (5)Threshold is the maximum value of ’s for all processedviews ’s. It is initialized to be zero at the very beginning.Since the computation of ’s is independent of each other, themodel can be updated in parallel. This increases its efficiency.Fig. 5. Vertex j of the adjusted model associated with view orientation i.Fig. 6. Coordinate system and generic model used in the simulation.In our approach, the target model is initialized to be thegeneric model. The updating rule (5) guarantees that the targetmodel converges and eventually converges into an ultimatestate as long as sufficient views are provided if the object isrigid. Since is always updated to be , the confidencelevel of which is maximum, it is obvious from (4) that, in theultimate target model, all vertices are of confidence level 1.The number of iterations required to obtain the ultimate targetmodel is input dependent. If a well-organized input sequence ofAuthorized licensed use limited to: Hong Kong Polytechnic University. Downloaded on July 27, 2009 at 22:46 from IEEE Xplore.  Restrictions apply. IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, VOL. 11, NO. 11, NOVEMBER 2001 1191(a) (b) (c)Fig. 7. Three test input images in different orientations.views are provided, the ultimate target model can be obtainedwith a few iterations. However, if a particular view of the ob-ject never appears, it is possible that the confidence levels ofsome vertices of are less than 1 and the ultimate target modelcannot be obtained forever.In practical applications, such as model-based coding, it isgenerally not necessary to obtain the ultimate target model.Once a particular view of the concerned object appears, itcan be incorporated into the current target model. The codingprocess can be based on the most updated target model insteadof the ultimate target model, as it already contains the bestknowledge we have had. Another argument is that, if someviews of the object of interest never appear, we need not botherwith them and it is sufficient to work with a target model whichis not ultimate in a way that the coordinates of the hiddenvertices are not properly defined.When the object of interest is not rigid, it is impossible toget an ultimate target model, as some vertices of its model maynot be stable. Using a head model to encode head-shoulder se-quences is one of the typical examples. In such a case, the codingprocess should be based on the most updated target model.III. SIMULATION RESULTSimulation has been carried out to evaluate the performanceof the proposed algorithm. The coordinate system and thegeneric model used in the simulation are shown in Fig. 6. Basedon the generic model on hand, a single spherical texture mapis first generated with three different views shown in Fig. 7 forlater application [6].The generic model is fitted to the object’s image accordingto the object’s orientation in the image. Our proposed method isthen used to adjust the generic model to form three intermediatemodels. In Figs. 8–10, the intermediate models generated fromthe three different views shown in Fig. 7 are shown.The intermediate model generated in each view is then used toupdate the target model. The extraction procedures in obtainingthe update model can be optimized with parallel computing. Theconfidence levels of the vertices of the intermediate models arecalculated and the target model is modified with a reference tothese parameters. Fig. 11 shows five views of the target model.Fig. 8. The intermediate model obtained with view 7(a).Fig. 9. The intermediate model obtained with view 7(b).Fig. 10. The intermediate model obtained with view 7(c).One can see that three appropriate views are enough to adapt ageneric model to an object.Authorized licensed use limited to: Hong Kong Polytechnic University. Downloaded on July 27, 2009 at 22:46 from IEEE Xplore.  Restrictions apply. 1192 IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, VOL. 11, NO. 11, NOVEMBER 2001Fig. 11. Resultant model in the simulation.Fig. 12. Rendered graphics models with a spherical texture map.Once the target model is obtained, one can map the sphericaltexture to the target model to construct different views of the ob-ject. Fig. 12 shows some texture-mapped results obtained withthe target model.IV. CONCLUSIONIn our work on model synthesis, we concern the practicalityof model-based coding techniques. With the help of our model-generation technique, several advantages can be achieved.1) It is robust to the object’s orientation and size in the view.2) The resultant model can be gradually built or updated withmodel update parameters extracted from different viewsunder the guidance of a set of significance functions.3) The extraction of update parameters from different viewscan be carried out in parallel without interference to eachother.Synthesizing a graphical model is always a time-toleratingprocedure in model-based coding systems. With the help of ourapproach, a specific human head model can be obtained moreefficiently as compared with other conventional approaches,which is therefore more attractive to the practical use of model-based coding techniques in real applications.REFERENCES[1] M. Kokuer and A. F. Clark, “Feature and model tracking for model-basedcoding,” in Proc. Int. Conf. Image Processing and Its Applications, 1992,pp. 135–138.[2] K. M. Lam and H. Yan, “An analytical-to-holistic approach for facerecognition based on a single frontal view,” IEEE Trans. Pattern Anal.Machine Intell., vol. 20, pp. 673–686, July 1998.[3] L. A. Tang and T. S. Huang, “Automatic construction of 3D human facemodels based on 2D images,” in Proc. IEEE ICIP’96, vol. 3, Lausanne,Switzerland, 1996, pp. 467–470.[4] L. Yin and A. Basu, “MPEG4 face modeling using fiducial points,”in Proc. IEEE ICIP’97, vol. 1, Santa Barbara, CA, USA, 1997, pp.109–112.[5] L. H. Chen and W. C. Lin, “Visual surface segmentation from stereo,”Image and Vis. Comput., vol. 15, pp. 95–106, 1997.[6] M. Siu and Y. H. Chan, “A robust universal texture extraction tech-nique for model-based coding,” in Proc. SPIE, Vision Geometry VIII,vol. 3811, Denver, CO, 1999, pp. 329–336.Authorized licensed use limited to: Hong Kong Polytechnic University. Downloaded on July 27, 2009 at 22:46 from IEEE Xplore.  Restrictions apply. 

A robust model generation technique for model-based video coding

Abstract—In conventional model-based coding schemes, prede-fined static models are generally used. These models cannot adapt to new situations, and hence, they have to be very specific and cannot be generated from a single generic model even though they are very similar. In this letter, we present a model-generation tech-nique that can gradually build a model and dynamically modify it according to new video frames scanned. The proposed technique is robust to the object’s orientation in the view and can be effi-ciently implemented with a parallel processing technique. As a re-sult, the proposed technique is more attractive to the practical use of model-based coding techniques in real applications. Index Terms—Model-based coding, model synthesis. I

Manson Siu

Yuk-hee Chan

Wan-chi Siu

CiteSeerX

A Robust Model Generation Technique for Model-Based Video Coding

http://www.en.polyu.edu.hk/~wcsiu/paper_store/Journal/2001/2001_J4-IEEE-SiuChan%26Siu.pdf

A robust model generation technique for model-based video coding

Abstract

Similar works

Full text

Available Versions

PolyU Institutional Repository

CiteSeerX