International audienceIn this paper, we present a novel method for in-situ 3D sketching of polyhedral scenes. A video camera is used as both an interaction and a tracking device, which makes the system particularly suitable for handheld devices such as PDAs and mobile phones. The efficiency and accuracy of the method are demonstrated using a miniature scene and a real outdoor scene

Simon, Gilles

In this paper, we present a novel method for in-situ 3D sketching of polyhedral scenes. A video camera is used as both an interaction and a tracking device, which makes the system particularly suitable for handheld devices such as PDAs and mobile phones. The efficiency and accuracy of the method are demonstrated using a miniature scene and a real outdoor scene

Gilles Simon

CiteSeerX

In-Situ 3D Sketching Using a Video Camera as an Interaction and Tracking Device

Hal-Diderot

HAL Descartes

INRIA a CCSD electronic archive server

HAL Id: inria-00474324https://hal.inria.fr/inria-00474324Submitted on 7 Jun 2010HAL is a multi-disciplinary open accessarchive for the deposit and dissemination of sci-entific research documents, whether they are pub-lished or not. The documents may come fromteaching and research institutions in France orabroad, or from public or private research centers.L’archive ouverte pluridisciplinaire HAL, estdestinée au dépôt et à la diffusion de documentsscientifiques de niveau recherche, publiés ou non,émanant des établissements d’enseignement et derecherche français ou étrangers, des laboratoirespublics ou privés.In-Situ 3D Sketching Using a Video Camera as anInteraction and Tracking DeviceGilles SimonTo cite this version:Gilles Simon. In-Situ 3D Sketching Using a Video Camera as an Interaction and Tracking Device.31st Annual Conference of the European Association for Computer Graphics - Eurographics 2010,May 2010, Norrköping, Sweden. pp.53-56. ￿inria-00474324￿EUROGRAPHICS 2010/ M. Cooper and K. Pulli Areas PaperIn-Situ 3D Sketching Using a Video Camera as an Interactionand Tracking DeviceGilles SIMON, Université Henri Poincaré / INRIA project-team MAGRIT, Nancy, FRANCEAbstractIn this paper, we present a novel method for in-situ 3D sketching of polyhedral scenes. A video camera is used asboth an interaction and a tracking device, which makes the system particularly suitable for handheld devices suchas PDAs and mobile phones. The efficiency and accuracy of the method are demonstrated using a miniature sceneand a real outdoor scene.Categories and Subject Descriptors (according to ACM CCS): Artificial Intelligence [I.2.10]: Vision and SceneUnderstanding—Modeling and recovery of physical attributes Computer Graphics [I.3.6]: Methodology andTechniques—Interaction techniques1. IntroductionAcquiring the 3D geometry of arbitrary scenes has been aprimary objective of both the computer vision and graph-ics communities for many decades. Applications are numer-ous in various domains such as construction, geographic in-formation systems, robotics and augmented reality (AR).Existing modeling methods usually rely on two separatestages: first some data about the scene (photographs, videos,physical measurements, laser scanners, ...) are acquired on-site. Then these data are treated off-line using some spe-cific manipulations and algorithms (e.g. [HDT∗07]). Unfor-tunately, this process has two drawbacks: first, some of theinvolved tasks (data acquisition and/or geometry extractionfrom these data) can be very tedious. Secondly, it is not guar-anteed that the desired structure is fully extractable from theacquired data and additional acquisitions are sometimes re-quired in order to supplement the missing parts. In this paper,we propose to bridge the gap between data acquisition andtheir exploitation. We describe a purely image-based systemfor in-situ 3D sketching of polyhedral scenes. The built partsof the scene are immediately shown superimposed on theenvironment, which allows the user to verify the geometryagainst the physical world in real-time.In-situ modeling has been introduced in [PT01] under thename of “construction at a distance”. The principle is to al-low mobile AR users to capture planar shaped objects fromthe physical world using simple 3D primitives and construc-tive solid geometry. The user interacts with the computer us-ing a set of pinch gloves and hand tracking and the cam-era is tracked using an inertial sensor and a GPS; this sys-tem thus requires several special equipments and devicesand is only suitable for outdoor use. By contrast, a purelyimage-based interactive model building system has recentlybeen proposed in [BMC08]. This work has several com-mon aspects with our work, such as the use of a camera-mouse and model-based tracking. However, the modelingtask is less constrained in our system in that it does not re-quire any initial template to be placed in the scene, nor thatthe current model partly stays in the camera field of viewwhile new structures are added. Moreover, defining a vertexin [BMC08] is based on parallax motion which is not wellappropriate for modeling objects far from the camera.Finally, our system can be seen as an immersive ver-sion of the widely used 3D drawing software GoogleSketchUpTM (http://sketchup.google.com), whose principlesare described e.g. in [OSD05]. This tool combines some ofthe features of pencil-and-paper sketching and some of thefeatures of CAD systems to provide a lightweight, gesture-based interface for 3D polyhedral modeling. In the latest re-leases of SketchUp, the user is able to align the world axesto match a photo perspective. With this done, he can createmodels using the photo as a direct reference; mouse strokesc© The Eurographics Association 2010.Gilles SIMON / In-Situ 3D SketchingFigure 1: 3-DOF camera rotations (in modeling mode) al-ternate with 6-DOF camera motions (in tracking mode).Thick lines represent the modeled faces of the scene.are converted into 3D-space using inverse ray intersectionswith the previously defined geometry (or the ground planeby default). All these principles have been taken up in ourimplementation, but with the important difference that weconsider dynamic video images instead of static ones. More-over, the video camera is used as the interaction device in-stead of a mouse, which makes our system particularly suit-able for mobile devices such as PDAs, wearable computersand mobile phones.The system alternates between two modes of opera-tions (Fig. 1):Modeling mode. Pure rotations are applied to the camera,e.g. when using a head-mounted display, this amounts tolook around the scene by only turning the head. Usingthe camera as an interaction device, the user is able bothto calibrate the camera and to define the scene geometry(section 2). It must be noticed that this mode may be usedalone, providing already interesting contributions to clas-sical single view metrology, as (i) by turning the head thereachable field of view is much larger than the camera’sfield of view and (ii) mouse manipulations are replaced bycamera manipulations, which can be of great interest in amobile context.Tracking mode. When at least one face of the scene hasbeen described and the camera undergoes a general mo-tion, a 6-DOF (degrees of freedom) camera tracking isperformed, based on the available geometry (section 3).This allows the user to get closer to some parts of thescene or make some new faces visible before continuingmodeling. In this mode, the geometry previously modeledhas to stay (at least partly) visible in the camera field ofview, which is not required in modeling mode. A pose re-covery procedure based on SIFT features can be calledupon user request or each time a tracking failure is de-tected by the system.The system starts in modeling mode. Once a 3D geometryhas been initialized, switches between modeling and track-ing modes are done automatically, depending on the motionapplied to the camera (Akaike’s motion model selection isused [VBS03]).Figure 2: Equivalence between moving cursor / fixed cam-era (left) and fixed cursor / moving camera (right).2. Modeling InteractionsA key idea of this work is that user interactions can be doneusing the video camera itself. Indeed, let us consider that theuser wants to draw a line stroke on a video image betweentwo physical points A and B of a real-world scene; there aretwo possible ways to do this (Fig. 2) : (i) the camera staysfixed and a mouse is used to move a cursor to the two respec-tive endpoints; (ii) the camera is rotated so that the physicalpoints apparently move to a fixed cursor, for instance at thecenter of the image. The second solution, which is used inour implementation, does not require mouse support. How-ever, the position of point A has to be updated from frameto frame while point B is aimed at. Fortunately, camera rota-tions only induce homographic deformations of the image,that can be computed easily e.g. using keypoint matches.In both cases a click has to be done when the cursor (orthe camera) is correctly positioned. In the second solution,mouse clicks can be replaced by key presses or any otherinput controls such as buttons, voice commands, etc. In oursystem, only three keys are used for all operations: one to“click” or “drag and drop” in the image, a second to cancelthe current operation or request a pose recovery and a thirdto scroll the tools menu.Before being able to model the scene, the user has to cal-ibrate the camera: this is done by indicating two sets of hor-izontal parallel lines orthogonal to each other (Fig. 3, framea). In addition, the user can drag and drop the origin of theworld frame, providing a ray of possible 3D positions of thatpoint. The depth of the origin is chosen so that the unit upvector has an arbitrary size in the image plane. The user canalso change the scale of the scene by moving the extremity ofthe unit up vector. Once the camera parameters are known,the user can start modeling. New faces are instantiated incontact with existing faces or the ground plane by default.Contacts are guaranteed by snapping the clicked points toexisting 3D points which appear in a specific color whenclose to the cursor: yellow for a face vertex, blue for themiddle of an edge and red for the closest point on an edge.Faces under the cursor are also specifically outlined and new3D vertices can be generated by using inverse ray intersec-c© The Eurographics Association 2010.Gilles SIMON / In-Situ 3D Sketchinga b c de f g hFigure 3: Snapshots of the system in use.tion. In addition, most actions are guided using a line stroke,which itself can be snapped to the projected world axes.Six different tools have been implemented which are sum-marized in Tab. 1. Fig. 3 illustrates how a simple house canbe modeled using the Add, Extrude, Cut and Move tools:a rectangular face is first created using two line strokes ABand BC: according to the priority rules described at bottomof Tab. 1, AB is instantiated in the ground plane and par-allel to the red axis (frame b); the vertex C is dragged inthe plane orthogonal to AB (frame c) and finally such thatBC is aligned with the blue axis (frame d); the new face isextruded forming a box and the top face of this box is sub-divided so that the cut line joins the middles of two oppositeedges (frame e); finally, the cut line is moved along the blueaxis in order to form the roof (frame f).Add a rectangular face by clicking three points A,B,C(∗) . The rectangle is gen-erated in the plane ABC using A and C as diagonally opposite vertices.Extract texture and SIFT features from the selected face.Extrude the selected face.Move the selected vertex, edge or face.Subdivide the selected face by joining two of its vertices.Delete the selected face.(∗) The inverse ray intersection priorities are: A: selected vertex > selected face > ground;B: selected vertex > projected world axis ‖ AB > plane(A);C: selected vertex > projectedworld axis ‖ BC > plane⊥ [AB]Table 1: Modeling tools used in our prototype.3. Camera TrackingCamera poses are computed using image content onlyand the modeled planar polygons [VBS03]: Harris cornersare matched from frame to frame using normalized cross-correlation of patches in a search window; these corners aredetected in the whole image for a 3-DOF tracking or insidethe projected faces of the model for a 6-DOF tracking; out-lier matches are rejected using a RANSAC computation ofthe planar homographies that best fit the sets of corners be-tween the subsequent views; finally, the camera pose is com-puted by iteratively minimizing the sum of squared transfererrors (SSTE) of the inlier matches, using the homographiesinduced by the pose parameters and the equations of the in-volved planes.This procedure is fast and robust but has two drawbacks:first, due to the recursive nature of the algorithm, errors in-duced by noise (e.g. in corner detection) accumulate overtime. Secondly, as the size of the search window has to bebounded both to reduce matching ambiguities and to achievereal-time, camera tracking may fail in case of fast motions.To tackle the first problem, we use the texture informa-tion extracted by the user during the modeling mode. Likein [RD06], the current textured model is rendered from thecurrent pose using the OpenGL graphic engine. Harris cor-ners are detected in the rendered view and matched with theHarris corners of the current frame. These matches, whichare merged with the set of matches obtained in the previousvideo frame, tend to reduce the drift. For performance issues,this correction procedure can be called only every k frames(k = 2 in our implementation).Tracking failures are detected automatically by testingwhether the number of inlier matches falls under a threshold.The user is also able to manually require for pose recoverywhen he visually estimates that it is necessary. Pose recoveryis basically based on SIFT feature matching [Low04]. SIFTfeatures have a higher computational cost than Harris cor-ners but are invariant to image scale and rotation and robustto some extend to change in viewpoint. When the "Extracttexture" tool is used, SIFT features are extracted inside thec© The Eurographics Association 2010.Gilles SIMON / In-Situ 3D Sketchingprojected faces of the model and stored (in a kd-tree) asso-ciated with the current pose. When the recovery procedureis called, the list of textured faces is traversed until one facegets enough matches with the SIFT features of the currentframe. If one face is found, it is put at the top of the list (sothat it will be evaluated first at the next tracking failure) and arough estimate of the current pose is obtained by minimizingthe SSTE of the inlier matches. This rough pose is then usedto render an image of the textured model and the correctionprocedure is called.The recovery procedure slightly differs in modelingmode: when the system enters this mode, an initial set ofSIFT features is detected in the whole image. These featuresare transferred from frame to frame using the RANSAC-estimated homographies. The image is subdivided into a 3x3grid; when one square of the grid is empty after the SIFT fea-tures have been transferred, new SIFT features are detectedinside this square and added to the current set of SIFT fea-tures (see the outdoor video). When a tracking failure occurs,SIFT features are detected in the current image and matchedwith the current set of SIFT features.4. ExperimentsAll experiments were performed using a Dell PrecisionM6300 laptop coupled with a simple Logitech webcam. Thesystem runs at video rate in standard mode and 2 to 6Hzwhen the recovery procedure is called. Two videos are as-sociated with this paper, showing the system in action. Aminiature scene is used to assess the accuracy of the method.Several modeling and tracking sequences alternate duringthe working session (Fig. 3). It can be distinguished in thevideo between when the system is running in modeling modeand when it is running in tracking mode, as the cross cursordisappears in the second case. Fig. 4 shows the errors ob-tained on the recovered 3D geometry (scaled so that the re-covered distance d1 is equal to the expected distance). Theseerrors are acceptable for many applications: for instance, toprove that the recovered model is suitable to perform AR, weadded a seventh tool to our platform, enabling the user to addvirtual objects under the camera cursor. The added objectsappear rigidly anchored in the scene, even for a large rangeof distances from the camera to the scene (Fig. 3, framesg,h).A(mm) ∆(mm) ∆/A(%)d1 61 Ref Refd2 88 -0.3 -0.3d3 38 1.5 3.9d4 92 4.2 4.6d5 75 -5.2 -6.9d6 34 -1.3 -3.8d7 139 -8.4 -6.0d8 21 0.8 3.8d9 95 -4.0 -4.2Figure 4: Recovered geometry with actual distances A (mm),absolute errors ∆ (mm) and relative errors ∆/A (%).The modeling mode is also illustrated using a real outdoorscene (an athletism stadiumwith a changing house). The firstpart of the video shows how the set of SIFT features is ex-panded. Several abrupt motions are done, requiring the re-covery procedure to be called. The camera is then calibratedand some modeling operations are performed before a bill-board is added along the track (Fig. 5).Figure 5: One snapshot of the outdoor sequence.5. ConclusionIn this paper, we have proved that in-situ modeling is possi-ble using a standard webcam as only device. The system canbe improved in several ways. For instance, including edgefeatures to the tracking algorithm [RD06] would help to han-dle such scenes like the athletism stadium where little tex-ture information appears. Localization in larger areas wouldalso require to make the recovery procedure more scalableand efficient. Such techniques like the Potentially VisibleSets [AWK∗09] may be useful for that purpose.References[AWK∗09] ARTH C., WAGNER D., KLOPSCHITZ M.,IRSCHARA A., SCHMALSTIEG D.: Wide Area Localiza-tion on Mobile Phones. In ISMAR 2009.[BMC08] BUNNUN P., MAYOL-CUEVAS W. W.: OutlinAR: anassisted interactive model building system with reduced compu-tational effort. In ISMAR 2008.[HDT∗07] VAN DEN HENGEL A., DICK A., THORMÄHLEN T.,WARD B., TORR P. H. S.: VideoTrace: Rapid Interactive SceneModelling from Video. In SIGGRAPH 2007 papers.[Low04] LOWE D. G.: Distinctive image features from scale-invariant keypoints. Int. Jour. of Computer Vision 60, 2 (2004).[OSD05] OH J.-Y., STUERZLINGER W., DANAHY J.: Compar-ing SESAME and Sketching on Paper for Conceptual 3D Design.In EUROGRAPHICS Workshop (2005).[PT01] PIEKARSKI W., THOMAS B. H.: Tinmith-Metro: NewOutdoor Techniques for Creating City Models with an Aug-mented Reality Wearable Computer. In ISWC 2001.[RD06] REITMAYR G., DRUMMOND T. W.: Going out: Robusttracking for outdoor augmented reality. In ISMAR 2006.[VBS03] VIGUERAS F., BERGER M.-O., SIMON G.: Iterativemulti-planar camera calibration: Improving stability using modelselection. In Vision, Video and Graphics (2003).c© The Eurographics Association 2010.

http://magrit.loria.fr/Papiers/simonEurographics09.pdf

In-Situ 3D Sketching Using a Video Camera as an Interaction and Tracking Device

Abstract

Similar works

Full text

Available Versions

CiteSeerX

Hal-Diderot

HAL Descartes

INRIA a CCSD electronic archive server