Kourogi Masakatsu

Kurata Takeshi

Muraoka Yoichi

Sakaue Katsuhiko

DSpace at Waseda University

A panorama-based technique for annotation overlay and its real-time implement at ion. Masakatsu Kourogit , Takeshi Kuratat , Katsuhiko Sakauet and Yoichi Muraokat t Graduate School of Science and Engineering, Waseda University SEletrotechnical Laboratory Abstract The panorama-based annotation method described in this paper uses a panoramic image as the source of information about the positions of the annotations. It finds image alignment parameters between an input frame and the panoramic image and then maps the positions of annotations from the panoramic image to the input frame and displays the input frame over- laid with those annotations. Camera movement from place to place is made possible by preparing a set of panoramic images in advance. The panoramic image that gives the least mean squares error of the image alignment is selected automatically and is appropri- ately switched as the camera moves around. The po- sition of the camera can be tracked by monitoring the switching of selected panoramic images. Experimental results show that this method can find image align- ment parameters, display input frames overlaid with the annotations, and switch the panoramic image ap- propriately in real-time. 1 Introduction Annotation overlay on a live video is an essential feature of augmented reality (AR), since it makes pos- sible various kinds of such applications as augmented memory, touring assistance, and amusements [1][2]. Most previous work based on computer vision tech- niques, has used artificial markers (fiducials) that are placed on a real-world environment [3][4][5]. How- ever, since these markers must be physically placed on every object to be annotated, it is generally diffi- cult to cover a large-scale environment. And because the annotations need to be large enough to be detect- ed in the image, fiducials cannot be effectively used to annotate large objects in an outdoor environmen- t, such as mountains and buildings. Other previous works [6] have achieved annotation overlay in an out- door environment by using a set of dedicated loca- tion/orientation sensors such as the satellite GPS and gyroscope. But the use of these sensors is restricted to appropriate environments. The GPS signals, for example, are blocked by buildings and thus cannot be used indoors. This paper describes a novel method based solely on vision for annotation overlay without fiducials. It uses panoramic images for image alignment between posi- tions of annotations and input video frames. The pro- posed method uses (1) a set of panoramic images ac- quired at various points in the environment, (2) anno- tations attached to the panoramas and (3) neighbor- hood relationships between panoramas as prior knowl- edge about the environment. The method estimates 0-7803-6536-4/00/$10.00 (c) 2000 IEEE image alignment parameters between an input frame and each of the panoramic images. Then it selects the panorama that gives the least mean squares error of image brightness by the alignment, and overlays the annotations of the selected panorama on the frame. Experimental results show that this method can, with low-cost PCs, locate and orient input frames and dis- play the frames overlaid with the annotations in real- time. 2 Panorama-based annotation overlay Our approach uses a panoramic image to which an- notations are manually attached as the source of in- formation. When an input frame is given, it is aligned with the referred panoramic image. Then, we can map the positions of annotations from the panorama to the frame, and thus can create the frame overlaid with the annotations as shown in Figure 1. annotation Input h e  Annotatedframe . Figure 1: Panorama-based annotation overlay. Two-dimensional image alignment, however, can- not handle the motion parallax caused by transla- tional displacement of the viewpoint of a camera. If the viewpoint from which video frames are captured is translated apart from the one the panoramic im- age is acquired, it is in principle impossible to align the video frames with the panorama. To handle the displacement, we use a set of panoramic images ac- quired at various points in the environment in ad- vance. Video frames are then aligned with the panora- ma acquired at the viewpoint nearest the one at which the video frames are captured. The referred panoram- ic image will be switched if necessary as the camer- a moves around. The neighborhood relationship be- tween panoramas are given to enable the switching to occur. By tracking which panorama is referred, we 657 can also estimate the position and trajectory of the user. An overview of the panorama-based approach is shown in Figure 2. Figure 2: Overview of panorama-based annotation. 2.1 Image alignment between frame and panorama We estimate image alignment parameters between an input frame and a referred panoramic image by us- ing a fast and robust gradient-based method [7] that can find affine or projective parameters of image align- ment between images. We use f f i e  model because of its stability of estimation. The f f i e  transformation matrix A consists of 6-parameters, and can be written as follows: A = [  : 0 0  : $ 1  (1) Since the method [7] is empirically known to have difficulties to stably estimate the parameters if trans- lational component (a3,ag) is large, we give multi- ple initial estimates to the method and then estimate the parameters respectively from the initial estimates. Then, we select the estimated result that gives the best match from them. Our method evaluates the results of image align- ment in terms of the mean squares error (MSE) of image brightness between the ima es. Suppose that a point (z,y) on the frame is transkrmed to the point (z’,y’) on the referred panorama by the &ne trans- form matrix A j + p .  Let If(”, y) and Ip $’, y’) panorama, let S be a set of points in the frame, and let N s  be the number of points in S. The MSE can be calculated by using the following equation: tively be image brightnesses of the the I lame and respec- theSince MSE calculation is computationally expensive, the number of the pixels N s  is reduced by selecting those with large absolute gradients of image bright- ness. 2.2 Scale factor of image alignment When the frame is aligned with the referred panora- ma by translation, rotation, and scaling, the scale fac- tor s can be calculated as s = Jq. (3) This factor represents the ratio of the size of an object in the frame to the size of the object in the panorama. Let f arid fp be the focal lengths of the cameras that capture frames and the panorama, respectively. If the scale factor is close to fp/ f, it can be assumed that the position where the frame is captured is close to the one at which the panorama is acquired. 2.3 Evaluation criterion of image align- ment The MSE of image brightness is not necessarily by itself a good measure of image alignment between a frame and a set of panoramas, since the MSE can be small even when the scale factor is far from fp/f. Our method therefore evaluates the result of image alignment as follows. If more than one results give MSEs smaller than a threshold, the method selects the one that gives the scale factor nearest f p / f .  2.4 Neighborhood relationship between To allow a user’s camera to move around in the environment, neighborhood relationships are given to any two panoramic images if the positions where the images are acquired are proximate. We define the neighborhood relationship between two panoramas Ipi and Ip j  by a data set as consisting of the following el- ements: panoramas 1. 2. 3. A region Bk in the panorama I p i .  The adjacent panorama 1,. The &ne transform parameters A ~ ~ : p ; - + p ~  for the transformation from Ipi to I p j  on the region Rk . Region Adjacent panorama Transformation parameters Rk 1.1 +pr adjac Figure 3: Neighborhood relationship. A point (zpi,ypi) in the panorama Ipi is trans- formed to the point ( z p j ,  ypj) in the panorama l p j  by using the f i n e  transform matrix A ~ ~ : p ~ + p ~  and the following equations: (4) 0-7803-6536-4/00/$10.00 (c) 2000 IEEE 658 of four conventional PCs (CPU: two Dual PentiumII- 450MHz, two Dual PentiumIII-500MHz, OS: Linux- 2.2.11 SMP supported) connected via 100-M ethernet. = A R ~ : P ; + P ~ A ~ + P ~  [ ij ] . (5) When image alignment parameters between a frame I f  and the panorama Ip i  transforms the central posi- tion (zf,, y ) of the frame to a point on the region Rk, the frame f ;  is likely to match with the correspond- ing region in the panorama Ip j  as shown in Figure 3. Thus, the method estimates the image alignment parameters between the frame and the panorama I p j  in parallel by using the product of two f f i e  transfor- m matrices A ~ ~ : p ~ + p ~ A f + p ~  as an initial estimate. If image alignment between the frame and an adja- cent panorama gives a result better than that given by the alignment between the frame and the currently referred panorama, the referred panorama for the next frame will be switched to the adjacent one. The neighborhood relationships have to be given to any two panoramas on which switching can occur by the user’s movement. Since the user is most likely to move in the direction he is looking, at least one neighborhood relationship needs to be given to the regions in the panoramas that the user can move to. 2.5 Searching for the panorama To find the panoramic image that includes an in- put video frame, we give multiple initial estimates to each of the panoramas in the set, so that at least one of the initial estimate will be sufficiently close to the true parameters. Then we select the best combination of parameters and panorama by using the criterion described in Section 2.3. Once the best panorama- parameters pair is found, the parameters are used as the initial estimate for image alignment of the next frame. 3 Experiments We implemented our method as software running on a PC cluster and evaluated it in experiments in which a user was equipped with a wearable display, a wearable camera, and a wearable computer. 3.1 Implementation The software of our implementation uses the Par- allel Virtual Machine (PVM) library [8 for data dis- independently from architecture and operating system of targeted computer. For high-performance compu- tation, we used parallel computation based on multi- thread programming model: the POSIX thread for UNIX and the Win32 thread for Windows NT/95. The unit of processing is the estimation, from one initial estimate, of the &ne parameters of image alignment between a frame and a panorama. In this software, the process of estimation is imple- mented as a thread code so that it can be speeded up by increasing the number of CPUs and PCs without having to rewrite the code. As shown in Figure 4, we used a small CCD cam- era to capture input video frames and used a head- worn display (HWD) to show the output frames over- laid with annotations. A mobile PC held by the us- er captured and compressed the video data by a fac- tor of 20 with JPEG encoding and transmitted it to the PC cluster via a wireless network complying with the IEEE 802.11 standard. The PC cluster consisted tribution and collection among PCs so t ll at it can run CCD camera head-worn display - -  Figure 4: User with a HWD and a camera. 3.2 Creation of the prior knowledge Before the experiments, a set of four panoramic im- ages were acquired at point A-D in Figure 5 by pan- ning a camera 360 degrees. These panoramas were created automatically by mapping the successive video frames to a cylindrical plane by using the method [7]. The created panoramas are shown in Figure 6. Then annotations were manually attached to the panora- mas and neighborhood relationships between panora- mas were given. - c o m d o r D r  W Workstation P Personal computer Tv: Television corridor - - Figure 5: Environmental map. 3.3 Experimental results In the experiments the user moved along the path A + B + C + D (Figure 5). The output video frames overlaid with annotations are shown in Figure 7, and the user’s estimated position and orientation are shown in Figure 8 and 9. It took 2000-3000 msec to search the set of four panoramas for the panoramic image that included the first input frame. The throughput of the annotation overlay was 100-120 msec or 8-10 frames per second and the delay is 600-800 msec. The results of annotation overlay show that the software implementation of the proposed method can robustly provide video frames overlaid with annota- tions in real-time. 4 Conclusion This paper describes an annotation overlay method that uses a set of panoramic images as a source of in- formation. Experimental results show the software im- plementation of the method can provide video frames overlaid with annotations in real-time. Acknowledgements This work was conducted as a part of the Real World Computing (RWC) Program. 0-7803-6536-4/00/$10.00 (c) 2000 IEEE 659 Figure 6: A set of panoramas acquired at point A-D (top to bottom). References R. Azuma, “A survey of augmented reality,” in P- resence: Teleoperators and Virtual Environments, vol. 6, no. 4, pp. 355-385, 1997. J. Rekimoto and K. Nagao, “The world through the computer: Computer augmented interaction with real-world environments,” in Proc. of UIST U. Neumann and J. Park, “Extendible object- centric tracking for augmented reality,” in Proc. of T. Starner, S. Mann, B. Rhodes, J. Levine, J. Healy, D. Kirsh, R. Picard and A., Pentland, “Augmented reality through wearable computing,” in Presence: Teleoperators and Virtual Environ- ments, vol. 6, no. 4, pp. 386-398, August 1997. D. Koller, G. Klinker, E. Rose, D. Breen, R. Whitaker and M. Tuceryan, “Real-time vision- based camera tracking for augmented reality,” in Proc. of VRST’97, pp. 87-94, 1997. S. Feiner, B. MacIntyre and T. Hollerer, “Wearing it out: First steps toward mobile augmented reali- ty systems,” in Mixed Reality - Merging Real and Virtual Worlds, pp. 363-377, Ohmusha-Springer Verlag, 1999. M. Kourogi, T. Kurata, J. Hoshino and Y .  Muraoka, “Real-time image mosaicing from a video sequence,” in Proc. of IcIP’99, Vol. 4, pp. 133- 137, 1999. A. Geist, “PVM: Parallel Virtual Machine, A user- s’ guide and tutorial for networked parallel com- puting,” MIT Press, 1994. ’95, pp. 29-36, 1995. VRAIS’98, pp. 148-155, 1998. Figure 7: Output frames overlaid with annotations. 0 50 100 150 200 250 300 350 400 frame number Figure 8: Estimation of the user’s position. 80 2 60 s40 3 a 20 B C O  3 -20 .- ’E $4 bo go 0 50 100 150 200 250 300 350 400 frame number Figure 9: Estimation of the user’s orientation. 0-7803-6536-4/001$10.00 (c) 2000 IEEE 660 

A panorama-based technique for annotation overlay and its real-time implementation

Kourogi, Masakatsu

28055

Kurata, Takeshi

28056

Sakaue, Katsuhiko

28057

Muraoka, Yoichi

28058

Waseda University Repository

https://waseda.repo.nii.ac.jp/?action=repository_action_common_download&item_id=17289&item_no=1&attribute_id=22&file_no=1

A panorama-based technique for annotation overlay and its real-time implementation

Abstract

Similar works

Full text

Available Versions

DSpace at Waseda University

Waseda University Repository