We present a novel multi-step technique for imperceptible geometry and radiometry calibration of projector-camera systems. Our approach can be used to display geometry and color corrected images on non-optimized surfaces at interactive rates while simultaneously performing a series of invisible structured light projections during runtime. It supports disjoint projector-camera configurations, fast and progressive improvements, as well as real-time correction rates of arbitrary graphical content. The calibration is automatically triggered when mis-registrations between camera, projector and surface are detected

Bimber, Oliver

Zollmann, Stefanie

Bimber, Oliver (Prof. Dr.)

Online-Publikationssystem der Bauhaus-Universität Weimar

EUROGRAPHICS 2007 / D. Cohen-Or and P. Slavík(Guest Editors)Volume 26 (2007), Number 3Imperceptible Calibration for Radiometric CompensationS. Zollmann and O. BimberBauhaus-University Weimar, GermanyAbstractWe present a novel multi-step technique for imperceptible geometry and radiometry calibration of projector-camera systems. Our approach can be used to display geometry and color corrected images on non-optimizedsurfaces at interactive rates while simultaneously performing a series of invisible structured light projections dur-ing runtime. It supports disjoint projector-camera configurations, fast and progressive improvements, as well asreal-time correction rates of arbitrary graphical content. The calibration is automatically triggered when mis-registrations between camera, projector and surface are detected.Categories and Subject Descriptors (according to ACM CCS): H.5.1 [INFORMATION INTERFACES AND PRE-SENTATION]: Multimedia Information SystemsArtificial, Augmented, and Virtual realities; I.3.3 [ComputerGraphics]: Picture/Image Generation I.4.8 [Image Processing and Computer Vision]: Scene Analysis1. Introduction and MotivationProjections onto complex everyday surfaces is challenging.Projector-camera systems together with appropriate sceneanalysis techniques and real-time image correction methodscan improve the quality of graphical content projected onsuch surfaces. To support this, surface properties, such asgeometry and reflectance parameters, need to be scannedin advance before using them for image corrections dur-ing run-time. This is known as radiometric compensation[NPGB03], [BEK05], [FGN05]. In most cases, a one-timecalibration process applies visible structured light projec-tion and camera feedback to establish the correspondencebetween camera and projector pixels and to measure the ra-diometric behavior of the surface pigments. The mappingsand color values can be stored in look-up textures that canbe processed by the GPU to perform pixel-displacementmapping and a pixel-precise color compensation in real-time [BEK05]. For many applications, however, a frequentre-calibration is necessary because the alignment of cameraand projectors with the surfaces changes over time (e.g., dueto mechanical expansion through heating, accidental offset,intended readjustment). In these cases, it is not desired todisrupt a presentation with visible light patterns.2. Related and Previous WorkA first proof of concept for embedding invisible struc-tured light patterns into DLP projections was introducedin [RWC∗98]. In this work, binary codes have been embed-ded by projecting temporally alternating code images andtheir complements. A variation of this approach was appliedby [WWC∗05] and [VVSC05] for 3-D video acquisition. Inall cases, however, the projected patterns are well detectableby human observers. Even in [RWC∗98] the code patternsmust be visible during eye movements† and while temporalcode transitions. In [GSHB07] properties of human percep-tion are taken into account for adapting the coding parame-ters depending on local characteristics of image and code.This makes a truly imperceptible temporal coding of bi-nary information possible. Cotting et. al. introduced a codingscheme [CNGF04] that synchronizes a camera to a specifictime slot of a DLP micro-mirror flipping sequence in whichimperceptible binary patterns are embedded. However, notall mirror states are available for all possible intensities. Tostill achieve the desired binary value during this period, colorvalues of the original image have to be adjusted. This re-sults in a tonal shift of the original image, which can be high- especially for upper and lower intensity ranges. Dynamic† caused by visual perception effect called phantom-arraysubmitted to EUROGRAPHICS 2007.2 S. Zollmann and O. Bimber / Imperceptible Calibration for Radiometric CompensationFigure 1: A series of invisible patterns (b-e) integrated into an image (a) and projected onto a complex surface (g) results insurface measurements (f-i) used for radiometric compensation (j).range reductions of up to 40-50% have been reported, andonly geometric calibrations are achieved.Instead of visible or invisible structured light projections,features of the captured distorted projection can be analyzedto support the calibration of a projector-camera systems.A first example was presented in [YW01], while advancedvariations have been described later. The possibility of es-tablishing the correspondence between projector and cam-era pixels in these cases, however, depends on the qualityof the detected images features. Consequently, this is not in-dependent on the projected content. Again, only geometriccorrections and simple intensity adjustments are currentlysupported by such techniques.Fujii et al. have described a dynamically adapted radio-metric compensation technique that supports changing pro-jection surfaces and moving projector-camera configurations[FGN05]. Their system requires a fixed co-axial alignmentof projector and camera. An optical registration of both de-vices makes a geometric calibration unnecessary. Since astrict optical alignment is not possible in every case, such anapproach is too inflexible for many applications (e.g., whenmultiple projectors are mounted at locations that are differ-ent from the position of the calibration camera). Further-more, the quality of the radiometric compensation stronglydepends on the projected image content.Our technique applies a temporal coding that is basedon [RWC∗98] with enhancements for human visual percep-tion [GSHB07]. Projectors and camera do not have to bemechanically or optically aligned. Geometric registration aswell as radiometric measurements are achieved through awell-cosen series of invisible structured light patterns em-bedded into the projected content (cf. figure 1). Images areprojected at interactive frame rates, while the image qualityis progressively improved after geometric mis-alignmentsare detected automatically.3. Coding and ReconstructionFor embedding a binary patternC into the projected originalimage O, a code image Pcode can be calculated by adding asmall value ∆ onto O to encode a binary 1, or subtracting ∆from O for encoding a binary 0. Thereby, Omight have to bescaled to ∆ ≤ O ≤ 1−∆. However, since ∆ is small (3-5%of the total intensity range in our case), this does not influ-ence the visual appearance of Omuch. When displaying, theprojected code image must be compensated by its comple-mentary image Pcomp = O−Pcode. Projecting these two im-ages with a refresh rate above the critical flicker frequency,one perceives the temporal integration of both (approx.(Pcode+Pcomp)/2). A dynamic adaptation of ∆ based on lo-cal image properties, such as spatial frequencies and localluminance in O and C will ensure that the code remains in-visible even through fast eye-movements and temporal codetransitions, as explained in [GSHB07]. A synchronized cal-ibration camera that captures both images (Pcode and Pcomp)can reconstruct the code (C = ((Pcode − Pcomp) <=> 0)).This method is robust against noise and invariant of the sur-face reflectance. As explained in section 4 it is used in earlycalibration steps that apply fast binary code patterns, such aspoints or stripes (cf. figures 1b,d).To encode intensity values (C = ∆) instead of binary val-ues,O is multiplied by ∆ to compute the code image (Pcode=O∆). The compensation image is then given by Pcomp =O(2−∆), and the embedded code can be reconstructed fromboth images with C = ∆ = 2Pcode/(Pcode+Pcomp). Embed-ding intensities offers the possibility to use advanced geo-metric calibration techniques, such as phase shifting [HZ06].It is used by later calibration steps (cf. figure 1e) that deliverprecise geometric mappings (explained in section 4). Insteadof a local adaptation of ∆ as for binary coding, only a globaladjustment based on average properties (spatial frequencyand luminance) of O and C can be used for intensity codingto preserve the correct intensity values.Instead of embedding binary or intensity values into thegray channel, all three color channels can be encoded differ-ently. This, however, requires neutralizing the mixing amongdifferent color channels caused by projector filters and cam-era sensor. Determining and applying color transformationis a common technique for radiometric compensation (e.g.,[NPGB03], [FGN05]), which has to be applied in this caseas well. Yet, we found that color coding is quite unreliablefor almost all embedded intensity codes and even for mostbinary codes that are projected on textured and colored sur-submitted to EUROGRAPHICS 2007.S. Zollmann and O. Bimber / Imperceptible Calibration for Radiometric Compensation 3faces. Thus we use the gray channel only for the sake ofstability. Note, that a correct coding and decoding is onlypossible if projectors and camera are linearized. A gammacorrection on O is required before embedding the code. Thissatisfies the non-linear response of the human visual system.All coding, decoding and correction calculations are carriedout with fragment shaders on the GPU to achieve real-timeperformance on a per-pixel basis.4. Multi-Step CalibrationBased on the coding techniques described in section 3, wepropose a multi-step calibration process that aims at deliv-ering approximated results quickly, while carrying out sub-sequent progressive improvements. All steps are impercepti-ble while presenting graphical or video content at interactiveframe rates.A constant dense pattern of binary points is encoded andprojected continuously. A mis-registration between projec-tor, camera and surface is detected automatically if the bi-nary difference between two subsequent captured code pat-terns is above a predefined threshold. The re-calibration istriggered immediately after subsequent code patterns matchagain (following an earlier mis-registration). The three cali-bration steps and their produced results are illustrated in fig-ure 1.Step 1: A sequence of binary point codes (cf. figure 1b)is projected to sample pixel correspondences between pro-jector and camera. Since only discrete positions are mea-sured but a continuos look-up table is require for pixel-displacement mapping on the GPU, intermediate values haveto be interpolated. This is achieved by a Delauny triangula-tion and an interpolation of the 2D displacement samplesduring off-screen rendering. The resulting look-up texture(cf. figure 1f) is passed to the GPU for geometric image cor-rection [BEK05]. A coarse geometric registration is the re-sult of setp 1 (cf. figure 2c).Step 2: The surface reflectance is analyzed next by em-bedding a uniform gray image I into the projection (cf. fig-ure 1c). Capturing the surface under a uniform gray illumi-nation results in a diffuse reflectance map (cf. figure 1g) thatis required by compensation techniques, such as [BEK05].The advantage of this is, that in contrast to the method usedin [FGN05], this approach is independent of the image con-tent, and therefore more reliable. However, the original im-age O has to provide a global minimum gray level of I. Ifthis is not the case,O is increased slightly in brightness. Notethat this does not result in local tonal shifts, as in [CNGF04].Depending on the exposure times and projector brightness, Ican be fairly small (3-5% of the total intensity range in ourcase). Thus, while Pcode = I and Pcomp =O−Pcode, the per-ceived result is (Pcode +Pcomp)/2, and the embedded grayimage is not visible.Step 3: The system is now roughly calibrated and displaysan approximated result. If no further mis-regitrations are de-tected, a more accurate (but also more time-consuming) cali-bration step is triggered. It is based on the 3-step phase shift-ing algorithm described in [HZ06], which measures precisecorrespondences for each projector/camera pixel. For phaseshifting, three sinusoidal patterns (each shifted by 120 de-gree, −120◦ for I1, 0◦ for I2 and +120◦ for I3) have to beprojected for x and y directions (cf. figure 1e for x). The cap-tured images are used to convert the decoded intensities intowrapped phase values for each pixel (cf. figure 1h for y), asin [HZ06]:φrel(x,y) = arctan(√3I1− I32I2− I1− I3 ) (1)The results of equation 1 ranges from−pi to pi and are calledwrapped phase values. In this context “wrapped” refers tothe fact that the phase values contain discontinuities. Forderiving continuous phase values (called unwrapped or ab-solute phase map) a transformation from relative values toabsolute values has to be applied. This transformation de-pends on the characteristic of the underlying surface. Forgeometries consisting of one single-connected componentthe phase unwrapping technique described in [ZLY07] canbe used. It removes the the 2pi discontinuities from wrappedphase values by examining the phase map gradients usingtwo additional maps indicating the quality of the calculatedphase values. For geometries consisting of non-connectedcomponents the projected calibration image is divided in toseveral rectangular regions first, whereas each region is de-fined by the periods of the relative phase values. The regionmap, that is applied to transform the phase values into ab-solute values, is determined by projecting 2ld(r) additionalgray code patterns (cf. figure 1d), where r is the number ofdifferent regions. The unique gray code k of each region ismultiplied with 2pi and added to the relative phase values.This results in absolute phase values (φabs = k2pi+φrel).After transforming the realtive phase values into abso-lute ones, triangulation and interpolation creates a continuoslook-up table (cf. figure 1i). This allows for a pixel-preciseupdate of the previous geometric calibration (cf. figure 2d).An additional reflectance analysis is not necessary. All com-putations are carried out in fragment shaders on the GPU.5. Summary and Future WorkIn this paper we presented a multi-step, imperceptible cal-ibration technique for projector-camera systems. To ourknowledge, this is the first imperceptible calibration ap-proach that 1) enables radiometric measurements indepen-dent of the image content, 2) supports intensity coding,and 3) allows a pixel-precise geometric calibration that nei-ther requires a fixed optical alignment nor a static (pre-calibrated) projector-camera arrangement. Our system ap-plies a series of invisible structured light patterns to detectmis-registrations, to compute and display corrected imagequickly, and to improve the results progressively. The visualsubmitted to EUROGRAPHICS 2007.4 S. Zollmann and O. Bimber / Imperceptible Calibration for Radiometric CompensationFigure 2: Sample results for different surfaces and multiple images: originals (b,f,j,n), without compensation (a,g,k,o) and withcompensation (c,d,h,l,p). Close-up after step 2 with slight error (c) and close-up after step 3 (d).quality of the displayed images, such as tonal shifts or ex-treme dynamic range reductions, is -except to a small con-stant scaling in brightness- not influenced.Assuming that b blending steps are required for seamlesscode transitions, as explained in [GSHB07], ld(p)b1 codeimages have to be captured for steps 1, where p is the num-ber coded binary points. For step 2, b2 code images haveto be captured. The optional gray code projection in step 3requires to capture 2ld(r)b3 code images (for x and y direc-tions), while for the intensity-based phase-shifting 6b4 codeimages are necessary (for x and y directions). Currently, atleast 50ms are required for acquire one code pattern (by cap-turing Pcode and Pcomp), image processing, coding/decodingcomputations and content rendering. This is equivalent to aframe rate of 20Hz for content presentation with simultane-ous calibration. However, the sub-images Pcode and Pcompare continuously projected with a constant speed of 120Hz,resulting in a perceived refresh rate of 60Hz. If for instance,as in our applications, b1 = 4, b2 = 10, b3 = b4 = 8, p= 64and r = 32 is required, then an approximated re-calibrationcan be displayed after 1.7s, while the high-quality updatecan be delivered after additional 6.4s. A decrease of theoverall calibration time can be achieved by multi-threading.Although the projection sequence remains sequential, de-manding computations could be processed during projectionand capturing. This optimization belongs to our future work.Furthermore, we want to investigate perception-based imageadaptation techniques [GB07] that allow to dynamically in-crease the local brightness in O (for ensuring the minimumgray level required for the reflectance measurements) whileminimizing the perceived difference to the unadapted ver-sion of O.References[BEK05] BIMBER O., EMMERLING A., KLEMMER T.:Embedded entertainment with smart projectors. IEEEComputer 38 (2005), 48–55.[CNGF04] COTTING D., NAEF M., GROSS M., FUCHSH.: Embedding imperceptible patterns into projected im-ages for simultaneous acquisition and display. In ISMAR’04: Proceedings of the Third IEEE and ACM Interna-tional Symposium on Mixed and Augmented Reality (IS-MAR’04) (Washington, DC, USA, 2004), IEEE ComputerSociety, pp. 100–109.[FGN05] FUJII K., GROSSBERG M., NAYAR S.: Aprojector-camera system with real-time photometricadaptation fordynamic environments. pp. I: 814–821.[GB07] GRUNDHOEFER A., BIMBER O.: Real-timeadaptive radiometric compensation. To appear in Trans-actions on Visualization and Graphics (2007).[GSHB07] GRUNDHÖFER A., SEEGER M., HÄNTSCHF., BIMBER O.: Coded projection and illumination fortelevision studios. Technical Report no. 843, Bauhaus-University Weimar (2007).[HZ06] HUANG P. S., ZHANG S.: Fast three-step phase-shifting algorithm. Appl. Opt. 45, 21 (2006), 5086–5091.[NPGB03] NAYAR S. K., PERI H., GROSSBERG M. D.,BELHUMEUR P. N.: A projection system with radio-metric compensation for screen imperfections. In Proc.ICCV Workshop on Projector-Camera Systems (PRO-CAMS) (2003).[RWC∗98] RASKAR R., WELCH G., CUTTS M., LAKEA., STESIN L., FUCHS H.: The office of the future: aunified approach to image-based modeling and spatiallyimmersive displays. In SIGGRAPH ’98: Proceedings ofthe 25th annual conference on Computer graphics and in-teractive techniques (New York, NY, USA, 1998), ACMPress, pp. 179–188.[VVSC05] VIEIRA M. B., VELHO L., SA A., CAR-VALHO P. C.: A camera-projector system for real-time 3dvideo. In CVPR ’05: Proceedings of the 2005 IEEE Com-puter Society Conference on Computer Vision and PatternRecognition (CVPR’05) - Workshops (Washington, DC,USA, 2005), IEEE Computer Society, p. 96.[WWC∗05] WASCHBÜSCH M., WÜRMLIN S., COTTINGsubmitted to EUROGRAPHICS 2007.S. Zollmann and O. Bimber / Imperceptible Calibration for Radiometric Compensation 5D., SADLO F., GROSS M. H.: Scalable 3d video of dy-namic scenes. The Visual Computer 21, 8-10 (2005), 629–638.[YW01] YANG R., WELCH G.: Automatic and continu-ous projector display surface calibration using every-dayimagery. InWSCG 2001 Conference Proceedings (2001),Skala V., (Ed.).[ZLY07] ZHANG S., LI X., YAU S.-T.: Multilevelquality-guided phase unwrapping algorithm for real-timethree-dimensional shape reconstruction. Applied Optics46, 1 (2007), 50–57.submitted to EUROGRAPHICS 2007.

Imperceptible Calibration for Radiometric Compensation

Digitale Bibliothek Thüringen

https://e-pub.uni-weimar.de/opus4/files/809/Imperceptible+Calibration.pdf

Imperceptible Calibration for Radiometric Compensation

Abstract

Similar works

Full text

Available Versions

Online-Publikationssystem der Bauhaus-Universität Weimar

Digitale Bibliothek Thüringen