This paper discusses a technique used for construction of high-resolution image mosaic from a videosequence and the synchronously logged camera attitude information. It allows one to infer geometric characteristics of the imaged terrain and hence improve the mosaic quality and reduce the computational burden. The technique is demonstrated using numerical modeling and is applied to videodata collected on Rainsford Island, Mass. Calculation of the transformation relating consecutive image frames is an essential operation affecting reliability of the whole mosaicing process. Improvements to the algorithm are suggested, which significantly decrease the possibility of convergence to an inappropriate solution

Cutter, Randy G, Jr.

Huff, Lloyd C

Rzhanov, Yuri

English

University of New Hampshire

University of New Hampshire
University of New Hampshire Scholars' Repository
Center for Coastal and Ocean Mapping Center for Coastal and Ocean Mapping
9-2002
Seafloor Video Mapping: Modeling, Algorithms,
Apparatus
Yuri Rzhanov
University of New Hampshire, Durham, Yuri.Rzhanov@unh.edu
Lloyd C. Huff
University of New Hampshire, Durham
Randy G. Cutter Jr.
University of New Hampshire, Durham
Follow this and additional works at: https://scholars.unh.edu/ccom
Part of the Computer Sciences Commons, and the Oceanography and Atmospheric Sciences and
Meteorology Commons
This Conference Proceeding is brought to you for free and open access by the Center for Coastal and Ocean Mapping at University of New Hampshire
Scholars' Repository. It has been accepted for inclusion in Center for Coastal and Ocean Mapping by an authorized administrator of University of New
Hampshire Scholars' Repository. For more information, please contact nicole.hentz@unh.edu.
Recommended Citation
Rzhanov, Yuri; Huff, Lloyd C.; and Cutter, Randy G. Jr., "Seafloor Video Mapping: Modeling, Algorithms, Apparatus" (2002).
International Conference on Image Processing (ICIP). 255.
https://scholars.unh.edu/ccom/255
SEAFLOOR VIDEO MAPPING: MODELING, ALGORITHMS, APPARATUS 
Y Rzhanov, L. HUE G. R. Cutter 
Center for Coastal and Ocean Mapping (C-COM) 
University of New Hampshire 
Durham 03824, USA 
email: yuri.rzhanov@unh.edu, lhuff@cisunix.unh.edu, gcutter@cisunix.unh.edu 
ABSTRACT 
This paper discusses technique used for construction of high- 
resolution image mosaic from a video sequence and the syn- 
chronously logged camera attitude information. It allows 
one to infer geometric characteristics of the imaged terrain 
and hence improve mosaic quality and reduce computational 
burden. The technique is demonstrated using numerical mod- 
eling and is applied to video data collected on Rainsford 
Island, Mass. 
Calculation of the transformation relating consecutive 
image frames is an essential operation affecting reliability 
of the whole mosaicing process. Improvements to the algo- 
rithm are suggested. which significantly decrease the possi- 
bility of convergence to an inappropriate solution. 
1. INTRODUCTION 
In the previous publications 11. 21 we have discussed po- 
tential use of auxiliary sensors (namely, attitude sensor and 
GPS receiver) for improvement of quality of image mosaics, 
in particular, for underwater imagery. These mosaics have 
important applications for visual survey of large areas or 
objects under the conditions of poor visibility. Our primary 
interest lies in area of benthic habitat mapping (e.g. [31). 
although pipeline inspection, sediment characterization. ar- 
chaeological and forensic site mapping could also be po- 
tential applications of this technique. lo name a few. Cur- 
rently video mosaicing is the only mapping method which 
provides resolution better than 1 cm per pixel and rapid con- 
tinuous coverage of large areas. 
It seems obvious that accurate measurement of camera 
position and orientation can greatly facilitate the construc- 
tion process of a synthetic mosaiced image. The difficulty 
however is that existing low-cost sensors are not accurate 
enough to be directly used in the mosaicing process. Indeed. 
a GPS receiver may provide information about location of 
the antenna with an accuracy of 10-20 cm. If the camera is 
underwater and antenna is on the surface, the accuracy is re- 
duced to 1-2 meters. yet the typical resolution of imagery is 
less than 1 cm. Even reasonably accurate measurements of 
the camera Euler angles are not sufficient to position a pixel 
to within 1 cm. For example, 0.5 degree error in tilt mea- 
surement easily translates into 2.5 cm error for 30 degree 
camera pitch (for typical imaging distances). 
Assumption of flat horizontal (FH) terrain allows to de- 
rive unique functional relationship between position and ori- 
entation of the camera and elements of transformation (ho- 
mography) used to create a mosaiced image. If terrain is 
indeed described well by the FH model, resulting mosaic is 
of good quality and camera motion can be recovered: dif- 
ference between homographies calculated from the camera 
vectors and ones obtained from image co-registration pro- 
cedures stays small. This difference going up indicates that 
the FH model becomes inapplicable. More sophisticated 
models, however, cannot solve for camera motion and ter- 
rain characteristics simultaneously - solution is not unique. 
By employing sensor measurements we eliminate ambigu- 
ity and infer some local topographic characteristics of the 
terrain. 
2. MODELING OF THE ACQUISITION AND 
RE-PROJECTION PROCESSES 
To understand the mosaicing process in details, a model of 
video acquisition was developed. The modelled imaged sur- 
face is described in terms of digital elevation model (DEM) 
in association with a raster image, draped over the topogra- 
phy. Both elevation and luminance (or color) values are de- 
fined on a regular grid and are assumed to change monoton- 
ically in between. The DEM is with respect to a "zero level 
plane", which is normal to Y axis of the associated system 
of coordinates and passes through point Y = H. Cam- 
era orientation and position is dey ibed  by 6 parameters. 
combined in an "Euler vector", E, components of which 
represect pitch. roll and yaw of the camera, and translation 
vectors determining offset with respect to the center of ori- 
gin, 0. In case of zero Euler vector. ideal pinhole camera 
is oriented along axis X ("North") and is looking at the ter- 
rain vertically down (along -Y direction). "West" direction 
0-7803-7622-6/02/$17.00 02002 IEEE 1 - 868 IEEE ICIP 2002 
then corresponds to Z axis. The acquired image (frame) lies 
in XZ-plane, "up" direction being "North". and "left" being 
"West". (The described system is widely used in graphic 
display field.) 
We also need to introduce number of pixels in the frame, 
Nx (columns) and Ny (rows), and camera field of view 
in one of the directions. say. horizontal. FoV. Then the 
normalized focal distance. F, is equal to z tan~ov ,z .  
In case of zero Euler vector, flat horizontal terrain and 
H = F, focal plane coincides with the imaged surface, and 
the mapping between grid points of the terrain and image 
pixels is determined by a unit homography. Decrease in H 
results in zooming in the imaged surface, so we can intro- 
duce zoom ratio Z = 6 .  Shift of the camera in the plane 
parallel to the terrain leads to the corresponding shift of the 
imaged area: shifts are convenient to normalize by num- 
ber of pixels in the horizontal direction (two images shifted 
horizontally with 50 percent overlap have normalized shift 
0, = 0.5). 
If the terrain is described by the FH model, geometric 
considerations provide a unique relationship between cam- 
era parameters (E, Z, D,, Dy) and coefficients of the rec- 
tification homography R, Le. transformation between the 
coordinates of terrain W and pixels of the frame I (see Ap- 
pendix). Re-projection process - mapping of pixels onto the 
terrain space - can be denoted symbolically as W = R x I. 
(Note that the normalized shifts in image space D, and 
Du correspond to shifts in 2- and X- directions in the real 
space.) 
Two frames IO and I1 imaging approximately the same 
area could be co-registered, that is, a homography To1 could 
,' be found, mapping frame Il onto the image space of 11, 
thus combining them in a mosaic. Denote this symbolically 
as IO = To1 x Il. If rectification (world) homography is 
known for frame IO, then the world homography for frame 
4 isR1 = &xTol ,  as W = R O x  IO = ( & x T o l ) x h  = 
R1 x I,. Other frames, related to IO through a chain of rel- 
ative homographies, Tj,j+l, could be added to the global 
mosaic in the same way: W = & x Tj,j+l x I k .  The 
latter formula shows that if the rectification homography for 
the initial frame, &,was estimated inaccurately, this would 
affect global mosaic as a whole, while numerical errors ac- 
cumulated in co-registration procedures of finding relative 
homographies, Tj,j+l, may lead to local distortions of the 
mosaic. 
If residual errors are negligible, rectification homogra- 
phies for frames in the acquired sequence can be directly 
used for construction of the global mosaic. Relative homo- 
graphies guarantee optimal painvise merging of the frames 
- rectification homographies guarantee optimal merging of 
all frames on a common image space (terrain map). 
Modeling of video acquisition process with non-flat ter- 
k-1  
j = O  
rain shows that the above statements do not hold true. Rel- 
ative homographies found using one of the co-registration 
procedures (typically, based on either feature tracking or 
optimization) may not correspond to any set of vectors de- 
scribing the camera, as the model, used to relate these pa- 
rameters to the elements of homography, fails. We suggest 
a more sophisticated model, which describes the terrain el- 
evation in terms of 2D low-order polynomials, with coeffi- 
cients b$ng model variables, as well as the camera vectors 
E and S. 
Unlike the case with the flat horizontal terrain. the solu- 
tion of the above problem is not unique. Indeed, flat tilted 
terrain and vertical camera case is indistinguishable from 
the case when the terrain is horizontal and the camera is 
tilted. In this situation we obtain the camera tile values from 
the sensor measurements. Sensor inaccuracies will then re- 
sult in inaccurate determination of the terrain topography. 
3. PROCESSING ALGORITHM IMPROVEMENTS 
Robust and accurate calculation of relative homographies is 
one of the most important parts of the mosaicing process. 
We are employing optimization technique based on the so 
called brightness constancy constraint. (It should be noted 
that depending on visibility and lighting conditions acquired 
images may need some pre-processing - filtering [4. 51 or 
de-trending Ill.) Although consecutive frames typically have 
much in common, successful optimization heavily depends 
on the initial guess used in iterative procedure. Our strat- 
egy is to set certain threshold for the average per-pixel error 
(optimization is considered to be successful if the final error 
is below this threshold) and to run optimization for several 
initial guesses (this stage can be easily parallelized). Two 
common candidates for initial guesses are: (a) successfully 
found transformation for the previous pair of frames. and 
(b) unit transformation. In approximately 2 percent of cases 
these guesses led to a non-global minimum with high resid- 
ual error. However in these cases estimates of camera Euler 
angles obtained from the sensor may be used to provide the 
initial guess for a relative homography. The algorithm is as 
follows: 
1. Using Euler angles for both frames Il and I2 corre- 
sponding approximate rectification homographies Rf 
and &A are calculated. Note that translations D, and 
D, remain zero for these estimates. 
2. Frames are re-projected onto flat horizontal plane us- 
ing these homographies: 
3. Featureless frequency domain-based technique [6, 11 
is used to estimate rigid affine transformation between 
I - 869 
re-projected images: 
Note that this method is non-iterative and highly 10- 
bust. 
4. Initial guess for the relative homography Tipt is ob- 
tained from the above transformations: 
From our experience, this technique, which is computation- 
ally intensive. provides optimal choice of initial guess for 
optimization procedure. 
Some failures in finding relative homography were as- 
sociated with non-stationary content of the frames - objects 
moving across the camera field of view. The technique 
outlined above was stili able to provide a reasonable ini- 
tial guess provided the moving objects did not occupy more 
than 30 percent of the frame. In this case, however. co- 
registration results in average per-pixel error being signifi- 
cantly higher than the pre-set threshold, and each situation 
requires manual intervention. Once it was established that 
co-registration procedure did not fail, the regions with high 
local pixel error were marked and in creation of the final 
mosaic only one instance of input video data was used. to 
decrease blurring effects. 
4. APPARATUS AND RESULTS 
A consumer grade Sony digital video camera is connected to 
a laptop computer by a Control-L device, allowing constant 
monitoring of timecode (which uniquely identifies recorded 
frame) more than 30 times per second. The NMEA out- 
put messages from the attitude sensor and GPS receiver are 
received on serial inputs of the computer. typically com- 
ing 5-8 times per second. The monitoring program logs ev- 
ery sensor message, synchronously with the timecode, GPS 
message and internal CPU time. Records from the fog file 
are used in post-processing, where frames corresponding to 
neighboring records are co-registered - this results in a chain 
of relative homographies relating any two frames in a se- 
quence. The equipment were arranged on a cross arm atop 
a pole: the GPS antenna was mounted in the center of the 
arm. the sensor and the camera - at opposite sides of it. 
that were not related to actual sensor orientation. To mini- 
mize their influence, we have smoothed the measurements. 
using sinc weighting function over 1 sec time span. Similar 
smoothing was applied to GPS position measurements. lati- 
tude and longitude readings being smoothed independently, 
Attitude sensor measurements were found to contain spikes 
as mosaiced areas are relatively small, typically less than 
100 meters. 
For the chosen initial frame it is assumed that sensor 
measurements do not contain errors, and that at that mo- 
ment the camera was located in the center of origin. This 
allows calculation of the rectification homography for the 
initial frame and, through the chain of relative transforma- 
tions. rectification homographies RTp for all consecutive 
frames. Employing the simplified model discussed above, 
camera vectors are found for each rectification homography. 
Each pair of camera vectors corresponds to some "model" 
rectification homography R;lode', and difference between 
RTp and R;rOdel indicates the deviation from the flat hori- 
zontal model. 
Figure 1 shows a comparison between the measured value 
of yaw (dotted line) and yaw calculated from the homogra- 
phies (solid line) as functions of the frame number. Mea- 
sured and calculated values differ only in the regions where 
homographies' error Ek =I1 E T p  - R;lodel 11 is significant 
(Figure 2). Frames with the large error are associated with 
images of terrain that cannot be described by the FH model. 
Fig. 1. Yaw of the camera: measured by the sensor and 
calculated. The survey procedure consisted of moving in a 
straight line, and periodic rotation of the pole. 
Local deviations from the flat terrain (with non-flat area 
much less than total area of the frame) do not significantly 
affect the calculated relative homography. but results in a lo- 
cal mismatch between the pixel values of the co-registered 
frames. This local per-pixel luminance difference may be 
processed using any "shape-from-stereo" technique, thus ob- 
taining information about shape of the imaged surface. For 
our purpose, however, it is sufficient to mark these mis- 
match areas at the pre-processing stage, and, when the fi- 
nal mosaic is created. to use a single instance of the input 
video data to fill them in. as opposed to weighted average, 
typically used for "feathering". 
I - 870 
Fig. 2. Error indicating failure of the flat horizontal model. 
5. CONCLUSIONS 
This paper outlines ways to use data from an attitude sen- 
sor attached to a video camera to simplify processing of 
acquired image sequences and to facilitate construction of 
a global mosaic - geo-coded image of the Underwater sur- 
face. The method has been applied to inter-tidal marine en- 
vironments and provides a cost-effective alternative to pre- 
viously reported methods [5]. The technique shows much 
promise for gathering new types of information from the 
seabed by using video imaging as opposed to traditional 
acoustic imaging. 
Appendix 
Let the Euler vector l? = (e,', $), and denote cr I cos r, ST % 
sin r ,  where r = 8-4, $. It is customary to write homogra- 
phy T as: 
.=(;E PO P l  .;.) P2 
where homography elements can be expressed as functions 
of camera parameters: 
Fig. 3. Example of video mosaic of intertidal marine envi- 
ronment. 
6. REFERENCES 
[I] Y. Rzhanov, L. M. Linnett, R. Forbes, "Underwater 
mosaicing for seabed mapping." 2000 International 
conference on Image Processing, ICIP2000, Vancou- 
ver, Canada, vol. 1. pp. 224-227, 2000. 
121 Y. Rzhanov, G. R. Cutter. L. Huff, "Sensor-assisted 
video mosaicing for seafloor mapping." 2001 Inter- 
national conference on Image Processing, ICIP2001, 
Thessaloniki, Greece, vol. 2. pp. 411-414.2001. 
[31 V. E. Kostylev. B. J. Todd, G. B. Fader, et al. "Benthic 
habitat mapping on the Scotian Shelf based on multi- 
beam bathymetry, surficial geology and sea floor pho- 
tographs." MARINE ECOLOGY- PROGRESS SE- 
RIES, v. 219. pp. 121-137, 2001. 
"Real-time 
video mosaicking of the ocean floor." IEEE Journal 
of Oceanic Enaineerina. vol. 20, DP. 229-241, 1995. 
141 R. L. Marks, S .  M. Rock, M. J. Lee, 
- I .. 
PO = U [ ( S W S '  - dc')/Z - (DdF)(s@s+c' + c!w+l 51 ", Singh, L, Whitcomb, D, yoerger, 0,  pizarro, ,'Mi- 
p3 = u i ( c w w  - ( D J F ) ( S B S + C ~  + +wi 
p4 = u i ( - c w w  + ( ~ , / ~ ) ( s ~ n l c +  - S$S+)I 
= 
- s'c')'z + (Dz/F)(soNc' - sd's4)1 crobathymetric mapping from underwater vehicles in 
the deep ocean." Computer Vision and Image Under- 
standing. vol. 79, pp. 143-161,2000. 
PZ = U[F(c@s$)/z  -DZ(cW)] 
ps = U[F(-s@)/Z - D,(cBc$)] 161 B. S. Reddy and B. N. Chatterji, "An FFT-based tech- 
ptj = -U(SBS+C~ +@sr$)/F nique for translation. rotation and scale-invariant im- 
P'I = U ( S W J C $  - s$s4)/F age registration." IEEE Transactions on Image Pro- 
cessing, vol. 5. pp. 1266-1271, 1996. 
where U = -cb'cqi 
I - 871 


Seafloor Video Mapping: Modeling, Algorithms, Apparatus

UNH Scholars' Repository

University of New HampshireUniversity of New Hampshire Scholars' RepositoryCenter for Coastal and Ocean Mapping Center for Coastal and Ocean Mapping9-2002Seafloor Video Mapping: Modeling, Algorithms,ApparatusYuri RzhanovUniversity of New Hampshire, Durham, Yuri.Rzhanov@unh.eduLloyd C. HuffUniversity of New Hampshire, DurhamRandy G. Cutter Jr.University of New Hampshire, DurhamFollow this and additional works at: https://scholars.unh.edu/ccomPart of the Computer Sciences Commons, and the Oceanography and Atmospheric Sciences andMeteorology CommonsThis Conference Proceeding is brought to you for free and open access by the Center for Coastal and Ocean Mapping at University of New HampshireScholars' Repository. It has been accepted for inclusion in Center for Coastal and Ocean Mapping by an authorized administrator of University of NewHampshire Scholars' Repository. For more information, please contact nicole.hentz@unh.edu.Recommended CitationRzhanov, Yuri; Huff, Lloyd C.; and Cutter, Randy G. Jr., "Seafloor Video Mapping: Modeling, Algorithms, Apparatus" (2002).International Conference on Image Processing (ICIP). 255.https://scholars.unh.edu/ccom/255SEAFLOOR VIDEO MAPPING: MODELING, ALGORITHMS, APPARATUS Y Rzhanov, L. HUE G. R. Cutter Center for Coastal and Ocean Mapping (C-COM) University of New Hampshire Durham 03824, USA email: yuri.rzhanov@unh.edu, lhuff@cisunix.unh.edu, gcutter@cisunix.unh.edu ABSTRACT This paper discusses technique used for construction of high- resolution image mosaic from a video sequence and the syn- chronously logged camera attitude information. It allows one to infer geometric characteristics of the imaged terrain and hence improve mosaic quality and reduce computational burden. The technique is demonstrated using numerical mod- eling and is applied to video data collected on Rainsford Island, Mass. Calculation of the transformation relating consecutive image frames is an essential operation affecting reliability of the whole mosaicing process. Improvements to the algo- rithm are suggested. which significantly decrease the possi- bility of convergence to an inappropriate solution. 1. INTRODUCTION In the previous publications 11. 21 we have discussed po- tential use of auxiliary sensors (namely, attitude sensor and GPS receiver) for improvement of quality of image mosaics, in particular, for underwater imagery. These mosaics have important applications for visual survey of large areas or objects under the conditions of poor visibility. Our primary interest lies in area of benthic habitat mapping (e.g. [31). although pipeline inspection, sediment characterization. ar- chaeological and forensic site mapping could also be po- tential applications of this technique. lo name a few. Cur- rently video mosaicing is the only mapping method which provides resolution better than 1 cm per pixel and rapid con- tinuous coverage of large areas. It seems obvious that accurate measurement of camera position and orientation can greatly facilitate the construc- tion process of a synthetic mosaiced image. The difficulty however is that existing low-cost sensors are not accurate enough to be directly used in the mosaicing process. Indeed. a GPS receiver may provide information about location of the antenna with an accuracy of 10-20 cm. If the camera is underwater and antenna is on the surface, the accuracy is re- duced to 1-2 meters. yet the typical resolution of imagery is less than 1 cm. Even reasonably accurate measurements of the camera Euler angles are not sufficient to position a pixel to within 1 cm. For example, 0.5 degree error in tilt mea- surement easily translates into 2.5 cm error for 30 degree camera pitch (for typical imaging distances). Assumption of flat horizontal (FH) terrain allows to de- rive unique functional relationship between position and ori- entation of the camera and elements of transformation (ho- mography) used to create a mosaiced image. If terrain is indeed described well by the FH model, resulting mosaic is of good quality and camera motion can be recovered: dif- ference between homographies calculated from the camera vectors and ones obtained from image co-registration pro- cedures stays small. This difference going up indicates that the FH model becomes inapplicable. More sophisticated models, however, cannot solve for camera motion and ter- rain characteristics simultaneously - solution is not unique. By employing sensor measurements we eliminate ambigu- ity and infer some local topographic characteristics of the terrain. 2. MODELING OF THE ACQUISITION AND RE-PROJECTION PROCESSES To understand the mosaicing process in details, a model of video acquisition was developed. The modelled imaged sur- face is described in terms of digital elevation model (DEM) in association with a raster image, draped over the topogra- phy. Both elevation and luminance (or color) values are de- fined on a regular grid and are assumed to change monoton- ically in between. The DEM is with respect to a "zero level plane", which is normal to Y axis of the associated system of coordinates and passes through point Y = H. Cam- era orientation and position is dey ibed  by 6 parameters. combined in an "Euler vector", E, components of which represect pitch. roll and yaw of the camera, and translation vectors determining offset with respect to the center of ori- gin, 0. In case of zero Euler vector. ideal pinhole camera is oriented along axis X ("North") and is looking at the ter- rain vertically down (along -Y direction). "West" direction 0-7803-7622-6/02/$17.00 02002 IEEE 1 - 868 IEEE ICIP 2002 then corresponds to Z axis. The acquired image (frame) lies in XZ-plane, "up" direction being "North". and "left" being "West". (The described system is widely used in graphic display field.) We also need to introduce number of pixels in the frame, Nx (columns) and Ny (rows), and camera field of view in one of the directions. say. horizontal. FoV. Then the normalized focal distance. F, is equal to z tan~ov ,z .  In case of zero Euler vector, flat horizontal terrain and H = F, focal plane coincides with the imaged surface, and the mapping between grid points of the terrain and image pixels is determined by a unit homography. Decrease in H results in zooming in the imaged surface, so we can intro- duce zoom ratio Z = 6 .  Shift of the camera in the plane parallel to the terrain leads to the corresponding shift of the imaged area: shifts are convenient to normalize by num- ber of pixels in the horizontal direction (two images shifted horizontally with 50 percent overlap have normalized shift 0, = 0.5). If the terrain is described by the FH model, geometric considerations provide a unique relationship between cam- era parameters (E, Z, D,, Dy) and coefficients of the rec- tification homography R, Le. transformation between the coordinates of terrain W and pixels of the frame I (see Ap- pendix). Re-projection process - mapping of pixels onto the terrain space - can be denoted symbolically as W = R x I. (Note that the normalized shifts in image space D, and Du correspond to shifts in 2- and X- directions in the real space.) Two frames IO and I1 imaging approximately the same area could be co-registered, that is, a homography To1 could ,' be found, mapping frame Il onto the image space of 11, thus combining them in a mosaic. Denote this symbolically as IO = To1 x Il. If rectification (world) homography is known for frame IO, then the world homography for frame 4 isR1 = &xTol ,  as W = R O x  IO = ( & x T o l ) x h  = R1 x I,. Other frames, related to IO through a chain of rel- ative homographies, Tj,j+l, could be added to the global mosaic in the same way: W = & x Tj,j+l x I k .  The latter formula shows that if the rectification homography for the initial frame, &,was estimated inaccurately, this would affect global mosaic as a whole, while numerical errors ac- cumulated in co-registration procedures of finding relative homographies, Tj,j+l, may lead to local distortions of the mosaic. If residual errors are negligible, rectification homogra- phies for frames in the acquired sequence can be directly used for construction of the global mosaic. Relative homo- graphies guarantee optimal painvise merging of the frames - rectification homographies guarantee optimal merging of all frames on a common image space (terrain map). Modeling of video acquisition process with non-flat ter- k-1  j = O  rain shows that the above statements do not hold true. Rel- ative homographies found using one of the co-registration procedures (typically, based on either feature tracking or optimization) may not correspond to any set of vectors de- scribing the camera, as the model, used to relate these pa- rameters to the elements of homography, fails. We suggest a more sophisticated model, which describes the terrain el- evation in terms of 2D low-order polynomials, with coeffi- cients b$ng model variables, as well as the camera vectors E and S. Unlike the case with the flat horizontal terrain. the solu- tion of the above problem is not unique. Indeed, flat tilted terrain and vertical camera case is indistinguishable from the case when the terrain is horizontal and the camera is tilted. In this situation we obtain the camera tile values from the sensor measurements. Sensor inaccuracies will then re- sult in inaccurate determination of the terrain topography. 3. PROCESSING ALGORITHM IMPROVEMENTS Robust and accurate calculation of relative homographies is one of the most important parts of the mosaicing process. We are employing optimization technique based on the so called brightness constancy constraint. (It should be noted that depending on visibility and lighting conditions acquired images may need some pre-processing - filtering [4. 51 or de-trending Ill.) Although consecutive frames typically have much in common, successful optimization heavily depends on the initial guess used in iterative procedure. Our strat- egy is to set certain threshold for the average per-pixel error (optimization is considered to be successful if the final error is below this threshold) and to run optimization for several initial guesses (this stage can be easily parallelized). Two common candidates for initial guesses are: (a) successfully found transformation for the previous pair of frames. and (b) unit transformation. In approximately 2 percent of cases these guesses led to a non-global minimum with high resid- ual error. However in these cases estimates of camera Euler angles obtained from the sensor may be used to provide the initial guess for a relative homography. The algorithm is as follows: 1. Using Euler angles for both frames Il and I2 corre- sponding approximate rectification homographies Rf and &A are calculated. Note that translations D, and D, remain zero for these estimates. 2. Frames are re-projected onto flat horizontal plane us- ing these homographies: 3. Featureless frequency domain-based technique [6, 11 is used to estimate rigid affine transformation between I - 869 re-projected images: Note that this method is non-iterative and highly 10- bust. 4. Initial guess for the relative homography Tipt is ob- tained from the above transformations: From our experience, this technique, which is computation- ally intensive. provides optimal choice of initial guess for optimization procedure. Some failures in finding relative homography were as- sociated with non-stationary content of the frames - objects moving across the camera field of view. The technique outlined above was stili able to provide a reasonable ini- tial guess provided the moving objects did not occupy more than 30 percent of the frame. In this case, however. co- registration results in average per-pixel error being signifi- cantly higher than the pre-set threshold, and each situation requires manual intervention. Once it was established that co-registration procedure did not fail, the regions with high local pixel error were marked and in creation of the final mosaic only one instance of input video data was used. to decrease blurring effects. 4. APPARATUS AND RESULTS A consumer grade Sony digital video camera is connected to a laptop computer by a Control-L device, allowing constant monitoring of timecode (which uniquely identifies recorded frame) more than 30 times per second. The NMEA out- put messages from the attitude sensor and GPS receiver are received on serial inputs of the computer. typically com- ing 5-8 times per second. The monitoring program logs ev- ery sensor message, synchronously with the timecode, GPS message and internal CPU time. Records from the fog file are used in post-processing, where frames corresponding to neighboring records are co-registered - this results in a chain of relative homographies relating any two frames in a se- quence. The equipment were arranged on a cross arm atop a pole: the GPS antenna was mounted in the center of the arm. the sensor and the camera - at opposite sides of it. that were not related to actual sensor orientation. To mini- mize their influence, we have smoothed the measurements. using sinc weighting function over 1 sec time span. Similar smoothing was applied to GPS position measurements. lati- tude and longitude readings being smoothed independently, Attitude sensor measurements were found to contain spikes as mosaiced areas are relatively small, typically less than 100 meters. For the chosen initial frame it is assumed that sensor measurements do not contain errors, and that at that mo- ment the camera was located in the center of origin. This allows calculation of the rectification homography for the initial frame and, through the chain of relative transforma- tions. rectification homographies RTp for all consecutive frames. Employing the simplified model discussed above, camera vectors are found for each rectification homography. Each pair of camera vectors corresponds to some "model" rectification homography R;lode', and difference between RTp and R;rOdel indicates the deviation from the flat hori- zontal model. Figure 1 shows a comparison between the measured value of yaw (dotted line) and yaw calculated from the homogra- phies (solid line) as functions of the frame number. Mea- sured and calculated values differ only in the regions where homographies' error Ek =I1 E T p  - R;lodel 11 is significant (Figure 2). Frames with the large error are associated with images of terrain that cannot be described by the FH model. Fig. 1. Yaw of the camera: measured by the sensor and calculated. The survey procedure consisted of moving in a straight line, and periodic rotation of the pole. Local deviations from the flat terrain (with non-flat area much less than total area of the frame) do not significantly affect the calculated relative homography. but results in a lo- cal mismatch between the pixel values of the co-registered frames. This local per-pixel luminance difference may be processed using any "shape-from-stereo" technique, thus ob- taining information about shape of the imaged surface. For our purpose, however, it is sufficient to mark these mis- match areas at the pre-processing stage, and, when the fi- nal mosaic is created. to use a single instance of the input video data to fill them in. as opposed to weighted average, typically used for "feathering". I - 870 Fig. 2. Error indicating failure of the flat horizontal model. 5. CONCLUSIONS This paper outlines ways to use data from an attitude sen- sor attached to a video camera to simplify processing of acquired image sequences and to facilitate construction of a global mosaic - geo-coded image of the Underwater sur- face. The method has been applied to inter-tidal marine en- vironments and provides a cost-effective alternative to pre- viously reported methods [5]. The technique shows much promise for gathering new types of information from the seabed by using video imaging as opposed to traditional acoustic imaging. Appendix Let the Euler vector l? = (e,', $), and denote cr I cos r, ST % sin r ,  where r = 8-4, $. It is customary to write homogra- phy T as: .=(;E PO P l  .;.) P2 where homography elements can be expressed as functions of camera parameters: Fig. 3. Example of video mosaic of intertidal marine envi- ronment. 6. REFERENCES [I] Y. Rzhanov, L. M. Linnett, R. Forbes, "Underwater mosaicing for seabed mapping." 2000 International conference on Image Processing, ICIP2000, Vancou- ver, Canada, vol. 1. pp. 224-227, 2000. 121 Y. Rzhanov, G. R. Cutter. L. Huff, "Sensor-assisted video mosaicing for seafloor mapping." 2001 Inter- national conference on Image Processing, ICIP2001, Thessaloniki, Greece, vol. 2. pp. 411-414.2001. [31 V. E. Kostylev. B. J. Todd, G. B. Fader, et al. "Benthic habitat mapping on the Scotian Shelf based on multi- beam bathymetry, surficial geology and sea floor pho- tographs." MARINE ECOLOGY- PROGRESS SE- RIES, v. 219. pp. 121-137, 2001. "Real-time video mosaicking of the ocean floor." IEEE Journal of Oceanic Enaineerina. vol. 20, DP. 229-241, 1995. 141 R. L. Marks, S .  M. Rock, M. J. Lee, - I .. PO = U [ ( S W S '  - dc')/Z - (DdF)(s@s+c' + c!w+l 51 ", Singh, L, Whitcomb, D, yoerger, 0,  pizarro, ,'Mi- p3 = u i ( c w w  - ( D J F ) ( S B S + C ~  + +wi p4 = u i ( - c w w  + ( ~ , / ~ ) ( s ~ n l c +  - S$S+)I = - s'c')'z + (Dz/F)(soNc' - sd's4)1 crobathymetric mapping from underwater vehicles in the deep ocean." Computer Vision and Image Under- standing. vol. 79, pp. 143-161,2000. PZ = U[F(c@s$)/z  -DZ(cW)] ps = U[F(-s@)/Z - D,(cBc$)] 161 B. S. Reddy and B. N. Chatterji, "An FFT-based tech- ptj = -U(SBS+C~ +@sr$)/F nique for translation. rotation and scale-invariant im- P'I = U ( S W J C $  - s$s4)/F age registration." IEEE Transactions on Image Pro- cessing, vol. 5. pp. 1266-1271, 1996. where U = -cb'cqi I - 871 

https://scholars.unh.edu/cgi/viewcontent.cgi?article=1254&amp;context=ccom

Seafloor Video Mapping: Modeling, Algorithms, Apparatus

Abstract

Similar works

Full text

Available Versions

University of New Hampshire

UNH Scholars' Repository