Efficient Surface-Aware Semi-Global Matching with Multi-View Plane-Sweep Sampling

Online augmentation of an oblique aerial image sequence with structural information is an essential aspect in the process of 3D scene interpretation and analysis. One key aspect in this is the efficient dense image matching and depth estimation. Here, the Semi-Global Matching (SGM) approach has proven to be one of the most widely used algorithms for efficient depth estimation, providing a good trade-off between accuracy and computational complexity. However, SGM only models a first-order smoothness assumption, thus favoring fronto-parallel surfaces. In this work, we present a hierarchical algorithm that allows for efficient depth and normal map estimation together with confidence measures for each estimate. Our algorithm relies on a plane-sweep multi-image matching followed by an extended SGM optimization that allows to incorporate local surface orientations, thus achieving more consistent and accurate estimates in areasmade up of slanted surfaces, inherent to oblique aerial imagery. We evaluate numerous configurations of our algorithm on two different datasets using an absolute and relative accuracy measure. In our evaluation, we show that the results of our approach are comparable to the ones achieved by refined Structure-from-Motion (SfM) pipelines, such as COLMAP, which are designed for offline processing. In contrast, however, our approach only considers a confined image bundle of an input sequence, thus allowing to perform an online and incremental computation at 1Hz–2Hz

Enhancment of dense urban digital surface models from VHR optical satellite stereo data by pre-segmentation and object detection

The generation of digital surface models (DSM) of urban areas from very high resolution (VHR) stereo satellite imagery requires advanced methods. In the classical approach of DSM generation from stereo satellite imagery, interest points are extracted and correlated between the stereo mates using an area based matching followed by a least-squares sub-pixel refinement step. After a region growing the 3D point list is triangulated to the resulting DSM. In urban areas this approach fails due to the size of the correlation window, which smoothes out the usual steep edges of buildings. Also missing correlations as for partly – in one or both of the images – occluded areas will simply be interpolated in the triangulation step. So an urban DSM generated with the classical approach results in a very smooth DSM with missing steep walls, narrow streets and courtyards. To overcome these problems algorithms from computer vision are introduced and adopted to satellite imagery. These algorithms do not work using local optimisation like the area-based matching but try to optimize a (semi-)global cost function. Analysis shows that dynamic programming approaches based on epipolar images like dynamic line warping or semiglobal matching yield the best results according to accuracy and processing time. These algorithms can also detect occlusions – areas not visible in one or both of the stereo images. Beside these also the time and memory consuming step of handling and triangulating large point lists can be omitted due to the direct operation on epipolar images and direct generation of a so called disparity image fitting exactly on the first of the stereo images. This disparity image – representing already a sort of a dense DSM – contains the distances measured in pixels in the epipolar direction (or a no-data value for a detected occlusion) for each pixel in the image. Despite the global optimization of the cost function many outliers, mismatches and erroneously detected occlusions remain, especially if only one stereo pair is available. To enhance these dense DSM – the disparity image – a pre-segmentation approach is presented in this paper. Since the disparity image is fitting exactly on the first of the two stereo partners (beforehand transformed to epipolar geometry) a direct correlation between image pixels and derived heights (the disparities) exist. This feature of the disparity image is exploited to integrate additional knowledge from the image into the DSM. This is done by segmenting the stereo image, transferring the segmentation information to the DSM and performing a statistical analysis on each of the created DSM segments. Based on this analysis and spectral information a coarse object detection and classification can be performed and in turn the DSM can be enhanced. After the description of the proposed method some results are shown and discussed

Stereo matching algorithm by propagation of correspondences and stereo vision instrumentation

A new image processing method is described for measuring the 3-D coordinates of a complex, biological surface. One of the problems in stereo vision is known as the accuracy-precision tradeoff problem. This thesis proposes a new method that promises to solve this problem. To do so, two issues are addressed. First, stereo vision instrumentation methods are described. This instrumentation includes a camera system as well as camera calibration, rectification, matching and triangulation. Second, the approach employs an array of cameras that allow accurate computation of the depth map of a surface by propagation of correspondences through pair-wise camera views. The new method proposed in this thesis employs an array of cameras, and preserves the small baseline advantage by finding accurate correspondences in pairs of adjacent cameras. These correspondences are then propagated along the consecutive pairs of cameras in the array until a large baseline is accomplished. The resulting large baseline disparities are then used for triangulation to achieve advantage of precision in depth measurement. The matching is done by an area-based intensity correlation function called Sum of Squared Differences (SSD). In this thesis, the feasibility of using these data for further processing to achieve surface or volume measurements in the future is discussed

Performance and analysis of feature tracking approaches in laser speckle instrumentation

This paper investigates the application of feature tracking algorithms as an alternative data processing method for laser speckle instrumentation. The approach is capable of determining both the speckle pattern translation and rotation and can therefore be used to detect the in-plane rotation and translation of an object simultaneously. A performance assessment of widely used feature detection and matching algorithms from the computer vision field, for both translation and rotation measurements from laser speckle patterns, is presented. The accuracy of translation measurements using the feature tracking approach was found to be similar to that of correlation-based processing with accuracies of 0.025–0.04 pixels and a typical precision of 0.02–0.09 pixels depending upon the method and image size used. The performance for in-plane rotation measurements are also presented with rotation measurement accuracies of <0.01 found to be achievable over an angle range of ±10 and of <0.1 over a range of ±25 ±25 , with a typical precision between 0.02 and 0.08 depending upon method and image size. The measurement range is found to be limited by the failure to match sufficient speckles at larger rotation angles. An analysis of each stage of the process was conducted to identify the most suitable approaches for use with laser speckle images and areas requiring further improvement. A quantitative approach to assessing different feature tracking methods is described, and reference data sets of experimentally translated and rotated speckle patterns from a range of surface finishes and surface roughness are presented. As a result, three areas that lead to the failure of the matching process are identified as areas for future investigation: the inability to detect the same features in partially decorrelated images leading to unmatchable features, the variance of computed feature orientation between frames leading to different descriptors being calculated for the same feature, and the failure of the matching processes due to the inability to discriminate between different features in speckle images

Image Segmentation, Registration, Compression, and Matching

A novel computational framework was developed of a 2D affine invariant matching exploiting a parameter space. Named as affine invariant parameter space (AIPS), the technique can be applied to many image-processing and computer-vision problems, including image registration, template matching, and object tracking from image sequence. The AIPS is formed by the parameters in an affine combination of a set of feature points in the image plane. In cases where the entire image can be assumed to have undergone a single affine transformation, the new AIPS match metric and matching framework becomes very effective (compared with the state-of-the-art methods at the time of this reporting). No knowledge about scaling or any other transformation parameters need to be known a priori to apply the AIPS framework. An automated suite of software tools has been created to provide accurate image segmentation (for data cleaning) and high-quality 2D image and 3D surface registration (for fusing multi-resolution terrain, image, and map data). These tools are capable of supporting existing GIS toolkits already in the marketplace, and will also be usable in a stand-alone fashion. The toolkit applies novel algorithmic approaches for image segmentation, feature extraction, and registration of 2D imagery and 3D surface data, which supports first-pass, batched, fully automatic feature extraction (for segmentation), and registration. A hierarchical and adaptive approach is taken for achieving automatic feature extraction, segmentation, and registration. Surface registration is the process of aligning two (or more) data sets to a common coordinate system, during which the transformation between their different coordinate systems is determined. Also developed here are a novel, volumetric surface modeling and compression technique that provide both quality-guaranteed mesh surface approximations and compaction of the model sizes by efficiently coding the geometry and connectivity/topology components of the generated models. The highly efficient triangular mesh compression compacts the connectivity information at the rate of 1.5-4 bits per vertex (on average for triangle meshes), while reducing the 3D geometry by 40-50 percent. Finally, taking into consideration the characteristics of 3D terrain data, and using the innovative, regularized binary decomposition mesh modeling, a multistage, pattern-drive modeling, and compression technique has been developed to provide an effective framework for compressing digital elevation model (DEM) surfaces, high-resolution aerial imagery, and other types of NASA data

Assessment of a photogrammetric approach for urban DSM extraction from tri-stereoscopic satellite imagery

Built-up environments are extremely complex for 3D surface modelling purposes. The main distortions that hamper 3D reconstruction from 2D imagery are image dissimilarities, concealed areas, shadows, height discontinuities and discrepancies between smooth terrain and man-made features. A methodology is proposed to improve automatic photogrammetric extraction of an urban surface model from high resolution satellite imagery with the emphasis on strategies to reduce the effects of the cited distortions and to make image matching more robust. Instead of a standard stereoscopic approach, a digital surface model is derived from tri-stereoscopic satellite imagery. This is based on an extensive multi-image matching strategy that fully benefits from the geometric and radiometric information contained in the three images. The bundled triplet consists of an IKONOS along-track pair and an additional near-nadir IKONOS image. For the tri-stereoscopic study a densely built-up area, extending from the centre of Istanbul to the urban fringe, is selected. The accuracy of the model extracted from the IKONOS triplet, as well as the model extracted from only the along-track stereopair, are assessed by comparison with 3D check points and 3D building vector data

MAMUD : contribution of HR satellite imagery to a better monitoring, modeling and understanding of urban dynamics

In this treatise the discussion of a methodology and results of semi-automatic city DSM extrac-tion from an Ikonos triplet, is introduced. Built-up areas are known as being complex for photogrammetric purposes, partly because of the steep changes in elevation caused by buildings and urban features. To make DSM extraction more robust and to cope with the specific problems of height displacement, concealed areas and shadow, a multi-image based approach is followed. For the VHR tri-stereoscopic study an area extending from the centre of Istanbul to the urban fringe is chosen. Research will concentrate, in first phase on the development of methods to optimize the extraction of photogrammetric products from the bundled Ikonos triplet. Optimal methods need to be found to improve the radiometry and geometry of the imagery, to improve the semi-automatically derivation of DSM’s and to improve the postprocessing of the products. Secondly we will also investigate the possibilities of creating stereo models out of images from the same sensor taken on a different date, e.g. one image of the stereo pair combined with the third image. Finally the photogrammetric products derived from the Ikonos stereo pair as well as the products created out of the triplet and the constructed stereo models will be investigated by comparison with a 3D reference. This evaluation should show the increase of accuracy when multi-imagery is used instead of stereo pairs

Processing Elastic Surfaces and Related Gradient Flows

Surface processing tools and techniques have a long history in the fields of computer graphics, computer aided geometric design and engineering. In this thesis we consider variational methods and geometric evolution problems for various surface processing applications including surface fairing, surface restoration and surface matching. Geometric evolution problems are often based on the gradient flow of geometric energies. The Willmore functional, defined as the integral of the squared mean curvature over the surface, is a geometric energy that measures the deviation of a surface from a sphere. Therefore, it is a suitable functional for surface restoration, where a destroyed surface patch is replaced by a smooth patch defined as the minimizer of the Willmore functional with boundary conditions for the position and the normal at the patch boundary. However, using the Willmore functional does not lead to satisfying results if an edge or a corner of the surface is destroyed. The anisotropic Willmore energy is a natural generalization of the Willmore energy which has crystal-shaped surfaces like cubes or octahedra as minimizers. The corresponding L2-gradient flow, the anisotropic Willmore flow, leads to a fourth-order partial differential equation that can be written as a system of two coupled second second order equations. Using linear Finite Elements, we develop a semi-implicit scheme for the anisotropic Willmore flow with boundary conditions. This approach suffer from significant restrictions on the time step size. Effectively, one usually has to enforce time steps smaller than the squared spatial grid size. Based on a natural approach for the time discretization of gradient flows we present a new scheme for the time and space discretization of the isotropic and anisotropic Willmore flow. The approach is variational and takes into account an approximation of the L2-distance between the surface at the current time step and the unknown surface at the new time step as well as a fully implicity approximation of the anisotropic Willmore functional at the new time step. To evaluate the anisotropic Willmore energy on the unknown surface of the next time step, we first ask for the solution of an inner, secondary variational problem describing a time step of anisotropic mean curvature motion. The time discrete velocity deduced from the solution of the latter problem is regarded as an approximation of the anisotropic mean curvature vector and enters the approximation of the actual anisotropic Willmore functional. The resulting two step time discretization of the Willmore flow is applied to polygonal curves and triangular surfaces and is independent of the co-dimension. Various numerical examples underline the stability of the new scheme, which enables time steps of the order of the spatial grid size. The Willmore functional of a surface is referred to as the elastic surface energy. Another interesting application of modeling elastic surfaces as minimizers of elastic energies is surface matching, where a correspondence between two surfaces is subject of investigation. There, we seek a mapping between two surfaces respecting certain properties of the surfaces. The approach is variational and based on well-established matching methods from image processing in the parameter domains of the surfaces instead of finding a correspondence between the two surfaces directly in 3D. Besides the appropriate modeling we analyze the derived model theoretically. The resulting deformations are globally smooth, one-to-one mappings. A physically proper morphing of characters in computer graphic is capable with the resulting computational approach