문서 경계와 3차원 재구성에 기반한 문서 이미지 평판화

학위논문(석사) -- 서울대학교대학원 : 자연과학대학 수리과학부, 2022. 8. 현동훈.In recent days, most of the scanned images are obtained from mobile devices such as cameras, smartphones, and tablets rather than traditional flatbed scanners. Contrary to the scanning process of the traditional scanners, capturing process of mobile devices might be accompanied by distortions in various forms such as perspective distortion, fold distortion, and page curls. In this thesis, we propose robust dewarping methods which correct such distortions based on the document boundary and 3D reconstruction. In the first method, we construct a curvilinear grid on the document image using the document boundary and reconstruct the document surface in the three dimensional space. Then we rectify the image using a family of local homographies computed from the reconstructed document surface. Although some of the steps of the proposed method have been proposed separately in other research, our approach exploited and combined their advantages to propose a robust dewarping process in addition to improving the stability in the overall process. Moreover, we refined the process by correcting the distorted text region boundary and developed this process into an independent dewarping method which is concise, straight-forward, and robust while still producing a well-rectified document image.최근에는 대부분의 스캔된 이미지들이 전통적인 평판스캐너가 아닌 카메라, 스마트폰, 태블릿 PC 등의 휴대기기들로부터 얻어진다. 이전 스캐너들의 스캐닝 과정과는 다르게 휴대기기들을 이용한 이미지 캡쳐링 과정은 원근왜곡, 종이의 접힘으로 인한 왜곡, 그리고 종이의 휘어짐으로 인한 왜곡 등 다양한 왜곡들을 수반할 수 있다. 이 논문에서는 이러한 왜곡들을 제거할 수 있는 문서 경계와 3차원 재구성에 기반한 강력한 디워핑 방법을 제안하고자 한다. 첫번째 방법에서는, 문서 경계를 이용하여 문서 이미지 위에 곡선으로 이루어진 그리드를 만들고, 3차원 공간 상의 문서 곡면을 재구성한다. 그리고 재구성된 문서 곡면으로부터 계산된 국소적 호모그래피들을 이용하여 이미지를 수정한다. 우리가 제안하는 방법의 몇몇 단계는 다른 연구에서 개별적으로 사용된 경우도 있지만, 우리는 전체적인 과정에서 안정성을 높이는 동시에 각 방법의 장점들을 이용하고 조합하여 강력한 디워핑 방법을 제안한다. 이에 더하여, 우리는 왜곡된 텍스트 영역의 경계를 수정하여 전체적인 과정을 보완하였고, 이 절차를 간결하고, 직관적이며, 강력하면서도 좋은 결과를 내는 독립적인 디워핑 방법으로 개발하였다.1. Introduction 1 2. Review on Camera Geometry 6 2.1. Basic Camera Model 6 2.2. 3D Reconstruction Problem 8 3. Related Works 10 3.1. Dewarping Methods based on the Text-lines 10 3.2. Dewarping Methods based on the Document Boundary 11 3.3. Dewarping Methods based on the Grid Construction 12 3.4. Dewarping Methods based on the Document Surface Model in 3D Space 13 4. Document Image Dewarping based on the Document Boundary and 3D Reconstruction 15 4.1. Input Document Image Processing 17 4.1.1. Binarization of the Input Document Image 17 4.1.2. Perspective Distortion Removal using the Document Boundary 19 4.2. Grid Construction on the Document Image 21 4.3. 3D Reconstruction of the Document Surface 23 4.3.1. Geometric Model 23 4.3.2. Normalization of the Grid Corners 24 4.3.3. 3D Reconstruction of the Document Surface 26 4.4. Rectification of the Document Image under a Family of Local Homographies 27 4.5. Global Rectification of the Document Image 29 5. Document Image Dewarping by Straightening Document Boundary Curves 33 6. Conclusion 37 Appendix A. 38 A.1. 4-point Algorithm 38 A.2. Optimization of the Cost Function 40 Bibliography 42 Abstract (in Korean) 47 Acknowledgement (in Korean) 48석

From Damage to Discovery Via Virtual Unwrapping: Reading the Scroll from En-Gedi

Computer imaging techniques are commonly used to preserve and share readable manuscripts, but capturing writing locked away in ancient, deteriorated documents poses an entirely different challenge. This software pipeline—referred to as “virtual unwrapping”—allows textual artifacts to be read completely and noninvasively. The systematic digital analysis of the extremely fragile En-Gedi scroll (the oldest Pentateuchal scroll in Hebrew outside of the Dead Sea Scrolls) reveals the writing hidden on its untouchable, disintegrating sheets. Our approach for recovering substantial ink-based text from a damaged object results in readable columns at such high quality that serious critical textual analysis can occur. Hence, this work creates a new pathway for subsequent textual discoveries buried within the confines of damaged materials

Computational grid generation for the design of free-form shells with complex boundary conditions

Free-form grid structures have been widely used in various public buildings, and many are bounded by complex curves including internal voids. Modern computational design software enables the rapid creation and exploration of such complex surface geometries for architectural design, but the resulting shapes lack an obvious way for engineers to create a discrete structural grid to support the surface that manifests the architect's intent. This paper presents an efficient design approach for the synthesis of free-form grid structures based on guideline and surface-flattening methods, which consider complex features and internal boundaries. The method employs a fast and straightforward approach, which achieves fluent lines with bars of balanced length. The parametric domain of a complete nonuniform rational basis spline (NURBS) surface is first divided into a number of patches, and a discrete free-form surface is formed by mapping dividing points onto the surface. The free-form surface is then flattened based on the principle of equal area. Accordingly, the flattened rectangular lattices are then fit to the two-dimensional (2D) surface, with grids formed by applying a guideline method. Subsequently, the intersections of the guidelines and the complex boundary are obtained, and the guidelines are divided equally between boundaries to produce grids connected at the dividing points. Finally, the 2D grids are mapped back onto the three-dimensional (3D) surface and a spring-mass relaxation method is employed to further improve the smoothness of the resulting grids. The paper concludes by presenting realistic examples to demonstrate the practical effectiveness of the proposed method.</p

Nested Explorative Maps: A new 3D canvas for conceptual design in architecture

International audienceIn this digital age, architects still need to alternate between paper sketches and 3D modeling software for their designs. Indeed, while 3D models enable to explore different views, creating them at very early stages might reduce creativity since they do not allow to superpose several tentative designs nor to refine them progressively, as sketches do. To enable exploratory design in 3D, we introduce Nested Explorative Maps, a new system dedicated to interactive design in architecture. Our model enables coarse to fine sketching of nested architectural structures, enabling to progressively sketch a 3D building from floor plan to interior design, thanks to a series of nested maps able to spread in 3D. Each map allows the visual representation of uncertainty as well as the interactive exploration of the alternative, tentative options. We validate the model through a user study conducted with professional architects, enabling us to highlight the potential of Nested Explorative Maps for conceptual design in architecture.En cette ère du numérique, les architectes doivent encore alterner entre le croquis papier et logiciels de modélisation 3D afin de réaliser leurs conceptions. En effet, les modèles 3D permettent d’explorer différentes vues mais leur création à un stade très précoce peut impliquer une perte de la créativité car ils ne permettent pas de superposer plusieurs plans provisoires ni de les affiner progressivement, comme le font les esquisses. Pour permettre la conception exploratoire dans l'espace 3D, nous présentons Nested Explorative Maps, un nouveau système dédié à la conception interactive en architecture. Notre modèle permet de dessiner du grossier aux détails des structures architecturales imbriquées, afin de dessiner progressivement un bâtiment en 3D, du plan à la décoration intérieure, grâce à une série de cartes imbriquées capables de se répandre en 3D. Chaque carte permet de représenter visuellement l’incertitude et d’explorer de manière interactive les différentes options possibles. Une étude utilisateur réalisée auprès d'architectes professionnels nous a permis de valider notre modèle et de mettre en évidence le potentiel des cartes exploratoires imbriquées pour la conception conceptuelle en architecture

Automatic Reconstruction of Textured 3D Models

Three dimensional modeling and visualization of environments is an increasingly important problem. This work addresses the problem of automatic 3D reconstruction and we present a system for unsupervised reconstruction of textured 3D models in the context of modeling indoor environments. We present solutions to all aspects of the modeling process and an integrated system for the automatic creation of large scale 3D models

An investigation on the framework of dressing virtual humans

This thesis was submitted for the degree of Doctor of Philosophy and awarded by Brunel University.Realistic human models are widely used in variety of applications. Much research has been carried out on improving realism of virtual humans from various aspects, such as body shapes, hair, and facial expressions and so on. In most occasions, these virtual humans need to wear garments. However, it is time-consuming and tedious to dress a human model using current software packages [Maya2004]. Several methods for dressing virtual humans have been proposed recently [Bourguignon2001, Turquin2004, Turquin2007 and Wang2003B]. The method proposed by Bourguignon et al [Bourguignon2001] can only generate 3D garment contour instead of 3D surface. The method presented by Turquin et al. [Turquin2004, Turquin2007] could generate various kinds of garments from sketches but their garments followed the shape of the body and the side of a garment looked not convincing because of using simple linear interpolation. The method proposed by Wang et al. [Wang2003B] lacked interactivity from users, so users had very limited control on the garment shape.This thesis proposes a framework for dressing virtual humans to obtain convincing dressing results, which overcomes problems existing in previous papers mentioned above by using nonlinear interpolation, level set-based shape modification, feature constraints and so on. Human models used in this thesis are reconstructed from real human body data obtained using a body scanning system. Semantic information is then extracted from human models to assist in generation of 3 dimensional (3D) garments. The proposed framework allows users to dress virtual humans using garment patterns and sketches. The proposed dressing method is based on semantic virtual humans. A semantic human model is a human body with semantic information represented by certain of structure and body features. The semantic human body is reconstructed from body scanned data from a real human body. After segmenting the human model into six parts some key features are extracted. These key features are used as constraints for garment construction.Simple 3D garment patterns are generated using the techniques of sweep and offset. To dress a virtual human, users just choose a garment pattern, which is put on the human body at the default position with a default size automatically. Users are allowed to change simple parameters to specify some sizes of a garment by sketching the desired position on the human body.To enable users to dress virtual humans by their own design styles in an intuitive way, this thesis proposes an approach for garment generation from user-drawn sketches. Users can directly draw sketches around reconstructed human bodies and then generates 3D garments based on user-drawn strokes. Some techniques for generating 3D garments and dressing virtual humans are proposed. The specific focus of the research lies in generation of 3D geometric garments, garment shape modification, local shape modification, garment surface processing and decoration creation. A sketch-based interface has been developed allowing users to draw garment contour representing the front-view shape of a garment, and the system can generate a 3D geometric garment surface accordingly. To improve realism of a garment surface, this thesis presents three methods as follows. Firstly, the procedure of garment vertices generation takes key body features as constraints. Secondly, an optimisation algorithm is carried out after generation of garment vertices to optimise positions of garment vertices. Finally, some mesh processing schemes are applied to further process the garment surface. Then, an elaborate 3D geometric garment surface can be obtained through this series of processing. Finally, this thesis proposes some modification and editing methods. The user-drawn sketches are processed into spline curves, which allow users to modify the existing garment shape by dragging the control points into desired positions. This makes it easy for users to obtain a more satisfactory garment shape compared with the existing one. Three decoration tools including a 3D pen, a brush and an embroidery tool, are provided letting users decorate the garment surface by adding some small 3D details such as brand names, symbols and so on. The prototype of the framework is developed using Microsoft Visual Studio C++,OpenGL and GPU programming

Resolving Ambiguities in Monocular 3D Reconstruction of Deformable Surfaces

In this thesis, we focus on the problem of recovering 3D shapes of deformable surfaces from a single camera. This problem is known to be ill-posed as for a given 2D input image there exist many 3D shapes that give visually identical projections. We present three methods which make headway towards resolving these ambiguities. We believe that our work represents a significant step towards making surface reconstruction methods of practical use. First, we propose a surface reconstruction method that overcomes the limitations of the state-of-the-art template-based and non-rigid structure from motion methods. We neither track points over many frames, nor require a sophisticated deformation model, or depend on a reference image. In our method, we establish correspondences between pairs of frames in which the shape is different and unknown. We then estimate homographies between corresponding local planar patches in both images. These yield approximate 3D reconstructions of points within each patch up to a scale factor. Since we consider overlapping patches, we can enforce them to be consistent over the whole surface. Finally, a local deformation model is used to fit a triangulated mesh to the 3D point cloud, which makes the reconstruction robust to both noise and outliers in the image data. Second, we propose a novel approach to recovering the 3D shape of a deformable surface from a monocular input by taking advantage of shading information in more generic contexts than conventional Shape-from-Shading (SfS) methods. This includes surfaces that may be fully or partially textured and lit by arbitrarily many light sources. To this end, given a lighting model, we learn the relationship between a shading pattern and the corresponding local surface shape. At run time, we first use this knowledge to recover the shape of surface patches and then enforce spatial consistency between the patches to produce a global 3D shape. Instead of treating texture as noise as in many SfS approaches, we exploit it as an additional source of information. We validate our approach quantitatively and qualitatively using both synthetic and real data. Third, we introduce a constrained latent variable model that inherently accounts for geometric constraints such as inextensibility defined on the mesh model. To this end, we learn a non-linear mapping from the latent space to the output space, which corresponds to vertex positions of a mesh model, such that the generated outputs comply with equality and inequality constraints expressed in terms of the problem variables. Since its output is encouraged to satisfy such constraints inherently, using our model removes the need for computationally expensive methods that enforce these constraints at run time. In addition, our approach is completely generic and could be used in many other different contexts as well, such as image classification to impose separation of the classes, and articulated tracking to constrain the space of possible poses

Processing Camera-captured Document Images: Geometric Rectification, Mosaicing, and Layout Structure Recognition

This dissertation explores three topics: 1) geometric rectification of cameracaptured document images, 2) camera-captured document mosaicing, and 3) layout structure recognition. The first two topics pertain to camera-based document image analysis, a new trend within the OCR community. Compared to typical scanners,cameras offer convenient, flexible, portable, and non-contact image capture, which enables many new applications and breathes new life into existing ones. The third topic is related to the need for efficient metadata extraction methods, critical for managing digitized documents. The kernel of our geometric rectification framework is a novel method for estimating document shape from a single camera-captured image. Our method uses texture flows detected in printed text areas and is insensitive to occlusion. Classification of planar versus curved documents is done automatically. For planar pages, we obtain full metric rectification. For curved pages, we estimate a planar-strip approximation based on properties of developable surfaces. Our method can process any planar or smoothly curved document captured from an arbitrary position without requiring 3D data, metric data, or camera calibration. For the second topic, we design a novel registration method for document images, which produces good results in difficult situations including large displacements, severe projective distortion, small overlapping areas, and lack of distinguishable feature points. We implement a selective image composition method that outperforms conventional image blending methods in overlapping areas. It eliminates double images caused by mis-registration and preserves the sharpness in overlapping areas. We solve the third topic with a graph-based model matching framework. Layout structures are modeled by graphs, which integrate local and global features and are extensible to new features in the future. Our model can handle large variation within a class and subtle differences between classes. Through graph matching, the layout structure of a document is discovered. Our layout structure recognition technique accomplishes document classification and logical component labeling at the same time. Our model learning method enables a model to adapt to changes in classes over time

Optical techniques for 3D surface reconstruction in computer-assisted laparoscopic surgery

One of the main challenges for computer-assisted surgery (CAS) is to determine the intra-opera- tive morphology and motion of soft-tissues. This information is prerequisite to the registration of multi-modal patient-specific data for enhancing the surgeon’s navigation capabilites by observ- ing beyond exposed tissue surfaces and for providing intelligent control of robotic-assisted in- struments. In minimally invasive surgery (MIS), optical techniques are an increasingly attractive approach for in vivo 3D reconstruction of the soft-tissue surface geometry. This paper reviews the state-of-the-art methods for optical intra-operative 3D reconstruction in laparoscopic surgery and discusses the technical challenges and future perspectives towards clinical translation. With the recent paradigm shift of surgical practice towards MIS and new developments in 3D opti- cal imaging, this is a timely discussion about technologies that could facilitate complex CAS procedures in dynamic and deformable anatomical regions