On Martian Surface Exploration: Development of Automated 3D Reconstruction and Super-Resolution Restoration Techniques for Mars Orbital Images

Very high spatial resolution imaging and topographic (3D) data play an important role in modern Mars science research and engineering applications. This work describes a set of image processing and machine learning methods to produce the “best possible” high-resolution and high-quality 3D and imaging products from existing Mars orbital imaging datasets. The research work is described in nine chapters of which seven are based on separate published journal papers. These include a) a hybrid photogrammetric processing chain that combines the advantages of different stereo matching algorithms to compute stereo disparity with optimal completeness, fine-scale details, and minimised matching artefacts; b) image and 3D co-registration methods that correct a target image and/or 3D data to a reference image and/or 3D data to achieve robust cross-instrument multi-resolution 3D and image co-alignment; c) a deep learning network and processing chain to estimate pixel-scale surface topography from single-view imagery that outperforms traditional photogrammetric methods in terms of product quality and processing speed; d) a deep learning-based single-image super-resolution restoration (SRR) method to enhance the quality and effective resolution of Mars orbital imagery; e) a subpixel-scale 3D processing system using a combination of photogrammetric 3D reconstruction, SRR, and photoclinometric 3D refinement; and f) an optimised subpixel-scale 3D processing system using coupled deep learning based single-view SRR and deep learning based 3D estimation to derive the best possible (in terms of visual quality, effective resolution, and accuracy) 3D products out of present epoch Mars orbital images. The resultant 3D imaging products from the above listed new developments are qualitatively and quantitatively evaluated either in comparison with products from the official NASA planetary data system (PDS) and/or ESA planetary science archive (PSA) releases, and/or in comparison with products generated with different open-source systems. Examples of the scientific application of these novel 3D imaging products are discussed

High Relief from Brush Painting

Relief is an art form part way between 3D sculpture and 2D painting. We present a novel approach for generating a texture-mapped high-relief model from a single brush painting. Our aim is to extract the brushstrokes from a painting and generate the individual corresponding relief proxies rather than recovering the exact depth map from the painting, which is a tricky computer vision problem, requiring assumptions that are rarely satisfied. The relief proxies of brushstrokes are then combined together to form a 2.5D high-relief model. To extract brushstrokes from 2D paintings, we apply layer decomposition and stroke segmentation by imposing boundary constraints. The segmented brushstrokes preserve the style of the input painting. By inflation and a displacement map of each brushstroke, the features of brushstrokes are preserved by the resultant high-relief model of the painting. We demonstrate that our approach is able to produce convincing high-reliefs from a variety of paintings(with humans, animals, flowers, etc.). As a secondary application, we show how our brushstroke extraction algorithm could be used for image editing. As a result, our brushstroke extraction algorithm is specifically geared towards paintings with each brushstroke drawn very purposefully, such as Chinese paintings, Rosemailing paintings, etc

Computer Assisted Relief Generation - a Survey

In this paper we present an overview of the achievements accomplished to date in the field of computer aided relief generation. We delineate the problem, classify the different solutions, analyze similarities, investigate the evelopment and review the approaches according to their particular relative strengths and weaknesses. In consequence this survey is likewise addressed to researchers and artists through providing valuable insights into the theory behind the different concepts in this field and augmenting the options available among the methods presented with regard to practical application

Developing serious games for cultural heritage: a state-of-the-art review

Although the widespread use of gaming for leisure purposes has been well documented, the use of games to support cultural heritage purposes, such as historical teaching and learning, or for enhancing museum visits, has been less well considered. The state-of-the-art in serious game technology is identical to that of the state-of-the-art in entertainment games technology. As a result, the field of serious heritage games concerns itself with recent advances in computer games, real-time computer graphics, virtual and augmented reality and artificial intelligence. On the other hand, the main strengths of serious gaming applications may be generalised as being in the areas of communication, visual expression of information, collaboration mechanisms, interactivity and entertainment. In this report, we will focus on the state-of-the-art with respect to the theories, methods and technologies used in serious heritage games. We provide an overview of existing literature of relevance to the domain, discuss the strengths and weaknesses of the described methods and point out unsolved problems and challenges. In addition, several case studies illustrating the application of methods and technologies used in cultural heritage are presented

Retracing the 1910 Carruthers Royal Geographical Society Expedition to the Turgen Mountains of Mongolia – Reconstruction of a Century of Glacial Change

The Turgen Mountains lie in northwestern Mongolia, roughly 80 kilometers south of the Russian border. The area was visited in 1910 by a Royal Geographical Society (RGS) expedition led by Douglas Carruthers. They undertook an extensive survey of the range and produced a detailed topographic map. They also documented the extent of the glaciers with photographs. This modern study consisted of three phases. The first step was to procure the historical documents from the RGS in London, including copies of the photos, journal entries, and the map. Field work in Mongolia entailed traveling to the remote study site and retracing portions the 1910 expedition. Camera locations were matched to the historical photographs and repeat images taken. In addition, the termini of the two main glacial lobes were surveyed by GPS. Finally, spatial analysis was conducted in the computer laboratory using a GIS to generate a „historic‟ elevation model from the 1910 map and compare it to a modern DEM generated from SRTM data. Map analysis software was employed to evaluate cartometric accuracy of the 1910 map against modern Russian topographic sheets. The results of the DEM and map analysis were then validated using the field GPS data and remotely sensed imagery to quantitatively describe the changes in the glacial system. The repeat photography was analyzed using photogrammetric techniques to measure glacier changes. Also, a custom cartographic product was produced in the style of the 1910 Carruthers map. It displays the extent of the glaciers in 2010 and the locations of repeat photography stations for future expeditions. Placing the results of this study alongside previous work paints a clear picture of the Turgen glacial regime over the last century. The results suggest that while the snow and ice volume on the summits appears to be intact, lower elevation glaciers show significant ablation. This study successively demonstrates the utility of using historic expedition documents to extend the modern record of glacial change

Intelligent visual media processing: when graphics meets vision

The computer graphics and computer vision communities have been working closely together in recent years, and a variety of algorithms and applications have been developed to analyze and manipulate the visual media around us. There are three major driving forces behind this phenomenon: i) the availability of big data from the Internet has created a demand for dealing with the ever increasing, vast amount of resources; ii) powerful processing tools, such as deep neural networks, provide e�ective ways for learning how to deal with heterogeneous visual data; iii) new data capture devices, such as the Kinect, bridge between algorithms for 2D image understanding and 3D model analysis. These driving forces have emerged only recently, and we believe that the computer graphics and computer vision communities are still in the beginning of their honeymoon phase. In this work we survey recent research on how computer vision techniques bene�t computer graphics techniques and vice versa, and cover research on analysis, manipulation, synthesis, and interaction. We also discuss existing problems and suggest possible further research directions

Of assembling small sculptures and disassembling large geometry

This thesis describes the research results and contributions that have been achieved during the author’s doctoral work. It is divided into two independent parts, each of which is devoted to a particular research aspect. The first part covers the true-to-detail creation of digital pieces of art, so-called relief sculptures, from given 3D models. The main goal is to limit the depth of the contained objects with respect to a certain perspective without compromising the initial three-dimensional impression. Here, the preservation of significant features and especially their sharpness is crucial. Therefore, it is necessary to overemphasize fine surface details to ensure their perceptibility in the more complanate relief. Our developments are aimed at amending the flexibility and user-friendliness during the generation process. The main focus is on providing real-time solutions with intuitive usability that make it possible to create precise, lifelike and aesthetic results. These goals are reached by a GPU implementation, the use of efficient filtering techniques, and the replacement of user defined parameters by adaptive values. Our methods are capable of processing dynamic scenes and allow the generation of seamless artistic reliefs which can be composed of multiple elements. The second part addresses the analysis of repetitive structures, so-called symmetries, within very large data sets. The automatic recognition of components and their patterns is a complex correspondence problem which has numerous applications ranging from information visualization over compression to automatic scene understanding. Recent algorithms reach their limits with a growing amount of data, since their runtimes rise quadratically. Our aim is to make even massive data sets manageable. Therefore, it is necessary to abstract features and to develop a suitable, low-dimensional descriptor which ensures an efficient, robust, and purposive search. A simple inspection of the proximity within the descriptor space helps to significantly reduce the number of necessary pairwise comparisons. Our method scales quasi-linearly and allows a rapid analysis of data sets which could not be handled by prior approaches because of their size.Die vorgelegte Arbeit beschreibt die wissenschaftlichen Ergebnisse und Beiträge, die während der vergangenen Promotionsphase entstanden sind. Sie gliedert sich in zwei voneinander unabhängige Teile, von denen jeder einem eigenen Forschungsschwerpunkt gewidmet ist. Der erste Teil beschäftigt sich mit der detailgetreuen Erzeugung digitaler Kunstwerke, sogenannter Reliefplastiken, aus gegebenen 3D-Modellen. Das Ziel ist es, die Objekte, abhängig von der Perspektive, stark in ihrer Tiefe zu limitieren, ohne dass der Eindruck der räumlichen Ausdehnung verloren geht. Hierbei kommt dem Aufrechterhalten der Schärfe signifikanter Merkmale besondere Bedeutung zu. Dafür ist es notwendig, die feinen Details der Objektoberfläche überzubetonen, um ihre Sichtbarkeit im flacheren Relief zu gewährleisten. Unsere Weiterentwicklungen zielen auf die Verbesserung der Flexibilität und Benutzerfreundlichkeit während des Enstehungsprozesses ab. Der Fokus liegt dabei auf dem Bereitstellen intuitiv bedienbarer Echtzeitlösungen, die die Erzeugung präziser, naturgetreuer und visuell ansprechender Resultate ermöglichen. Diese Ziele werden durch eine GPU-Implementierung, den Einsatz effizienter Filtertechniken sowie das Ersetzen benutzergesteuerter Parameter durch adaptive Werte erreicht. Unsere Methoden erlauben das Verarbeiten dynamischer Szenen und die Erstellung nahtloser, kunstvoller Reliefs, die aus mehreren Elementen und Perspektiven zusammengesetzt sein können. Der zweite Teil behandelt die Analyse wiederkehrender Stukturen, sogenannter Symmetrien, innerhalb sehr großer Datensätze. Das automatische Erkennen von Komponenten und deren Muster ist ein komplexes Korrespondenzproblem mit zahlreichen Anwendungen, von der Informationsvisualisierung über Kompression bis hin zum automatischen Verstehen. Mit zunehmender Datenmenge geraten die etablierten Algorithmen an ihre Grenzen, da ihre Laufzeiten quadratisch ansteigen. Unser Ziel ist es, auch massive Datensätze handhabbar zu machen. Dazu ist es notwendig, Merkmale zu abstrahieren und einen passenden niedrigdimensionalen Deskriptor zu entwickeln, der eine effiziente, robuste und zielführende Suche erlaubt. Eine simple Betrachtung der Nachbarschaft innerhalb der Deskriptoren hilft dabei, die Anzahl notwendiger paarweiser Vergleiche signifikant zu reduzieren. Unser Verfahren skaliert quasi-linear und ermöglicht somit eine rasche Auswertung auch auf Daten, die für bisherige Methoden zu groß waren

Sketch-based interactive shape deformation using shading isophotes

De plus en plus d'importance est accordée à la création d'objets 3D en raison des récents essors technologiques. Il est donc crucial de fournir des outils appropriés et accessibles aux utilisateurs de tous les horizons. Malheureusement, les outils traditionnellement utilisés en création 3D sont conçus pour des professionnels, exigent des formations complexes et de longue durée, et ne sont pas adaptés à ceux inexperimentés qui forment la vaste majorité des utilisateurs potentiels. Nous proposons un outil de création simplifié qui utilise des méthodes inspirées d'esquisses. Dans un premier temps, le maillage désiré est créé à partir d'un contour tracé. L'intérieur est gonflé suivant la méthode de Dvoroznak et al. Dans un deuxième temps, la hauteur des sommets du maillage est manipulée en modifiant les courbes formées par l'ombrage. Cet ombrage provient d'un modèle de réflexion Lambertien pour une lumière directionnelle donnée. Notre méthode consiste à utiliser les courbes formées par la méthode des charactéristiques associée au problème de figure dérivée de l'ombre (Shape-From-Shading). Avec les courbes, nous identifions les régions affectées par la modification de l'ombrage. L'une de ces régions sera utilisée pour interpoler l'ombrage d'après la nouvelle isophote. À partir de ce nouvel ombrage, les courbes de la méthode des characteristiques seront utilisées afin de trouver le nouveau déplacement en s'assurant d'altérer uniquement la région affectée par le changement dans l'ombrage. Les maillages créés peuvent ensuite être combinés suivant la méthode proposée par Dvoroznak et al. afin de former un maillage unique et complexe. Notre outil se veut plus intuitif que les outils traditionnels de création. Nos résultats en illustrent le potentiel.Due to recent technological advances, the creation of 3D objects is becoming more important. It is critical to offer appropriate and accessible tools to users from diverse backgrounds. Unfortunately, the tools traditionally used in 3D creation are designed for professionals, require complex and time-consuming training, and are unsuitable for inexperienced users who form the vast majority of potential users. We propose a simplified creation tool that uses sketch-based methods. First, the desired mesh is created from a traced outline. The interior is inflated following the method of Dvoroznak et al. Second, the height (displacement) of the mesh is achieved by altering the strips created by shading. Shading is the result of a Lambertian reflection model for a given directional light. Our method consists of using the strips from the method of characteristics applied to solve Shape-From-Shading. Using the strips, we identify the regions affected by the change in shading. One of these regions will be used to interpolate the shading according to the new isophote. From this new shading, the characteristic strips will be used to find the new height, ensuring that only the region affected by the change in shading is altered. The meshes created can then be combined, inspired by the method proposed by Dvoroznak et al. to form a single, complex mesh. Our tool is designed to be more intuitive than the ones provided by professional 3D software. Our results illustrate its potential