Mobile graphics: SIGGRAPH Asia 2017 course

Peer ReviewedPostprint (published version

Casual 3D photography

We present an algorithm that enables casual 3D photography. Given a set of input photos captured with a hand-held cell phone or DSLR camera, our algorithm reconstructs a 3D photo, a central panoramic, textured, normal mapped, multi-layered geometric mesh representation. 3D photos can be stored compactly and are optimized for being rendered from viewpoints that are near the capture viewpoints. They can be rendered using a standard rasterization pipeline to produce perspective views with motion parallax. When viewed in VR, 3D photos provide geometrically consistent views for both eyes. Our geometric representation also allows interacting with the scene using 3D geometry-aware effects, such as adding new objects to the scene and artistic lighting effects. Our 3D photo reconstruction algorithm starts with a standard structure from motion and multi-view stereo reconstruction of the scene. The dense stereo reconstruction is made robust to the imperfect capture conditions using a novel near envelope cost volume prior that discards erroneous near depth hypotheses. We propose a novel parallax-tolerant stitching algorithm that warps the depth maps into the central panorama and stitches two color-and-depth panoramas for the front and back scene surfaces. The two panoramas are fused into a single non-redundant, well-connected geometric mesh. We provide videos demonstrating users interactively viewing and manipulating our 3D photos

Viewpoint-Free Photography for Virtual Reality

Viewpoint-free photography, i.e., interactively controlling the viewpoint of a photograph after capture, is a standing challenge. In this thesis, we investigate algorithms to enable viewpoint-free photography for virtual reality (VR) from casual capture, i.e., from footage easily captured with consumer cameras. We build on an extensive body of work in image-based rendering (IBR). Given images of an object or scene, IBR methods aim to predict the appearance of an image taken from a novel perspective. Most IBR methods focus on full or near-interpolation, where the output viewpoints either lie directly between captured images, or nearby. These methods are not suitable for VR, where the user has significant range of motion and can look in all directions. Thus, it is essential to create viewpoint-free photos with a wide field-of-view and sufficient positional freedom to cover the range of motion a user might experience in VR. We focus on two VR experiences: 1) Seated VR experiences, where the user can lean in different directions. This simplifies the problem, as the scene is only observed from a small range of viewpoints. Thus, we focus on easy capture, showing how to turn panorama-style capture into 3D photos, a simple representation for viewpoint-free photos, and also how to speed up processing so users can see the final result on-site. 2) Room-scale VR experiences, where the user can explore vastly different perspectives. This is challenging: More input footage is needed, maintaining real-time display rates becomes difficult, view-dependent appearance and object backsides need to be modelled, all while preventing noticeable mistakes. We address these challenges by: (1) creating refined geometry for each input photograph, (2) using a fast tiled rendering algorithm to achieve real-time display rates, and (3) using a convolutional neural network to hide visual mistakes during compositing. Overall, we provide evidence that viewpoint-free photography is feasible from casual capture. We thoroughly compare with the state-of-the-art, showing that our methods achieve both a numerical improvement and a clear increase in visual quality for both seated and room-scale VR experiences

Senseable Spaces: from a theoretical perspective to the application in augmented environments

openGrazie all’ enorme diffusione di dispositivi senzienti nella vita di tutti i giorni, nell’ ultimo decennio abbiamo assistito ad un cambio definitivo nel modo in cui gli utenti interagiscono con lo spazio circostante. Viene coniato il termine Spazio Sensibile, per descrivere quegli spazi in grado di fornire servizi contestuali agli utenti, misurando e analizzando le dinamiche che in esso avvengono, e di reagire conseguentemente a questo continuo flusso di dati bidirezionale. La ricerca è stata condotta abbracciando diversi domini di applicazione, le cui singole esigenze hanno reso necessario testare il concetto di Spazi Sensibili in diverse declinazioni, mantenendo al centro della ricerca l’utente, con la duplice accezione di end-user e manager. Molteplici sono i contributi rispetto allo stato dell’ arte. Il concetto di Spazio Sensibile è stato calato nel settore dei Beni Culturali, degli Spazi Pubblici, delle Geosciences e del Retail. I casi studio nei musei e nella archeologia dimostrano come l’ utilizzo della Realtà Aumentata possa essere sfruttata di fronte a un dipinto o in outdoor per la visualizzazione di modelli complessi, In ambito urbano, il monitoraggio di dati generati dagli utenti ha consentito di capire le dinamiche di un evento di massa, durante il quale le stesse persone fruivano di servizi contestuali. Una innovativa applicazione di Realtà Aumentata è stata come servizio per facilitare l’ ispezione di fasce tampone lungo i fiumi, standardizzando flussi di dati e modelli provenienti da un Sistema Informativo Territoriale. Infine, un robusto sistema di indoor localization è stato istallato in ambiente retail, per scopi classificazione dei percorsi e per determinare le potenzialità di un punto vendita. La tesi è inoltre una dimostrazione di come Space Sensing e Geomatica siano discipline complementari: la geomatica consente di acquisire e misurare dati geo spaziali e spazio temporali a diversa scala, lo Space Sensing utilizza questi dati per fornire servizi all’ utente precisi e contestuali.Given the tremendous growth of ubiquitous services in our daily lives, during the last few decades we have witnessed a definitive change in the way users' experience their surroundings. At the current state of art, devices are able to sense the environment and users’ location, enabling them to experience improved digital services, creating synergistic loop between the use of the technology, and the use of the space itself. We coined the term Senseable Space, to define the kinds of spaces able to provide users with contextual services, to measure and analyse their dynamics and to react accordingly, in a seamless exchange of information. Following the paradigm of Senseable Spaces as the main thread, we selected a set of experiences carried out in different fields; central to this investigation there is of course the user, placed in the dual roles of end-user and manager. The main contribution of this thesis lies in the definition of this new paradigm, realized in the following domains: Cultural Heritage, Public Open Spaces, Geosciences and Retail. For the Cultural Heritage panorama, different pilot projects have been constructed from creating museum based installations to developing mobile applications for archaeological settings. Dealing with urban areas, app-based services are designed to facilitate the route finding in a urban park and to provide contextual information in a city festival. We also outlined a novel application to facilitate the on-site inspection by risk managers thanks to the use of Augmented Reality services. Finally, a robust indoor localization system has been developed, designed to ease customer profiling in the retail sector. The thesis also demonstrates how Space Sensing and Geomatics are complementary to one another, given the assumption that the branches of Geomatics cover all the different scales of data collection, whilst Space Sensing gives one the possibility to provide the services at the correct location, at the correct time.INGEGNERIA DELL'INFORMAZIONEembargoed_20181001Pierdicca, RobertoPierdicca, Robert

Senseable Spaces: from a theoretical perspective to the application in augmented environments

Grazie all’ enorme diffusione di dispositivi senzienti nella vita di tutti i giorni, nell’ ultimo decennio abbiamo assistito ad un cambio definitivo nel modo in cui gli utenti interagiscono con lo spazio circostante. Viene coniato il termine Spazio Sensibile, per descrivere quegli spazi in grado di fornire servizi contestuali agli utenti, misurando e analizzando le dinamiche che in esso avvengono, e di reagire conseguentemente a questo continuo flusso di dati bidirezionale. La ricerca è stata condotta abbracciando diversi domini di applicazione, le cui singole esigenze hanno reso necessario testare il concetto di Spazi Sensibili in diverse declinazioni, mantenendo al centro della ricerca l’utente, con la duplice accezione di end-user e manager. Molteplici sono i contributi rispetto allo stato dell’ arte. Il concetto di Spazio Sensibile è stato calato nel settore dei Beni Culturali, degli Spazi Pubblici, delle Geosciences e del Retail. I casi studio nei musei e nella archeologia dimostrano come l’ utilizzo della Realtà Aumentata possa essere sfruttata di fronte a un dipinto o in outdoor per la visualizzazione di modelli complessi, In ambito urbano, il monitoraggio di dati generati dagli utenti ha consentito di capire le dinamiche di un evento di massa, durante il quale le stesse persone fruivano di servizi contestuali. Una innovativa applicazione di Realtà Aumentata è stata come servizio per facilitare l’ ispezione di fasce tampone lungo i fiumi, standardizzando flussi di dati e modelli provenienti da un Sistema Informativo Territoriale. Infine, un robusto sistema di indoor localization è stato istallato in ambiente retail, per scopi classificazione dei percorsi e per determinare le potenzialità di un punto vendita. La tesi è inoltre una dimostrazione di come Space Sensing e Geomatica siano discipline complementari: la geomatica consente di acquisire e misurare dati geo spaziali e spazio temporali a diversa scala, lo Space Sensing utilizza questi dati per fornire servizi all’ utente precisi e contestuali.Given the tremendous growth of ubiquitous services in our daily lives, during the last few decades we have witnessed a definitive change in the way users' experience their surroundings. At the current state of art, devices are able to sense the environment and users’ location, enabling them to experience improved digital services, creating synergistic loop between the use of the technology, and the use of the space itself. We coined the term Senseable Space, to define the kinds of spaces able to provide users with contextual services, to measure and analyse their dynamics and to react accordingly, in a seamless exchange of information. Following the paradigm of Senseable Spaces as the main thread, we selected a set of experiences carried out in different fields; central to this investigation there is of course the user, placed in the dual roles of end-user and manager. The main contribution of this thesis lies in the definition of this new paradigm, realized in the following domains: Cultural Heritage, Public Open Spaces, Geosciences and Retail. For the Cultural Heritage panorama, different pilot projects have been constructed from creating museum based installations to developing mobile applications for archaeological settings. Dealing with urban areas, app-based services are designed to facilitate the route finding in a urban park and to provide contextual information in a city festival. We also outlined a novel application to facilitate the on-site inspection by risk managers thanks to the use of Augmented Reality services. Finally, a robust indoor localization system has been developed, designed to ease customer profiling in the retail sector. The thesis also demonstrates how Space Sensing and Geomatics are complementary to one another, given the assumption that the branches of Geomatics cover all the different scales of data collection, whilst Space Sensing gives one the possibility to provide the services at the correct location, at the correct time

TMO: Textured Mesh Acquisition of Objects with a Mobile Device by using Differentiable Rendering

We present a new pipeline for acquiring a textured mesh in the wild with a single smartphone which offers access to images, depth maps, and valid poses. Our method first introduces an RGBD-aided structure from motion, which can yield filtered depth maps and refines camera poses guided by corresponding depth. Then, we adopt the neural implicit surface reconstruction method, which allows for high-quality mesh and develops a new training process for applying a regularization provided by classical multi-view stereo methods. Moreover, we apply a differentiable rendering to fine-tune incomplete texture maps and generate textures which are perceptually closer to the original scene. Our pipeline can be applied to any common objects in the real world without the need for either in-the-lab environments or accurate mask images. We demonstrate results of captured objects with complex shapes and validate our method numerically against existing 3D reconstruction and texture mapping methods.Comment: Accepted to CVPR23. Project Page: https://jh-choi.github.io/TMO

Applying Augmented Reality to Outdoors Industrial Use

Augmented Reality (AR) is currently gaining popularity in multiple different fields. However, the technology for AR still requires development in both hardware and software when considering industrial use. In order to create immersive AR applications, more accurate pose estimation techniques to define virtual camera location are required. The algorithms for pose estimation often require a lot of processing power, which makes robust pose estimation a difficult task when using mobile devices or designated AR tools. The difficulties are even larger in outdoor scenarios where the environment can vary a lot and is often unprepared for AR. This thesis aims to research different possibilities for creating AR applications for outdoor environments. Both hardware and software solutions are considered, but the focus is more on software. The majority of the thesis focuses on different visual pose estimation and tracking techniques for natural features. During the thesis, multiple different solutions were tested for outdoor AR. One commercial AR SDK was tested, and three different custom software solutions were developed for an Android tablet. The custom software solutions were an algorithm for combining data from magnetometer and a gyroscope, a natural feature tracker and a tracker based on panorama images. The tracker based on panorama images was implemented based on an existing scientific publication, and the presented tracker was further developed by integrating it to Unity 3D and adding a possibility for augmenting content. This thesis concludes that AR is very close to becoming a usable tool for professional use. The commercial solutions currently available are not yet ready for creating tools for professional use, but especially for different visualization tasks some custom solutions are capable of achieving a required robustness. The panorama tracker implemented in this thesis seems like a promising tool for robust pose estimation in unprepared outdoor environments.Lisätyn todellisuuden suosio on tällä hetkellä kasvamassa usealla eri alalla. Saatavilla olevat ohjelmistot sekä laitteet eivät vielä riitä lisätyn todellisuuden soveltamiseen ammattimaisessa käytössä. Erityisesti posen estimointi vaatii tarkempia menetelmiä, jotta immersiivisten lisätyn todellisuuden sovellusten kehittäminen olisi mahdollista. Posen estimointiin (laitteen asennon- sekä paikan arviointiin) käytetyt algoritmit ovat usein monimutkaisia, joten ne vaativat merkittävästi laskentatehoa. Laskentatehon vaatimukset ovat usein haasteellisia varsinkin mobiililaitteita sekä lisätyn todellisuuden laitteita käytettäessä. Lisäongelmia tuottaa myös ulkotilat, jossa ympäristö voi muuttua usein ja ympäristöä ei ole valmisteltu lisätyn todellisuuden sovelluksille. Diplomityön tarkoituksena on tutkia mahdollisuuksia lisätyn todellisuuden sovellusten kehittämiseen ulkotiloihin. Sekä laitteisto- että ohjelmistopohjaisia ratkaisuja käsitellään. Ohjelmistopohjaisia ratkaisuja käsitellään työssä laitteistopohjaisia ratkaisuja laajemmin. Suurin osa diplomityöstä keskittyy erilaisiin visuaalisiin posen estimointi tekniikoihin, jotka perustuvat kuvasta tunnistettujen luonnollisten piirteiden seurantaan. Työn aikana testattiin useita ratkaisuja ulkotiloihin soveltuvaan lisättyyn todellisuuteen. Yhtä kaupallista työkalua testattiin, jonka lisäksi toteutettiin kolme omaa sovellusta Android tableteille. Työn aikana kehitetyt sovellukset olivat yksinkertainen algoritmi gyroskoopin ja magnetometrin datan yhdistämiseen, luonnollisen piirteiden seuranta-algoritmi sekä panoraamakuvaan perustuva seuranta-algoritmi. Panoraamakuvaan perustuva seuranta-algoritmi on toteuteutettu toisen tieteellisen julkaisun pohjalta, ja algoritmia jatkokehitettiin integroimalla se Unity 3D:hen. Unity 3D-integrointi mahdollisti myös sisällön esittämisen lisätyn todellisuuden avulla. Työn lopputuloksena todetaan, että lisätyn todellisuuden teknologia on lähellä pistettä, jossa lisätyn todellisuuden työkaluja voitaisiin käyttää ammattimaisessa käytössä. Tällä hetkellä saatavilla olevat kaupalliset työkalut eivät vielä pääse ammattikäytön vaatimalle tasolle, mutta erityisesti visualisointitehtäviin soveltuvia ei-kaupallisia ratkaisuja on jo olemassa. Lisäksi työn aikana toteutetun panoraamakuviin perustuvan seuranta-algoritmin todetaan olevan lupaava työkalu posen estimointiin ulkotiloissa.Siirretty Doriast

Mobile Wound Assessment and 3D Modeling from a Single Image

The prevalence of camera-enabled mobile phones have made mobile wound assessment a viable treatment option for millions of previously difficult to reach patients. We have designed a complete mobile wound assessment platform to ameliorate the many challenges related to chronic wound care. Chronic wounds and infections are the most severe, costly and fatal types of wounds, placing them at the center of mobile wound assessment. Wound physicians assess thousands of single-view wound images from all over the world, and it may be difficult to determine the location of the wound on the body, for example, if the wound is taken at close range. In our solution, end-users capture an image of the wound by taking a picture with their mobile camera. The wound image is segmented and classified using modern convolution neural networks, and is stored securely in the cloud for remote tracking. We use an interactive semi-automated approach to allow users to specify the location of the wound on the body. To accomplish this we have created, to the best our knowledge, the first 3D human surface anatomy labeling system, based off the current NYU and Anatomy Mapper labeling systems. To interactively view wounds in 3D, we have presented an efficient projective texture mapping algorithm for texturing wounds onto a 3D human anatomy model. In so doing, we have demonstrated an approach to 3D wound reconstruction that works even for a single wound image

Combining Occupancy Grids with a Polygonal Obstacle World Model for Autonomous Flights

This chapter presents a mapping process that can be applied to autonomous systems for obstacle avoidance and trajectory planning. It is an improvement over commonly applied obstacle mapping techniques, such as occupancy grids. Problems encountered in large outdoor scenarios are tackled and a compressed map that can be sent on low-bandwidth networks is produced. The approach is real-time capable and works in full 3-D environments. The efficiency of the proposed approach is demonstrated under real operational conditions on an unmanned aerial vehicle using stereo vision for distance measurement