Serious Games in Cultural Heritage

Although the widespread use of gaming for leisure purposes has been well documented, the use of games to support cultural heritage purposes, such as historical teaching and learning, or for enhancing museum visits, has been less well considered. The state-of-the-art in serious game technology is identical to that of the state-of-the-art in entertainment games technology. As a result the field of serious heritage games concerns itself with recent advances in computer games, real-time computer graphics, virtual and augmented reality and artificial intelligence. On the other hand, the main strengths of serious gaming applications may be generalised as being in the areas of communication, visual expression of information, collaboration mechanisms, interactivity and entertainment. In this report, we will focus on the state-of-the-art with respect to the theories, methods and technologies used in serious heritage games. We provide an overview of existing literature of relevance to the domain, discuss the strengths and weaknesses of the described methods and point out unsolved problems and challenges. In addition, several case studies illustrating the application of methods and technologies used in cultural heritage are presented

Developing serious games for cultural heritage: a state-of-the-art review

Although the widespread use of gaming for leisure purposes has been well documented, the use of games to support cultural heritage purposes, such as historical teaching and learning, or for enhancing museum visits, has been less well considered. The state-of-the-art in serious game technology is identical to that of the state-of-the-art in entertainment games technology. As a result, the field of serious heritage games concerns itself with recent advances in computer games, real-time computer graphics, virtual and augmented reality and artificial intelligence. On the other hand, the main strengths of serious gaming applications may be generalised as being in the areas of communication, visual expression of information, collaboration mechanisms, interactivity and entertainment. In this report, we will focus on the state-of-the-art with respect to the theories, methods and technologies used in serious heritage games. We provide an overview of existing literature of relevance to the domain, discuss the strengths and weaknesses of the described methods and point out unsolved problems and challenges. In addition, several case studies illustrating the application of methods and technologies used in cultural heritage are presented

Self-supervised Outdoor Scene Relighting

Outdoor scene relighting is a challenging problem that requires good understanding of the scene geometry, illumination and albedo. Current techniques are completely supervised, requiring high quality synthetic renderings to train a solution. Such renderings are synthesized using priors learned from limited data. In contrast, we propose a self-supervised approach for relighting. Our approach is trained only on corpora of images collected from the internet without any user-supervision. This virtually endless source of training data allows training a general relighting solution. Our approach first decomposes an image into its albedo, geometry and illumination. A novel relighting is then produced by modifying the illumination parameters. Our solution capture shadow using a dedicated shadow prediction map, and does not rely on accurate geometry estimation. We evaluate our technique subjectively and objectively using a new dataset with ground-truth relighting. Results show the ability of our technique to produce photo-realistic and physically plausible results, that generalizes to unseen scenes.Comment: Published in ECCV '20, http://gvv.mpi-inf.mpg.de/projects/SelfRelight

OutCast: Outdoor Single-image Relighting with Cast Shadows

We propose a relighting method for outdoor images. Our method mainly focuses on predicting cast shadows in arbitrary novel lighting directions from a single image while also accounting for shading and global effects such the sun light color and clouds. Previous solutions for this problem rely on reconstructing occluder geometry, e.g. using multi-view stereo, which requires many images of the scene. Instead, in this work we make use of a noisy off-the-shelf single-image depth map estimation as a source of geometry. Whilst this can be a good guide for some lighting effects, the resulting depth map quality is insufficient for directly ray-tracing the shadows. Addressing this, we propose a learned image space ray-marching layer that converts the approximate depth map into a deep 3D representation that is fused into occlusion queries using a learned traversal. Our proposed method achieves, for the first time, state-of-the-art relighting results, with only a single image as input. For supplementary material visit our project page at: https://dgriffiths.uk/outcast.Comment: Eurographics 2022 - Accepte

Free-viewpoint Indoor Neural Relighting from Multi-view Stereo

We introduce a neural relighting algorithm for captured indoors scenes, that allows interactive free-viewpoint navigation. Our method allows illumination to be changed synthetically, while coherently rendering cast shadows and complex glossy materials. We start with multiple images of the scene and a 3D mesh obtained by multi-view stereo (MVS) reconstruction. We assume that lighting is well-explained as the sum of a view-independent diffuse component and a view-dependent glossy term concentrated around the mirror reflection direction. We design a convolutional network around input feature maps that facilitate learning of an implicit representation of scene materials and illumination, enabling both relighting and free-viewpoint navigation. We generate these input maps by exploiting the best elements of both image-based and physically-based rendering. We sample the input views to estimate diffuse scene irradiance, and compute the new illumination caused by user-specified light sources using path tracing. To facilitate the network's understanding of materials and synthesize plausible glossy reflections, we reproject the views and compute mirror images. We train the network on a synthetic dataset where each scene is also reconstructed with MVS. We show results of our algorithm relighting real indoor scenes and performing free-viewpoint navigation with complex and realistic glossy reflections, which so far remained out of reach for view-synthesis techniques

Multi-view Relighting using a Geometry-Aware Network

International audienceWe propose the first learning-based algorithm that can relight images in a plausible and controllable manner given multiple views of an outdoor scene. In particular, we introduce a geometry-aware neural network that utilizes multiple geometry cues (normal maps, specular direction, etc.) and source and target shadow masks computed from a noisy proxy geometry obtained by multi-view stereo. Our model is a three-stage pipeline: two subnetworks refine the source and target shadow masks, and a third performs the final relighting. Furthermore, we introduce a novel representation for the shadow masks, which we call RGB shadow images. They reproject the colors from all views into the shadowed pixels and enable our network to cope with inacuraccies in the proxy and the non-locality of the shadow casting interactions. Acquiring large-scale multi-view relighting datasets for real scenes is challenging, so we train our network on photorealistic synthetic data. At train time, we also compute a noisy stereo-based geometric proxy, this time from the synthetic renderings. This allows us to bridge the gap between the real and synthetic domains. Our model generalizes well to real scenes. It can alter the illumination of drone footage, image-based renderings, textured mesh reconstructions, and even internet photo collections

Free-viewpoint Indoor Neural Relighting from Multi-view Stereo

OPAL-MesoInternational audienceWe introduce a neural relighting algorithm for captured indoors scenes, that allows interactive free-viewpoint navigation. Our method allows illumination to be changed synthetically, while coherently rendering cast shadows and complex glossy materials. We start with multiple images of the scene and a 3D mesh obtained by multi-view stereo (MVS) reconstruction. We assume that lighting is well-explained as the sum of a view-independent diffuse component and a view-dependent glossy term concentrated around the mirror reflection direction. We design a convolutional network around input feature maps that facilitate learning of an implicit representation of scene materials and illumination, enabling both relighting and free-viewpoint navigation. We generate these input maps by exploiting the best elements of both image-based and physically-based rendering. We sample the input views to estimate diffuse scene irradiance, and compute the new illumination caused by user-specified light sources using path tracing. To facilitate the network's understanding of materials and synthesize plausible glossy reflections, we reproject the views and compute mirror images. We train the network on a synthetic dataset where each scene is also reconstructed with MVS. We show results of our algorithm relighting real indoor scenes and performing free-viewpoint navigation with complex and realistic glossy reflections, which so far remained out of reach for view-synthesis techniques

RTI-based techniques and tools for digital surrogates

Dissertação de mestrado em Informática.Digital representations of Cultural Heritage (CH) and Natural Science (NS) artefacts often use collected data from the real world together with computer graphics techniques and skilled human intervention. A scholar will trust these representations when they are extended to digital surrogates, with all empir ical provenance information logged in. and use them when they are robust, affordable and easy to access and handle. This was the main context and motivation for this dissertation. Image-based techniques are a popular way to acquire and model surface materials. Reflectance Transformation Imaging (RTI) provide a powerful and efficient tool to build a model of the surface of an artefact, with a reliable 3D visualization, as analysed in this dissertation. An automated process pipeline was developed to acquire and build an RTI representation as a digital surrogate, in an open source context, the RTIbuilder. This automated process pipeline uses computer vision algorithms to acquire the information required by the generation stage and offers the user a set of tools for the most commonly used image processing procedures. The RTIbuilder not only apply software engineering techniques to solve the user’s problems, but also addresses most of the concerns posed by the scientific method: it removes the unreliable human factor from the RTI gen eration process during the workflow automation, by storing and preserving all the empirical provenance that include the unaltered empirical data gath ered, the process and parameters history and all human intervention over the gathered empirical information and process parameters. Workshops and tutorials were given to audiences with CH and NS profes sionals, and the RTIbuilder was field tested in several locations worldwide. Feedbacks were valuable to tune the application and opened new tracks for future work.Representações digitais de artefactos da herança cultural e das ciências natu rais usam dados do mundo real em combinação com técnicas de computação gráfica e intervenção de profissionais especializados. Um investigador apenas poderá confiar nestas representações digitais se elas forem convertidas em substitutos digitais, que deverão incluir toda a informação de proveniência. Técnicas baseadas em imagem são uma forma tradicional de capturar e modelar materiais. "Reflectance Transformation Imaging (RTI)" disponibi liza uma ferramenta poderosa e eficiente para construir uma representação da superfície de um artefacto, com uma visualização 3D fidedigna, tal como se demonstra ao longo desta dissertação. Foi desenvolvido um encadeamento de processos automático para capturar e produzir representações RTI enquanto substitutos digitais, num contexto de software aberto, dando origem a uma ferramenta designada RTIbuilder. Esta metodologia usa algoritmos de visão computacional para adquirir e calcular os parâmetros de entrada necessários à geração das representações RTI. Para além disso, disponibiliza um conjunto de ferramentas para as técnicas de processamento mais utilizadas no processo. O RTIbuilder, para além de aplicar técnicas de engenharia de software na resolução de problemas do utilizador, vai também ao encontro dos principais problemas decorrentes do método científico: remove a falibilidade humana do processo de geração de RTI, uma vez que armazena e preserva a proveniência empírica. A proveniência empírica engloba as observações empíricas recolhi das e inalteradas, a sequência de processos e respectivos parâmetros usados, bem como todo o histórico de manipulação da informação empírica recolhida e parâmetros de processo. Foram ministrados workshops e tutoriais a profissionais da herança cul tural e ciencias naturais e o RTIbuilder foi testado em situações reais em vários locais a nível mundial. Os resultados desses testes foram imprescindíveis para afinar a aplicação e abriu novos caminhos para trabalho futuro