Analysis and Development of Augmented Reality Applications for the Dissemination of Cultural Heritage

Tesis por compendio[ES] La RA consiste en la superposición de elementos virtuales sobre el entorno real, de manera que el usuario percibe estos elementos como si formaran parte de la realidad que está visualizando. Las aplicaciones de RA en dispositivos móviles permiten visualizar el contenido virtual a través de la cámara del dispositivo. La RA es una herramienta de divulgación muy potente ya que permite añadir a la realidad cualquier tipo de información, desde un simple texto informativo a un modelo 3D interactivo. Tiene infinitas utilidades, puede servir de guía en un museo, puede mostrar la recreación de un monumento destruido, o como en el caso de estudio aquí presentado, ayudar a la interpretación de pinturas rupestres. Esta tesis parte de la idea de que la RA puede mejorar mucho la interpretación del arte rupestre sin alterar ni dañar las pinturas. Puede servir para atraer a un público mayor, dar a conocer la historia de las pinturas rupestres y que al mismo tiempo el visitante tenga una experiencia mucho más enriquecedora. A lo largo de la tesis se ha estudiado en profundidad la técnica de visualización de RA mediante dispositivos móviles. Se han analizado las diferentes librerías de programación mediante casos de estudio en entornos reales y examinado los factores que pueden afectar al reconocimiento de las pinturas. Se ha desarrollado una aplicación de RA aplicada a un caso real de pinturas rupestres y posteriormente ha sido evaluada por un grupo de personas. Finalmente, se ha estudiado el efecto de la luz solar y sus cambios a lo largo del día sobre el reconocimiento de imágenes en entornos al aire libre. Este trabajo proporciona un punto de partida para el desarrollo de aplicaciones de RA aplicadas a la difusión del patrimonio cultural, especialmente centrado en el arte rupestre, un entorno que sufre de unas dificultades añadidas debido a su localización, dificultad de reconocimiento de puntos característicos en las pinturas y los cambios en la luz solar, problemas que se han tratado de resolver a lo largo del estudio. Las principales conclusiones han sido muy favorables, partiendo de librerías de programación disponibles y gratuitas. Se han podido desarrollar un conjunto de aplicaciones de RA en diferentes lugares. Las valoraciones han sido muy positivas, los usuarios que han probado las aplicaciones afirman que la interpretación de las pinturas les resulta más fácil y consiguen entender mejor el propósito de las mismas. El principal inconveniente encontrado es la falta de conocimiento sobre esta técnica y la pérdida de realismo en algunos casos debido a la oclusión, es decir, que los objetos virtuales no se posicionen por detrás de los objetos reales. La buena noticia es que esta tecnología evoluciona muy rápido y durante el desarrollo de la tesis ha habido avances muy grandes, entre ellos, el desarrollo de nuevas librerías de programación desarrolladas por Google y Apple, que proporcionan las herramientas necesarias para crear aplicaciones muy potentes e immersivas, donde el usuario se sentirá parte de los entornos creados.[CA] La RA consisteix en la superposició d'elements virtuals sobre l'entorn real, de manera que l'usuari percep aquests elements com si formaren part de la realitat que està visualitzant. Les aplicacions de RA en dispositius mòbils permeten visualitzar el contingut virtual a través de la cambra del dispositiu. La RA és una eina de divulgació molt potent ja que permet afegir a la realitat qualsevol tipus d'informació, des d'un simple text informatiu a un model 3D interactiu. Té infinites utilitats, pot servir de guia en un museu, pot mostrar la recreació d'un monument destruït, o com en el cas d'estudi ací presentat, ajudar a la interpretació de pintures rupestres. Aquesta tesi parteix de la idea que la RA pot millorar molt la interpretació de l'art rupestre sense alterar ni danyar les pintures. Pot servir per a atraure a un públic major, donar a conéixer la història de les pintures rupestres i que al mateix temps el visitant tinga una experiència molt més enriquidora. Al llarg de la tesi s'ha estudiat en profunditat la tècnica de visualització de RA mitjançant dispositius mòbils. S'han analitzat les diferents llibreries de programació mitjançant casos d'estudi en entorns reals i analitzat els factors que poden afectar el reconeixement de les pintures. S'ha desenvolupat una aplicació de RA aplicada a un cas real de pintures rupestres i posteriorment ha sigut avaluada per un grup de persones. Finalment, s'ha estudiat l'efecte de la llum solar i els seus canvis al llarg del dia sobre el reconeixement d'imatges en entorns a l'aire lliure. Aquest treball proporciona un punt de partida per al desenvolupament d'aplicacions de RA aplicades a la difusió del patrimoni cultural, especialment centrat en l'art rupestre, un entorn que pateix d'unes dificultats afegides a causa de la seua localització, dificultat de reconeixement de punts característics en les pintures i els canvis en la llum solar, problemes que s'han tractat de resoldre al llarg de l'estudi. Les principals conclusions han sigut molt favorables, partint de llibreries de programació disponibles i gratuïtes. S'han pogut desenvolupar un conjunt d'aplicacions de RA en diferents llocs. Les valoracions han sigut molt positives, els usuaris que han provat les aplicacions afirmen que la interpretació de les pintures els resulta més fàcil i aconsegueixen entendre millor el propòsit d'aquestes. El principal inconvenient trobat és la falta de coneixement sobre aquesta tècnica i la perduda de realisme en alguns casos a causa de l'oclusió, és a dir, que els objectes virtuals no es posicionen per darrere dels objectes reals. La bona notícia és que aquesta tecnologia evoluciona molt ràpid i durant el desenvolupament de la tesi hi ha hagut avanços molt grans, entre ells, el desenvolupament de noves llibreries de programació per Google i Apple, que proporcionen les eines necessàries per a crear aplicacions molt potents i immersives, on l'usuari se sentirà part dels entorns creats.[EN] AR consists of superimposing virtual elements on the real environment, so that the user perceives these elements as if they were part of the reality they are looking at. AR applications on smartphones allow virtual content to be visualised through the device's camera. AR is a very powerful tool for dissemination as it allows any type of information to be added to reality, from a simple informative text to an interactive 3D model. It can be used as a guide in a museum, it can show the recreation of a destroyed monument, or, as in the case study presented here, it can help in the interpretation of cave paintings. This thesis is based on the idea that AR can greatly enhance the interpretation of rock art without affecting or damaging the paintings. It can be used to attract a wider audience, to introduce the history of the rock art paintings and at the same time provide the visitor with a much more enriching experience. Throughout the thesis, the technique of AR visualisation using mobile devices has been studied in-depth. The different programming libraries have been analysed by means of case studies in real environments as well as the factors that can affect the paintings recognition. An AR application applied to a real case of rock art paintings has been developed and subsequently evaluated by a group of people. Finally, the effect of sunlight and its changes throughout the day on image recognition in outdoor environments has been studied. This work provides a starting point for the AR applications development applied to the dissemination of cultural heritage, especially focused on rock art, an environment that suffers from additional difficulties due to its location, the difficulty of characteristic points recognition and changes in sunlight, problems that have been tried to solve throughout the study. The main outcomes have been very favourable, using freely available programming libraries, and it has been possible to develop a set of AR applications in different places. The evaluations have been very positive, with users who have tested the applications confirming that the interpretation of the paintings is easier for them and they can better understand the purpose of the paintings. The major drawback is the lack of knowledge about this technique and the loss of realism in some cases due to occlusion, i.e. the virtual objects are not positioned behind the real objects. The good news is that this technology is evolving very fast and during the development of the thesis there have been great advances, among them, the development of new programming libraries developed by Google and Apple, which provide the necessary tools to create very powerful and immersive applications, where the user will feel part of the virtual environments created.Blanco Pons, S. (2021). Analysis and Development of Augmented Reality Applications for the Dissemination of Cultural Heritage [Tesis doctoral]. Universitat Politècnica de València. https://doi.org/10.4995/Thesis/10251/178895TESISCompendi

Augmented reality application assessment for disseminating rock art

[EN] Currently, marker-based tracking is the most used method to develop augmented reality (AR) applications (apps). However, this method cannot be applied in some complex and outdoor settings such as prehistoric rock art sites owing to the fact that the usage of markers is restricted on site. Thus, natural feature tracking methods have to be used. There is a wide range of libraries to develop AR apps based on natural feature tracking. In this paper, a comparative study of Vuforia and ARToolKit libraries is carried out, analysing factors such as distance, occlusion and lighting conditions that affect user experience in both indoor and outdoor environments, and eventually the app developer. Our analysis confirms that Vuforia¿s user experience indoor is better, faster and flicker-free whether the images are properly enhanced, but it does not work properly on site. Therefore, the development of AR apps for complex outdoor environments such as rock art sites should be performed with ARToolKit.The authors gratefully acknowledge the support from the Spanish Ministerio de Economia y Competitividad to the project HAR2014-59873-R. Similarly, the authors want to express their gratitude to the General Directorate of Culture and Heritage, Conselleria d'Educacio, Investigacio, Cultura i Esport, Generalitat Valenciana for letting us access and carry out research at the archaeological site.Blanco-Pons, S.; Carrión-Ruiz, B.; Lerma, JL. (2018). Augmented reality application assessment for disseminating rock art. Multimedia Tools and Applications. 78(8):10265-10286. https://doi.org/10.1007/s11042-018-6609-xS1026510286788Alahi A., Ortiz R., Vandergheynst P (2012) FREAK: fast retina keypoint. Comput Vis Pattern Recognit 510–517 . doi: https://doi.org/10.1109/CVPR.2012.6247715Amin D, Govilkar S (2015) Comparative study of augmented reality Sdk’S. Int J Comput Sci Appl 5:11–26. https://doi.org/10.1227/01.NEU.0000297044.82035.57ARCore ARCore - Google Developer | ARCore | Google Developers. https://developers.google.com/ar/ . Accessed 26 Jun 2018ARKit ARKit - Apple Developer. https://developer.apple.com/arkit/ . Accessed 26 Jun 2018ARToolkit (2017) ARToolkit. https://archive.artoolkit.org/ . Accessed 2 Oct 2017ARToolkit (2017) About. https://artoolkit.org/about-artoolkit . Accessed 11 Apr 2017ARToolkit (2017) Documentation. https://artoolkit.org/documentation/ . Accessed 12 Apr 2017ArUco ArUco: A minimal library for Augmented Reality applications based on OpenCV | Aplicaciones de la Visión Artificial. https://www.uco.es/investiga/grupos/ava/node/26 . Accessed 19 Apr 2018Azuma R (1997) A survey of augmented reality. Presence Teleoperators Virt Environ 6:355–385 . doi: 10.1.1.30.4999Azuma R, Baillot Y, Feiner S et al (2001) Recent advances in augmented reality. Ieee Comput Graph Appl 34–47. doi: https://doi.org/10.4061/2011/908468Blanco-Novoa O, Fernandez-Carames TM, Fraga-Lamas P, Vilar-Montesinos M (2018) A practical evaluation of commercial industrial augmented reality systems in an industry 4.0 shipyard. IEEE Access 6:1–1. https://doi.org/10.1109/ACCESS.2018.2802699Blanco-Pons S, Carrión-Ruiz B, Lerma JL (2016) Review of augmented reality and virtual reality techniques in rock art. Proc 8th Int Congress Archaeol Comput Graph Cult Herit Innov ‘ARQUEOLÓGICA 2.0L: 176–183Brancati N, Caggianese G, Frucci M et al (2017) Experiencing touchless interaction with augmented content on wearable head-mounted displays in cultural heritage applications. Pers Ubiquitous Comput 21:203–217. https://doi.org/10.1007/s00779-016-0987-8Cagalaban G, Kim S (2010) Multiple object tracking in unprepared environments using combined feature for augmented reality applications. Springer, Berlin, HeidelbergCamera-Calibration Camera Calibration App for Android [ARToolkit]. https://archive.artoolkit.org/documentation/doku.php?id=4_Android:android_camera_calibration . Accessed 16 Oct 2017Carmigniani J, Furht B, Anisetti M et al (2011) Augmented reality technologies, systems and applications. Multimed Tools Appl 51:341–377. https://doi.org/10.1007/s11042-010-0660-6Carrión-Ruiz B, Blanco-Pons S, Lerma JL (2016) Digital image analysis of the visible region through simulation of rock art paintings. Proc 8th Int Congress Archaeol Comput Graph, Cult Heritage Innov ‘ARQUEOLÓGICA 2.0.’: 169–175Chen CY, Chang BR, Sen HP (2014) Multimedia augmented reality information system for museum guidance. Pers Ubiquitous Comput 18:315–322. https://doi.org/10.1007/s00779-013-0647-1CRYENGINE CRYENGINE | The complete solution for next generation game development by Crytek. https://www.cryengine.com/ . Accessed 7 Jun 2017Domingo I, Carrión B, Blanco S, Lerma JL (2015) Evaluating conventional and advanced visible image enhancement solutions to produce digital tracings at el Carche rock art shelter. Digit Appl Archaeol Cult Herit 2:79–88. https://doi.org/10.1016/j.daach.2015.01.001Dos Santos AB, Dourado JB, Bezerra A (2016) ARToolkit and Qualcomm Vuforia: An Analytical Collation. Proc - 18th Symp Virt Augment Real SVR 2016:229–233. https://doi.org/10.1109/SVR.2016.46DroidAR (2017) DroidAR by bitstars. https://bitstars.github.io/droidar/ . Accessed 10 Dec 2017Engine U (2017) Unreal Engine. https://www.unrealengine.com/ . Accessed 10 Oct 2017Fiala M (2005) ARTag, a fiducial marker system using digital techniques. Proc IEEE Comput Soc Conf Comput Vis Pattern Recogn 2:590–596. https://doi.org/10.1109/CVPR.2005.74Fischer J, Eichler M, Bartz D, Straßer W (2007) A hybrid tracking method for surgical augmented reality. Comput Graph 31:39–52. https://doi.org/10.1016/j.cag.2006.09.007González C, Vallejo D, Albusac J, Castro J (2011) Realidad Aumentada. Un enfoque práctico con ARToolKit y Blender. 2–120Gutierrez JM, Molinero MA, Soto-Martín O, Medina CR (2015) Augmented reality technology spreads information about historical graffiti in temple of Debod. Procedia Comput Sci 75:390–397. https://doi.org/10.1016/j.procs.2015.12.262Haladová ZB, Szemzö R, Kovačovský T, Žižka J (2015) Utilizing Multispectral Scanning and Augmented Reality for Enhancement and Visualization of the Wooden Sculpture Restoration Process. Procedia Comput Sci 67:340–347. https://doi.org/10.1016/j.procs.2015.09.278Jamali SS, Shiratuddin MF, Wong KW, Oskam CL (2015) Utilising mobile-augmented reality for learning human anatomy. Procedia - Soc Behav Sci 197:659–668. https://doi.org/10.1016/j.sbspro.2015.07.054Khan D, Ullah S, Rabbi I (2015) Factors affecting the design and tracking of ARToolKit markers. Comput Stand Interf 41:56–66. https://doi.org/10.1016/j.csi.2015.02.006Khan D, Ullah S, Yan D et al (2018) Robust tracking through the design of high quality fiducial markers: an optimization tool for ARToolKit. IEEE Access 4:22421–22433. https://doi.org/10.1109/ACCESS.2018.2801028Kim SL, Suk HJ, Kang JH, et al (2014) Using unity 3D to facilitate mobile augmented reality game development. Internet things (WF-IoT), 2014 IEEE World Forum 21–26 . doi: https://doi.org/10.1109/WF-IoT.2014.6803110Kounavis CD, Kasimati AE, Zamani ED (2012) Enhancing the tourism experience through mobile augmented reality: challenges and prospects. Int J Eng Bus Manag 4:1–6. https://doi.org/10.5772/51644La Delfa GC, Monteleone S, Catania V et al (2016) Performance analysis of visualmarkers for indoor navigation systems. Front Inf Technol Electron Eng 17:730–740. https://doi.org/10.1631/FITEE.1500324Liu S, Ge S, Yu H (2016) Research on Robustness recognition algorithms in augmented reality. 3rd Int Conf Inf Sci Control Eng: 547–552. doi: https://doi.org/10.1109/ICISCE.2016.123Lowe DG (2004) Distinctive image features from scale invariant keypoints. Int J Comput Vis 60:91–11020042. https://doi.org/10.1023/B:VISI.0000029664.99615.94Lytridis C, Tsinakos A, Kazanidis I (2018) ARTutor—an augmented reality platform for interactive distance learning. Educ Sci 8:6. https://doi.org/10.3390/educsci8010006Marchand E, Uchiyama H, Spindler F et al (2016) Pose estimation for augmented reality : a hands-on survey. IEEE Trans Vis Comput Graph 22:2633–2651. https://doi.org/10.1109/TVCG.2015.2513408Martínez R, Villaverde V (2002) La cova dels cavalls en el Barranc de la ValltortaMarto AGR, Sousa AA, de Gonçalves A (2017) DinofelisAR demo augmented reality based on natural features. 12a Conferência Ibérica Sist e Tecnol Informação, Lisboa 64:852–861. https://doi.org/10.1016/j.procs.2015.08.638Moreels P, Perona P (2007) Evaluation of feature detectors and descriptors based on 3D objects. Int J Comput Vis 73:263–284. https://doi.org/10.1007/s11263-006-9967-1Pierdicca R, Frontoni E, Zingaretti P et al (2015) Making visible the invisible. augmented reality visualization for 3D reconstructions of archaeological sites. Augment Virt Real Sec Int Conf AVR 2015 9254:25–37. https://doi.org/10.1007/978-3-319-22888-4Rabbi I, Ullah S, Javed M, Zen K (2014) Analysis of ARToolKit fiducial markers attributes for robust tracking. 1st Int Conf Recent Trends Inf Commun Technol Anal 281–290Radkowski R, Oliver J (2013) Natural feature tracking augmented reality for on-site assembly assistance systems. In: Shumaker R (ed) Virtual, Augmented and Mixed Reality. Systems and Applications. VAMR 2013. Lecture Notes in Computer Science. Springer, Berlin, Heidelberg, pp 281–290Ridel B, Reuter P, Laviole J et al (2014) The revealing flashlight: interactive spatial augmented reality for detail exploration of cultural heritage artifacts. J Comput Cult Herit 7(6):1–6:18. https://doi.org/10.1145/2611376Seo J, Shim J, Choi JH, et al (2011) Enhancing marker-based AR technology. Lect Notes Comput Sci (including Subser Lect Notes Artif Intell Lect Notes Bioinformatics) 6773 LNCS:97–104 . doi: https://doi.org/10.1007/978-3-642-22021-0_12Seo J, Shim J, Choi JH et al (2011) Enhancing marker-based AR technology. In: International conference on virtual and mixed reality. virtual and mixed reality - new trends. Springer, Berlin, Heidelberg, pp 97–104Siltanen S (2015) Diminished reality for augmented reality interior design. Vis Comput 33:1–16. https://doi.org/10.1007/s00371-015-1174-zSörös G, Seichter H, Rautek P, Gröller E (2011) Augmented visualization with natural feature tracking. Proc 10th Int Conf Mob Ubiquitous Multimed 4–12. doi: https://doi.org/10.1145/2107596.2107597Uchiyama H, Marchand E (2012) Object detection and pose tracking for augmented reality: recent approaches. 18th Korea-Japan Jt Work Front Comput Vis 1–8Unity Unity. https://unity3d.com/es . Accessed 12 Oct 2017Vuforia (2017) Vuforia. https://www.vuforia.com/ . Accessed 2 Oct 2017Vuforia (2017) Vuforia-VuMark. https://library.vuforia.com/articles/Training/VuMark . Accessed 4 Apr 2017Vuforia (2017) Image targets. https://library.vuforia.com/articles/Training/Image-Target-Guide . Accessed 11 Apr 2017Wang H, Qin J, Zhang F (2015) A new interaction method for augmented reality based on ARToolKit. 2015 8th Int Congr Image Signal Process 578–583. doi: https://doi.org/10.1109/CISP.2015.7407945Wang G, Wang B, Zhong F et al (2015) Global optimal searching for textureless 3D object tracking. Vis Comput 31:979–988. https://doi.org/10.1007/s00371-015-1098-7Wu S, Oerlemans A, Bakker EM, Lew MS (2017) A comprehensive evaluation of local detectors and descriptors. Signal Process Image Commun 59:150–167. https://doi.org/10.1016/J.IMAGE.2017.06.010Xu Y, Wu Y, Zhou H, View M (2018) Multi-scale Voxel Hashing and Efficient 3D Representation for Mobile Augmented Reality. Cvpr 1618–1625 . doi: https://doi.org/10.1109/CVPRW.2018.0020

Registration Combining Wide and Narrow Baseline Feature Tracking Techniques for Markerless AR Systems

Augmented reality (AR) is a field of computer research which deals with the combination of real world and computer generated data. Registration is one of the most difficult problems currently limiting the usability of AR systems. In this paper, we propose a novel natural feature tracking based registration method for AR applications. The proposed method has following advantages: (1) it is simple and efficient, as no man-made markers are needed for both indoor and outdoor AR applications; moreover, it can work with arbitrary geometric shapes including planar, near planar and non planar structures which really enhance the usability of AR systems. (2) Thanks to the reduced SIFT based augmented optical flow tracker, the virtual scene can still be augmented on the specified areas even under the circumstances of occlusion and large changes in viewpoint during the entire process. (3) It is easy to use, because the adaptive classification tree based matching strategy can give us fast and accurate initialization, even when the initial camera is different from the reference image to a large degree. Experimental evaluations validate the performance of the proposed method for online pose tracking and augmentation

A dataset of annotated omnidirectional videos for distancing applications

Omnidirectional (or 360◦ ) cameras are acquisition devices that, in the next few years, could have a big impact on video surveillance applications, research, and industry, as they can record a spherical view of a whole environment from every perspective. This paper presents two new contributions to the research community: the CVIP360 dataset, an annotated dataset of 360◦ videos for distancing applications, and a new method to estimate the distances of objects in a scene from a single 360◦ image. The CVIP360 dataset includes 16 videos acquired outdoors and indoors, annotated by adding information about the pedestrians in the scene (bounding boxes) and the distances to the camera of some points in the 3D world by using markers at fixed and known intervals. The proposed distance estimation algorithm is based on geometry facts regarding the acquisition process of the omnidirectional device, and is uncalibrated in practice: the only required parameter is the camera height. The proposed algorithm was tested on the CVIP360 dataset, and empirical results demonstrate that the estimation error is negligible for distancing applications

A Survey on Augmented Reality Challenges and Tracking

This survey paper presents a classification of different challenges and tracking techniques in the field of augmented reality. The challenges in augmented reality are categorized into performance challenges, alignment challenges, interaction challenges, mobility/portability challenges and visualization challenges. Augmented reality tracking techniques are mainly divided into sensor-based tracking, visionbased tracking and hybrid tracking. The sensor-based tracking is further divided into optical tracking, magnetic tracking, acoustic tracking, inertial tracking or any combination of these to form hybrid sensors tracking. Similarly, the vision-based tracking is divided into marker-based tracking and markerless tracking. Each tracking technique has its advantages and limitations. Hybrid tracking provides a robust and accurate tracking but it involves financial and tehnical difficulties

Indoor Mapping and Reconstruction with Mobile Augmented Reality Sensor Systems

Augmented Reality (AR) ermöglicht es, virtuelle, dreidimensionale Inhalte direkt innerhalb der realen Umgebung darzustellen. Anstatt jedoch beliebige virtuelle Objekte an einem willkürlichen Ort anzuzeigen, kann AR Technologie auch genutzt werden, um Geodaten in situ an jenem Ort darzustellen, auf den sich die Daten beziehen. Damit eröffnet AR die Möglichkeit, die reale Welt durch virtuelle, ortbezogene Informationen anzureichern. Im Rahmen der vorliegenen Arbeit wird diese Spielart von AR als "Fused Reality" definiert und eingehend diskutiert. Der praktische Mehrwert, den dieses Konzept der Fused Reality bietet, lässt sich gut am Beispiel seiner Anwendung im Zusammenhang mit digitalen Gebäudemodellen demonstrieren, wo sich gebäudespezifische Informationen - beispielsweise der Verlauf von Leitungen und Kabeln innerhalb der Wände - lagegerecht am realen Objekt darstellen lassen. Um das skizzierte Konzept einer Indoor Fused Reality Anwendung realisieren zu können, müssen einige grundlegende Bedingungen erfüllt sein. So kann ein bestimmtes Gebäude nur dann mit ortsbezogenen Informationen augmentiert werden, wenn von diesem Gebäude ein digitales Modell verfügbar ist. Zwar werden größere Bauprojekt heutzutage oft unter Zuhilfename von Building Information Modelling (BIM) geplant und durchgeführt, sodass ein digitales Modell direkt zusammen mit dem realen Gebäude ensteht, jedoch sind im Falle älterer Bestandsgebäude digitale Modelle meist nicht verfügbar. Ein digitales Modell eines bestehenden Gebäudes manuell zu erstellen, ist zwar möglich, jedoch mit großem Aufwand verbunden. Ist ein passendes Gebäudemodell vorhanden, muss ein AR Gerät außerdem in der Lage sein, die eigene Position und Orientierung im Gebäude relativ zu diesem Modell bestimmen zu können, um Augmentierungen lagegerecht anzeigen zu können. Im Rahmen dieser Arbeit werden diverse Aspekte der angesprochenen Problematik untersucht und diskutiert. Dabei werden zunächst verschiedene Möglichkeiten diskutiert, Indoor-Gebäudegeometrie mittels Sensorsystemen zu erfassen. Anschließend wird eine Untersuchung präsentiert, inwiefern moderne AR Geräte, die in der Regel ebenfalls über eine Vielzahl an Sensoren verfügen, ebenfalls geeignet sind, als Indoor-Mapping-Systeme eingesetzt zu werden. Die resultierenden Indoor Mapping Datensätze können daraufhin genutzt werden, um automatisiert Gebäudemodelle zu rekonstruieren. Zu diesem Zweck wird ein automatisiertes, voxel-basiertes Indoor-Rekonstruktionsverfahren vorgestellt. Dieses wird außerdem auf der Grundlage vierer zu diesem Zweck erfasster Datensätze mit zugehörigen Referenzdaten quantitativ evaluiert. Desweiteren werden verschiedene Möglichkeiten diskutiert, mobile AR Geräte innerhalb eines Gebäudes und des zugehörigen Gebäudemodells zu lokalisieren. In diesem Kontext wird außerdem auch die Evaluierung einer Marker-basierten Indoor-Lokalisierungsmethode präsentiert. Abschließend wird zudem ein neuer Ansatz, Indoor-Mapping Datensätze an den Achsen des Koordinatensystems auszurichten, vorgestellt

Augmented Reality Markerless Multi-Image Outdoor Tracking System for the Historical Buildings on Parliament Hill

[EN] Augmented Reality (AR) applications have experienced extraordinary growth recently, evolving into a well-established method for the dissemination and communication of content related to cultural heritage¿including education. AR applications have been used in museums and gallery exhibitions and virtual reconstructions of historic interiors. However, the circumstances of an outdoor environment can be problematic. This paper presents a methodology to develop immersive AR applications based on the recognition of outdoor buildings. To demonstrate this methodology, a case study focused on the Parliament Buildings National Historic Site in Ottawa, Canada has been conducted. The site is currently undergoing a multiyear rehabilitation program that will make access to parts of this national monument inaccessible to the public. AR experiences, including simulated photo merging of historic and present content, are proposed as one tool that can enrich the Parliament Hill visit during the rehabilitation. Outdoor AR experiences are limited by factors, such as variable lighting (and shadows) conditions, caused by changes in the environment (objects height and orientation, obstructions, occlusions), the weather, and the time of day. This paper proposes a workflow to solve some of these issues from a multi-image tracking approach.This work has been developed under the framework of the New Paradigms/New Tools for Heritage Conservation in Canada, a project funded through the Social Sciences and Humanities Research Council of Canada (SSHRC).Blanco-Pons, S.; Carrión-Ruiz, B.; Duong, M.; Chartrand, J.; Fai, S.; Lerma, JL. (2019). Augmented Reality Markerless Multi-Image Outdoor Tracking System for the Historical Buildings on Parliament Hill. Sustainability. 11(16):1-15. https://doi.org/10.3390/su11164268S1151116Bekele, M. K., Pierdicca, R., Frontoni, E., Malinverni, E. S., & Gain, J. (2018). A Survey of Augmented, Virtual, and Mixed Reality for Cultural Heritage. Journal on Computing and Cultural Heritage, 11(2), 1-36. doi:10.1145/3145534Gimeno, J., Portalés, C., Coma, I., Fernández, M., & Martínez, B. (2017). Combining traditional and indirect augmented reality for indoor crowded environments. A case study on the Casa Batlló museum. Computers & Graphics, 69, 92-103. doi:10.1016/j.cag.2017.09.001Kolivand, H., El Rhalibi, A., Shahrizal Sunar, M., & Saba, T. (2018). ReVitAge: Realistic virtual heritage taking shadows and sky illumination into account. Journal of Cultural Heritage, 32, 166-175. doi:10.1016/j.culher.2018.01.020Amakawa, J., & Westin, J. (2017). New Philadelphia: using augmented reality to interpret slavery and reconstruction era historical sites. International Journal of Heritage Studies, 24(3), 315-331. doi:10.1080/13527258.2017.1378909Kim, J.-B., & Park, C. (2011). Development of Mobile AR Tour Application for the National Palace Museum of Korea. Lecture Notes in Computer Science, 55-60. doi:10.1007/978-3-642-22021-0_7Barrile, V., Fotia, A., Bilotta, G., & De Carlo, D. (2019). Integration of geomatics methodologies and creation of a cultural heritage app using augmented reality. Virtual Archaeology Review, 10(20), 40. doi:10.4995/var.2019.10361Analysis of Tracking Accuracy for Single-Camera Square-Marker-Based Tracking. In Third Workshop on Virtual and Augmented Reality of the GI-Fachgruppe VR/AR, Koblenz, Germany, 2006http://campar.in.tum.de/Chair/PublicationDetail?pub=pentenrieder2006giCirulis, A., & Brigmanis, K. B. (2013). 3D Outdoor Augmented Reality for Architecture and Urban Planning. Procedia Computer Science, 25, 71-79. doi:10.1016/j.procs.2013.11.009You, S., Neumann, U., & Azuma, R. (1999). Orientation tracking for outdoor augmented reality registration. IEEE Computer Graphics and Applications, 19(6), 36-42. doi:10.1109/38.799738Wither, J., Tsai, Y.-T., & Azuma, R. (2011). Indirect augmented reality. Computers & Graphics, 35(4), 810-822. doi:10.1016/j.cag.2011.04.010Radkowski, R., & Oliver, J. (2013). Natural Feature Tracking Augmented Reality for On-Site Assembly Assistance Systems. Lecture Notes in Computer Science, 281-290. doi:10.1007/978-3-642-39420-1_30Rao, J., Qiao, Y., Ren, F., Wang, J., & Du, Q. (2017). A Mobile Outdoor Augmented Reality Method Combining Deep Learning Object Detection and Spatial Relationships for Geovisualization. Sensors, 17(9), 1951. doi:10.3390/s17091951Hoppe, H., DeRose, T., Duchamp, T., McDonald, J., & Stuetzle, W. (1993). Mesh optimization. Proceedings of the 20th annual conference on Computer graphics and interactive techniques - SIGGRAPH ’93. doi:10.1145/166117.166119Rossignac, J., & Borrel, P. (1993). Multi-resolution 3D approximations for rendering complex scenes. Modeling in Computer Graphics, 455-465. doi:10.1007/978-3-642-78114-8_29Gross, M. H., Staadt, O. G., & Gatti, R. (1996). Efficient triangular surface approximations using wavelets and quadtree data structures. IEEE Transactions on Visualization and Computer Graphics, 2(2), 130-143. doi:10.1109/2945.506225Botsch, M., Pauly, M., Rossl, C., Bischoff, S., & Kobbelt, L. (2006). Geometric modeling based on triangle meshes. ACM SIGGRAPH 2006 Courses on - SIGGRAPH ’06. doi:10.1145/1185657.1185839Pietroni, N., Tarini, M., & Cignoni, P. (2010). Almost Isometric Mesh Parameterization through Abstract Domains. IEEE Transactions on Visualization and Computer Graphics, 16(4), 621-635. doi:10.1109/tvcg.2009.96Khan, D., Yan, D.-M., Ding, F., Zhuang, Y., & Zhang, X. (2018). Surface remeshing with robust user-guided segmentation. Computational Visual Media, 4(2), 113-122. doi:10.1007/s41095-018-0107-yGuidi, G., Russo, M., Ercoli, S., Remondino, F., Rizzi, A., & Menna, F. (2009). A Multi-Resolution Methodology for the 3D Modeling of Large and Complex Archeological Areas. International Journal of Architectural Computing, 7(1), 39-55. doi:10.1260/147807709788549439Remondino, F., & El-Hakim, S. (2006). Image-based 3D Modelling: A Review. The Photogrammetric Record, 21(115), 269-291. doi:10.1111/j.1477-9730.2006.00383.xBruno, F., Bruno, S., De Sensi, G., Luchi, M.-L., Mancuso, S., & Muzzupappa, M. (2010). From 3D reconstruction to virtual reality: A complete methodology for digital archaeological exhibition. Journal of Cultural Heritage, 11(1), 42-49. doi:10.1016/j.culher.2009.02.006Unity, The Photogrammetry Workflowhttps://unity.com/solutions/photogrammetry.Blanco, S., Carrión, B., & Lerma, J. L. (2016). REVIEW OF AUGMENTED REALITY AND VIRTUAL REALITY TECHNIQUES IN ROCK ART. Proceedings of the ARQUEOLÓGICA 2.0 8th International Congress on Archaeology, Computer Graphics, Cultural Heritage and Innovation. doi:10.4995/arqueologica8.2016.3561Behzadan, A. H., & Kamat, V. R. (2010). Scalable Algorithm for Resolving Incorrect Occlusion in Dynamic Augmented Reality Engineering Environments. Computer-Aided Civil and Infrastructure Engineering, 25(1), 3-19. doi:10.1111/j.1467-8667.2009.00601.xTian, Y., Long, Y., Xia, D., Yao, H., & Zhang, J. (2015). Handling occlusions in augmented reality based on 3D reconstruction method. Neurocomputing, 156, 96-104. doi:10.1016/j.neucom.2014.12.081Tian, Y., Guan, T., & Wang, C. (2010). Real-Time Occlusion Handling in Augmented Reality Based on an Object Tracking Approach. Sensors, 10(4), 2885-2900. doi:10.3390/s10040288

MonoPerfCap: Human Performance Capture from Monocular Video

We present the first marker-less approach for temporally coherent 3D performance capture of a human with general clothing from monocular video. Our approach reconstructs articulated human skeleton motion as well as medium-scale non-rigid surface deformations in general scenes. Human performance capture is a challenging problem due to the large range of articulation, potentially fast motion, and considerable non-rigid deformations, even from multi-view data. Reconstruction from monocular video alone is drastically more challenging, since strong occlusions and the inherent depth ambiguity lead to a highly ill-posed reconstruction problem. We tackle these challenges by a novel approach that employs sparse 2D and 3D human pose detections from a convolutional neural network using a batch-based pose estimation strategy. Joint recovery of per-batch motion allows to resolve the ambiguities of the monocular reconstruction problem based on a low dimensional trajectory subspace. In addition, we propose refinement of the surface geometry based on fully automatically extracted silhouettes to enable medium-scale non-rigid alignment. We demonstrate state-of-the-art performance capture results that enable exciting applications such as video editing and free viewpoint video, previously infeasible from monocular video. Our qualitative and quantitative evaluation demonstrates that our approach significantly outperforms previous monocular methods in terms of accuracy, robustness and scene complexity that can be handled.Comment: Accepted to ACM TOG 2018, to be presented on SIGGRAPH 201