Visual Servoing from Deep Neural Networks

We present a deep neural network-based method to perform high-precision, robust and real-time 6 DOF visual servoing. The paper describes how to create a dataset simulating various perturbations (occlusions and lighting conditions) from a single real-world image of the scene. A convolutional neural network is fine-tuned using this dataset to estimate the relative pose between two images of the same scene. The output of the network is then employed in a visual servoing control scheme. The method converges robustly even in difficult real-world settings with strong lighting variations and occlusions.A positioning error of less than one millimeter is obtained in experiments with a 6 DOF robot.Comment: fixed authors lis

Augmented reality application assessment for disseminating rock art

[EN] Currently, marker-based tracking is the most used method to develop augmented reality (AR) applications (apps). However, this method cannot be applied in some complex and outdoor settings such as prehistoric rock art sites owing to the fact that the usage of markers is restricted on site. Thus, natural feature tracking methods have to be used. There is a wide range of libraries to develop AR apps based on natural feature tracking. In this paper, a comparative study of Vuforia and ARToolKit libraries is carried out, analysing factors such as distance, occlusion and lighting conditions that affect user experience in both indoor and outdoor environments, and eventually the app developer. Our analysis confirms that Vuforia¿s user experience indoor is better, faster and flicker-free whether the images are properly enhanced, but it does not work properly on site. Therefore, the development of AR apps for complex outdoor environments such as rock art sites should be performed with ARToolKit.The authors gratefully acknowledge the support from the Spanish Ministerio de Economia y Competitividad to the project HAR2014-59873-R. Similarly, the authors want to express their gratitude to the General Directorate of Culture and Heritage, Conselleria d'Educacio, Investigacio, Cultura i Esport, Generalitat Valenciana for letting us access and carry out research at the archaeological site.Blanco-Pons, S.; Carrión-Ruiz, B.; Lerma, JL. (2018). Augmented reality application assessment for disseminating rock art. Multimedia Tools and Applications. 78(8):10265-10286. https://doi.org/10.1007/s11042-018-6609-xS1026510286788Alahi A., Ortiz R., Vandergheynst P (2012) FREAK: fast retina keypoint. Comput Vis Pattern Recognit 510–517 . doi: https://doi.org/10.1109/CVPR.2012.6247715Amin D, Govilkar S (2015) Comparative study of augmented reality Sdk’S. Int J Comput Sci Appl 5:11–26. https://doi.org/10.1227/01.NEU.0000297044.82035.57ARCore ARCore - Google Developer | ARCore | Google Developers. https://developers.google.com/ar/ . Accessed 26 Jun 2018ARKit ARKit - Apple Developer. https://developer.apple.com/arkit/ . Accessed 26 Jun 2018ARToolkit (2017) ARToolkit. https://archive.artoolkit.org/ . Accessed 2 Oct 2017ARToolkit (2017) About. https://artoolkit.org/about-artoolkit . Accessed 11 Apr 2017ARToolkit (2017) Documentation. https://artoolkit.org/documentation/ . Accessed 12 Apr 2017ArUco ArUco: A minimal library for Augmented Reality applications based on OpenCV | Aplicaciones de la Visión Artificial. https://www.uco.es/investiga/grupos/ava/node/26 . Accessed 19 Apr 2018Azuma R (1997) A survey of augmented reality. Presence Teleoperators Virt Environ 6:355–385 . doi: 10.1.1.30.4999Azuma R, Baillot Y, Feiner S et al (2001) Recent advances in augmented reality. Ieee Comput Graph Appl 34–47. doi: https://doi.org/10.4061/2011/908468Blanco-Novoa O, Fernandez-Carames TM, Fraga-Lamas P, Vilar-Montesinos M (2018) A practical evaluation of commercial industrial augmented reality systems in an industry 4.0 shipyard. IEEE Access 6:1–1. https://doi.org/10.1109/ACCESS.2018.2802699Blanco-Pons S, Carrión-Ruiz B, Lerma JL (2016) Review of augmented reality and virtual reality techniques in rock art. Proc 8th Int Congress Archaeol Comput Graph Cult Herit Innov ‘ARQUEOLÓGICA 2.0L: 176–183Brancati N, Caggianese G, Frucci M et al (2017) Experiencing touchless interaction with augmented content on wearable head-mounted displays in cultural heritage applications. Pers Ubiquitous Comput 21:203–217. https://doi.org/10.1007/s00779-016-0987-8Cagalaban G, Kim S (2010) Multiple object tracking in unprepared environments using combined feature for augmented reality applications. Springer, Berlin, HeidelbergCamera-Calibration Camera Calibration App for Android [ARToolkit]. https://archive.artoolkit.org/documentation/doku.php?id=4_Android:android_camera_calibration . Accessed 16 Oct 2017Carmigniani J, Furht B, Anisetti M et al (2011) Augmented reality technologies, systems and applications. Multimed Tools Appl 51:341–377. https://doi.org/10.1007/s11042-010-0660-6Carrión-Ruiz B, Blanco-Pons S, Lerma JL (2016) Digital image analysis of the visible region through simulation of rock art paintings. Proc 8th Int Congress Archaeol Comput Graph, Cult Heritage Innov ‘ARQUEOLÓGICA 2.0.’: 169–175Chen CY, Chang BR, Sen HP (2014) Multimedia augmented reality information system for museum guidance. Pers Ubiquitous Comput 18:315–322. https://doi.org/10.1007/s00779-013-0647-1CRYENGINE CRYENGINE | The complete solution for next generation game development by Crytek. https://www.cryengine.com/ . Accessed 7 Jun 2017Domingo I, Carrión B, Blanco S, Lerma JL (2015) Evaluating conventional and advanced visible image enhancement solutions to produce digital tracings at el Carche rock art shelter. Digit Appl Archaeol Cult Herit 2:79–88. https://doi.org/10.1016/j.daach.2015.01.001Dos Santos AB, Dourado JB, Bezerra A (2016) ARToolkit and Qualcomm Vuforia: An Analytical Collation. Proc - 18th Symp Virt Augment Real SVR 2016:229–233. https://doi.org/10.1109/SVR.2016.46DroidAR (2017) DroidAR by bitstars. https://bitstars.github.io/droidar/ . Accessed 10 Dec 2017Engine U (2017) Unreal Engine. https://www.unrealengine.com/ . Accessed 10 Oct 2017Fiala M (2005) ARTag, a fiducial marker system using digital techniques. Proc IEEE Comput Soc Conf Comput Vis Pattern Recogn 2:590–596. https://doi.org/10.1109/CVPR.2005.74Fischer J, Eichler M, Bartz D, Straßer W (2007) A hybrid tracking method for surgical augmented reality. Comput Graph 31:39–52. https://doi.org/10.1016/j.cag.2006.09.007González C, Vallejo D, Albusac J, Castro J (2011) Realidad Aumentada. Un enfoque práctico con ARToolKit y Blender. 2–120Gutierrez JM, Molinero MA, Soto-Martín O, Medina CR (2015) Augmented reality technology spreads information about historical graffiti in temple of Debod. Procedia Comput Sci 75:390–397. https://doi.org/10.1016/j.procs.2015.12.262Haladová ZB, Szemzö R, Kovačovský T, Žižka J (2015) Utilizing Multispectral Scanning and Augmented Reality for Enhancement and Visualization of the Wooden Sculpture Restoration Process. Procedia Comput Sci 67:340–347. https://doi.org/10.1016/j.procs.2015.09.278Jamali SS, Shiratuddin MF, Wong KW, Oskam CL (2015) Utilising mobile-augmented reality for learning human anatomy. Procedia - Soc Behav Sci 197:659–668. https://doi.org/10.1016/j.sbspro.2015.07.054Khan D, Ullah S, Rabbi I (2015) Factors affecting the design and tracking of ARToolKit markers. Comput Stand Interf 41:56–66. https://doi.org/10.1016/j.csi.2015.02.006Khan D, Ullah S, Yan D et al (2018) Robust tracking through the design of high quality fiducial markers: an optimization tool for ARToolKit. IEEE Access 4:22421–22433. https://doi.org/10.1109/ACCESS.2018.2801028Kim SL, Suk HJ, Kang JH, et al (2014) Using unity 3D to facilitate mobile augmented reality game development. Internet things (WF-IoT), 2014 IEEE World Forum 21–26 . doi: https://doi.org/10.1109/WF-IoT.2014.6803110Kounavis CD, Kasimati AE, Zamani ED (2012) Enhancing the tourism experience through mobile augmented reality: challenges and prospects. Int J Eng Bus Manag 4:1–6. https://doi.org/10.5772/51644La Delfa GC, Monteleone S, Catania V et al (2016) Performance analysis of visualmarkers for indoor navigation systems. Front Inf Technol Electron Eng 17:730–740. https://doi.org/10.1631/FITEE.1500324Liu S, Ge S, Yu H (2016) Research on Robustness recognition algorithms in augmented reality. 3rd Int Conf Inf Sci Control Eng: 547–552. doi: https://doi.org/10.1109/ICISCE.2016.123Lowe DG (2004) Distinctive image features from scale invariant keypoints. Int J Comput Vis 60:91–11020042. https://doi.org/10.1023/B:VISI.0000029664.99615.94Lytridis C, Tsinakos A, Kazanidis I (2018) ARTutor—an augmented reality platform for interactive distance learning. Educ Sci 8:6. https://doi.org/10.3390/educsci8010006Marchand E, Uchiyama H, Spindler F et al (2016) Pose estimation for augmented reality : a hands-on survey. IEEE Trans Vis Comput Graph 22:2633–2651. https://doi.org/10.1109/TVCG.2015.2513408Martínez R, Villaverde V (2002) La cova dels cavalls en el Barranc de la ValltortaMarto AGR, Sousa AA, de Gonçalves A (2017) DinofelisAR demo augmented reality based on natural features. 12a Conferência Ibérica Sist e Tecnol Informação, Lisboa 64:852–861. https://doi.org/10.1016/j.procs.2015.08.638Moreels P, Perona P (2007) Evaluation of feature detectors and descriptors based on 3D objects. Int J Comput Vis 73:263–284. https://doi.org/10.1007/s11263-006-9967-1Pierdicca R, Frontoni E, Zingaretti P et al (2015) Making visible the invisible. augmented reality visualization for 3D reconstructions of archaeological sites. Augment Virt Real Sec Int Conf AVR 2015 9254:25–37. https://doi.org/10.1007/978-3-319-22888-4Rabbi I, Ullah S, Javed M, Zen K (2014) Analysis of ARToolKit fiducial markers attributes for robust tracking. 1st Int Conf Recent Trends Inf Commun Technol Anal 281–290Radkowski R, Oliver J (2013) Natural feature tracking augmented reality for on-site assembly assistance systems. In: Shumaker R (ed) Virtual, Augmented and Mixed Reality. Systems and Applications. VAMR 2013. Lecture Notes in Computer Science. Springer, Berlin, Heidelberg, pp 281–290Ridel B, Reuter P, Laviole J et al (2014) The revealing flashlight: interactive spatial augmented reality for detail exploration of cultural heritage artifacts. J Comput Cult Herit 7(6):1–6:18. https://doi.org/10.1145/2611376Seo J, Shim J, Choi JH, et al (2011) Enhancing marker-based AR technology. Lect Notes Comput Sci (including Subser Lect Notes Artif Intell Lect Notes Bioinformatics) 6773 LNCS:97–104 . doi: https://doi.org/10.1007/978-3-642-22021-0_12Seo J, Shim J, Choi JH et al (2011) Enhancing marker-based AR technology. In: International conference on virtual and mixed reality. virtual and mixed reality - new trends. Springer, Berlin, Heidelberg, pp 97–104Siltanen S (2015) Diminished reality for augmented reality interior design. Vis Comput 33:1–16. https://doi.org/10.1007/s00371-015-1174-zSörös G, Seichter H, Rautek P, Gröller E (2011) Augmented visualization with natural feature tracking. Proc 10th Int Conf Mob Ubiquitous Multimed 4–12. doi: https://doi.org/10.1145/2107596.2107597Uchiyama H, Marchand E (2012) Object detection and pose tracking for augmented reality: recent approaches. 18th Korea-Japan Jt Work Front Comput Vis 1–8Unity Unity. https://unity3d.com/es . Accessed 12 Oct 2017Vuforia (2017) Vuforia. https://www.vuforia.com/ . Accessed 2 Oct 2017Vuforia (2017) Vuforia-VuMark. https://library.vuforia.com/articles/Training/VuMark . Accessed 4 Apr 2017Vuforia (2017) Image targets. https://library.vuforia.com/articles/Training/Image-Target-Guide . Accessed 11 Apr 2017Wang H, Qin J, Zhang F (2015) A new interaction method for augmented reality based on ARToolKit. 2015 8th Int Congr Image Signal Process 578–583. doi: https://doi.org/10.1109/CISP.2015.7407945Wang G, Wang B, Zhong F et al (2015) Global optimal searching for textureless 3D object tracking. Vis Comput 31:979–988. https://doi.org/10.1007/s00371-015-1098-7Wu S, Oerlemans A, Bakker EM, Lew MS (2017) A comprehensive evaluation of local detectors and descriptors. Signal Process Image Commun 59:150–167. https://doi.org/10.1016/J.IMAGE.2017.06.010Xu Y, Wu Y, Zhou H, View M (2018) Multi-scale Voxel Hashing and Efficient 3D Representation for Mobile Augmented Reality. Cvpr 1618–1625 . doi: https://doi.org/10.1109/CVPRW.2018.0020

Analysis and Development of Augmented Reality Applications for the Dissemination of Cultural Heritage

Tesis por compendio[ES] La RA consiste en la superposición de elementos virtuales sobre el entorno real, de manera que el usuario percibe estos elementos como si formaran parte de la realidad que está visualizando. Las aplicaciones de RA en dispositivos móviles permiten visualizar el contenido virtual a través de la cámara del dispositivo. La RA es una herramienta de divulgación muy potente ya que permite añadir a la realidad cualquier tipo de información, desde un simple texto informativo a un modelo 3D interactivo. Tiene infinitas utilidades, puede servir de guía en un museo, puede mostrar la recreación de un monumento destruido, o como en el caso de estudio aquí presentado, ayudar a la interpretación de pinturas rupestres. Esta tesis parte de la idea de que la RA puede mejorar mucho la interpretación del arte rupestre sin alterar ni dañar las pinturas. Puede servir para atraer a un público mayor, dar a conocer la historia de las pinturas rupestres y que al mismo tiempo el visitante tenga una experiencia mucho más enriquecedora. A lo largo de la tesis se ha estudiado en profundidad la técnica de visualización de RA mediante dispositivos móviles. Se han analizado las diferentes librerías de programación mediante casos de estudio en entornos reales y examinado los factores que pueden afectar al reconocimiento de las pinturas. Se ha desarrollado una aplicación de RA aplicada a un caso real de pinturas rupestres y posteriormente ha sido evaluada por un grupo de personas. Finalmente, se ha estudiado el efecto de la luz solar y sus cambios a lo largo del día sobre el reconocimiento de imágenes en entornos al aire libre. Este trabajo proporciona un punto de partida para el desarrollo de aplicaciones de RA aplicadas a la difusión del patrimonio cultural, especialmente centrado en el arte rupestre, un entorno que sufre de unas dificultades añadidas debido a su localización, dificultad de reconocimiento de puntos característicos en las pinturas y los cambios en la luz solar, problemas que se han tratado de resolver a lo largo del estudio. Las principales conclusiones han sido muy favorables, partiendo de librerías de programación disponibles y gratuitas. Se han podido desarrollar un conjunto de aplicaciones de RA en diferentes lugares. Las valoraciones han sido muy positivas, los usuarios que han probado las aplicaciones afirman que la interpretación de las pinturas les resulta más fácil y consiguen entender mejor el propósito de las mismas. El principal inconveniente encontrado es la falta de conocimiento sobre esta técnica y la pérdida de realismo en algunos casos debido a la oclusión, es decir, que los objetos virtuales no se posicionen por detrás de los objetos reales. La buena noticia es que esta tecnología evoluciona muy rápido y durante el desarrollo de la tesis ha habido avances muy grandes, entre ellos, el desarrollo de nuevas librerías de programación desarrolladas por Google y Apple, que proporcionan las herramientas necesarias para crear aplicaciones muy potentes e immersivas, donde el usuario se sentirá parte de los entornos creados.[CA] La RA consisteix en la superposició d'elements virtuals sobre l'entorn real, de manera que l'usuari percep aquests elements com si formaren part de la realitat que està visualitzant. Les aplicacions de RA en dispositius mòbils permeten visualitzar el contingut virtual a través de la cambra del dispositiu. La RA és una eina de divulgació molt potent ja que permet afegir a la realitat qualsevol tipus d'informació, des d'un simple text informatiu a un model 3D interactiu. Té infinites utilitats, pot servir de guia en un museu, pot mostrar la recreació d'un monument destruït, o com en el cas d'estudi ací presentat, ajudar a la interpretació de pintures rupestres. Aquesta tesi parteix de la idea que la RA pot millorar molt la interpretació de l'art rupestre sense alterar ni danyar les pintures. Pot servir per a atraure a un públic major, donar a conéixer la història de les pintures rupestres i que al mateix temps el visitant tinga una experiència molt més enriquidora. Al llarg de la tesi s'ha estudiat en profunditat la tècnica de visualització de RA mitjançant dispositius mòbils. S'han analitzat les diferents llibreries de programació mitjançant casos d'estudi en entorns reals i analitzat els factors que poden afectar el reconeixement de les pintures. S'ha desenvolupat una aplicació de RA aplicada a un cas real de pintures rupestres i posteriorment ha sigut avaluada per un grup de persones. Finalment, s'ha estudiat l'efecte de la llum solar i els seus canvis al llarg del dia sobre el reconeixement d'imatges en entorns a l'aire lliure. Aquest treball proporciona un punt de partida per al desenvolupament d'aplicacions de RA aplicades a la difusió del patrimoni cultural, especialment centrat en l'art rupestre, un entorn que pateix d'unes dificultats afegides a causa de la seua localització, dificultat de reconeixement de punts característics en les pintures i els canvis en la llum solar, problemes que s'han tractat de resoldre al llarg de l'estudi. Les principals conclusions han sigut molt favorables, partint de llibreries de programació disponibles i gratuïtes. S'han pogut desenvolupar un conjunt d'aplicacions de RA en diferents llocs. Les valoracions han sigut molt positives, els usuaris que han provat les aplicacions afirmen que la interpretació de les pintures els resulta més fàcil i aconsegueixen entendre millor el propòsit d'aquestes. El principal inconvenient trobat és la falta de coneixement sobre aquesta tècnica i la perduda de realisme en alguns casos a causa de l'oclusió, és a dir, que els objectes virtuals no es posicionen per darrere dels objectes reals. La bona notícia és que aquesta tecnologia evoluciona molt ràpid i durant el desenvolupament de la tesi hi ha hagut avanços molt grans, entre ells, el desenvolupament de noves llibreries de programació per Google i Apple, que proporcionen les eines necessàries per a crear aplicacions molt potents i immersives, on l'usuari se sentirà part dels entorns creats.[EN] AR consists of superimposing virtual elements on the real environment, so that the user perceives these elements as if they were part of the reality they are looking at. AR applications on smartphones allow virtual content to be visualised through the device's camera. AR is a very powerful tool for dissemination as it allows any type of information to be added to reality, from a simple informative text to an interactive 3D model. It can be used as a guide in a museum, it can show the recreation of a destroyed monument, or, as in the case study presented here, it can help in the interpretation of cave paintings. This thesis is based on the idea that AR can greatly enhance the interpretation of rock art without affecting or damaging the paintings. It can be used to attract a wider audience, to introduce the history of the rock art paintings and at the same time provide the visitor with a much more enriching experience. Throughout the thesis, the technique of AR visualisation using mobile devices has been studied in-depth. The different programming libraries have been analysed by means of case studies in real environments as well as the factors that can affect the paintings recognition. An AR application applied to a real case of rock art paintings has been developed and subsequently evaluated by a group of people. Finally, the effect of sunlight and its changes throughout the day on image recognition in outdoor environments has been studied. This work provides a starting point for the AR applications development applied to the dissemination of cultural heritage, especially focused on rock art, an environment that suffers from additional difficulties due to its location, the difficulty of characteristic points recognition and changes in sunlight, problems that have been tried to solve throughout the study. The main outcomes have been very favourable, using freely available programming libraries, and it has been possible to develop a set of AR applications in different places. The evaluations have been very positive, with users who have tested the applications confirming that the interpretation of the paintings is easier for them and they can better understand the purpose of the paintings. The major drawback is the lack of knowledge about this technique and the loss of realism in some cases due to occlusion, i.e. the virtual objects are not positioned behind the real objects. The good news is that this technology is evolving very fast and during the development of the thesis there have been great advances, among them, the development of new programming libraries developed by Google and Apple, which provide the necessary tools to create very powerful and immersive applications, where the user will feel part of the virtual environments created.Blanco Pons, S. (2021). Analysis and Development of Augmented Reality Applications for the Dissemination of Cultural Heritage [Tesis doctoral]. Universitat Politècnica de València. https://doi.org/10.4995/Thesis/10251/178895TESISCompendi

Image-based rendering and synthesis

Multiview imaging (MVI) is currently the focus of some research as it has a wide range of applications and opens up research in other topics and applications, including virtual view synthesis for three-dimensional (3D) television (3DTV) and entertainment. However, a large amount of storage is needed by multiview systems and are difficult to construct. The concept behind allowing 3D scenes and objects to be visualized in a realistic way without full 3D model reconstruction is image-based rendering (IBR). Using images as the primary substrate, IBR has many potential applications including for video games, virtual travel and others. The technique creates new views of scenes which are reconstructed from a collection of densely sampled images or videos. The IBR concept has different classification such as knowing 3D models and the lighting conditions and be rendered using conventional graphic techniques. Another is lightfield or lumigraph rendering which depends on dense sampling with no or very little geometry for rendering without recovering the exact 3D-models.published_or_final_versio