Augmented Reality based 3D Human Hands Tracking from Monocular True Images Using Convolutional Neural Network

Precise modeling of hand tracking from monocular moving camera calibration parameters using semantic cues is an active area of research concern for the researchers due to lack of accuracy and computational overheads. In this context, deep learning based framework, i.e. convolutional neural network based human hands tracking as well as recognizing pose of hands in the current camera frame become active research problem. In addition, tracking based on monocular camera needs to be addressed due to updated technology such as Unity3D engine and other related augmented reality plugins. This research aims to track human hands in continuous frame by using the tracked points to draw 3D model of the hands as an overlay in the original tracked image. In the proposed methodology, Unity3D environment was used for localizing hand object in augmented reality (AR). Later, convolutional neural network was used to detect hand palm and hand keypoints based on cropped region of interest (ROI). Proposed method by this research achieved accuracy rate of 99.2% where single monocular true images were used for tracking. Experimental validation shows the efficiency of the proposed methodology.Peer reviewe

Quadcopter Based Object Detection and Localization

This paper presents the development and integration of an X-configuration quadcopter with an IP camera for objectdetection based on the color of an object. A KK2 microcontroller is used to control the quadcopter movements. The controller has builtingyros that provide heading rate information, which are used to control the movement of the quadcopter. The parts of the whole UAV are selectedand integrated. The calibration processes are handled after installing necessary controller codes. This is to make quadcopter fly moothly according to its set commands. The IP camera is then attached to the quadcopter frame. Mobile camera with GPS receiver is used for objectdetection and localization. Images are captured from the IP camera; the three-dimensional position information for the quadcopter is recorded by the GPS receiver, and then the whole data are sent repeatedly from the Mobile to a ground station computer via wireless communication link. Two algorithms are next implemented to acquire and analyze the Images received form the IP camera. The results show that the images was captured and analyzed successfully, and the objects were detected according to their color and its approximate position was reported

Forecasting User Attention During Everyday Mobile Interactions Using Device-Integrated and Wearable Sensors

Visual attention is highly fragmented during mobile interactions, but the erratic nature of attention shifts currently limits attentive user interfaces to adapting after the fact, i.e. after shifts have already happened. We instead study attention forecasting -- the challenging task of predicting users' gaze behaviour (overt visual attention) in the near future. We present a novel long-term dataset of everyday mobile phone interactions, continuously recorded from 20 participants engaged in common activities on a university campus over 4.5 hours each (more than 90 hours in total). We propose a proof-of-concept method that uses device-integrated sensors and body-worn cameras to encode rich information on device usage and users' visual scene. We demonstrate that our method can forecast bidirectional attention shifts and predict whether the primary attentional focus is on the handheld mobile device. We study the impact of different feature sets on performance and discuss the significant potential but also remaining challenges of forecasting user attention during mobile interactions.Comment: 13 pages, 9 figure

Digital twin and its implementations in the civil engineering sector

Digital Twin (DT) concept has recently emerged in civil engineering; however, some problems still need to be addressed. First, DT can be easily confused with Building Information Modelling (BIM) and Cyber-Physical Systems (CPS). Second, the constituents of DT applications in this sector are not well-defined. Also, what the DT can bring to the civil engineering industry is still ambiguous. To address these problems, we reviewed 468 articles related to DT, BIM and CPS, proposed a DT definition and its constituents in civil engineering and compared DT with BIM and CPS. Then we reviewed 134 papers related to DT in the civil engineering sector out of 468 papers in detail. We extracted DT research clusters based on the co-occurrence analysis of paper keywords' and the relevant DT constituents. This research helps establish the state-of-the-art of DT in the civil engineering sector and suggests future DT development

Enhancing heritage fruition through 3D virtual models and augmented reality: An application to Roman artefacts

The spatial characteristics of museum exhibitions may limit visitors' experience of the artefacts on display. In the case of large artefacts, limited space may affect their whole visualization, or inhibit the visualization of the details farthest from the observer. In other cases, the storage of artefacts in distant sites (museums or archaeological areas) may influence their knowledge process or the possibility for comparative analysis. Moreover, the precarious state of preservation of some artefacts, with damaged or missing parts, makes it difficult to perceive their original appearance. To overcome these limitations, we propose an integrated approach based on 3D virtual models and Augmented Reality (AR) to enhance the fruition of artefacts, improving their visualization, analysis and personal/shared knowledge, also by overcoming space and time constraints. The final AR application is an easily accessible tool for most users from a mobile device, used both inside and outside museums, opening new perspectives for fruition. The framework encourages the use of free and open source software and standard devices, to maximize their dissemination and exploit the potential of such technologies, which is far greater than current use in the cultural heritage field. Selected case studies to test and validate the integrated framework are proposed, dealing with some Roman artefacts found in the area of Modena (Italy). The first is a Roman floor mosaic, found in Savignano sul Panaro (near Modena) in 2011, of which less than half of its original 4.5 x 6.9 m surface is preserved. The others are two Roman funerary lion sculptures: the first is one of two lions flanking the main door of Modena Cathedral, and the second, well-preserved but damaged, is housed in the Museo Lapidario Estense of Modena. Finally, the application was tested by museum experts and visitors both inside and outside the museum, and positively assessed

Mejora de la difusión de contenidos culturales del patrimonio mediante modelos virtuales 3dD y realidad aumentada: una aplicación a los artefactos romanos

[EN] The spatial characteristics of museum exhibitions may limit visitors’ experience of the artefacts on display. In the case of large artefacts, limited space may affect their whole visualization, or inhibit the visualization of the details farthest from the observer. In other cases, the storage of artefacts in distant sites (museums or archaeological areas) may influence their knowledge process or the possibility for comparative analysis. Moreover, the precarious state of preservation of some artefacts, with damaged or missing parts, makes it difficult to perceive their original appearance. To overcome these limitations, we propose an integrated approach based on 3D virtual models and Augmented Reality (AR) to enhance the fruition of artefacts, improving their visualization, analysis and personal/shared knowledge, also by overcoming space and time constraints. The final AR application is an easily accessible tool for most users from a mobile device, used both inside and outside museums, opening new perspectives for fruition. The framework encourages the use of free and open source software and standard devices, to maximize their dissemination and exploit the potential of such technologies, which is far greater than current use in the cultural heritage field. Selected case studies to test and validate the integrated framework are proposed, dealing with some Roman artefacts found in the area of Modena (Italy). The first is a Roman floor mosaic, found in Savignano sul Panaro (near Modena) in 2011, of which less than half of its original 4.5 x 6.9 m surface is preserved. The others are two Roman funerary lion sculptures: the first is one of two lions flanking the main door of Modena Cathedral, and the second, well-preserved but damaged, is housed in the Museo Lapidario Estense of Modena. Finally, the application was tested by museum experts and visitors both inside and outside the museum, and positively assessed. Highlights: Digital practice is not understood as a prerogative of a small number of people, but as a tool to guarantee and expand artefact fruition, using standard devices and free and open source software. Experimentation of new settings to re-contextualize artefacts and establish possible links among them, offering engaging and customized experiences to improve their accessibility and enjoyment. Promotion of artefact fruition not only in but also outside museums, such as in a classroom or an open and shared space, opening to new approaches in the fruition of cultural heritage.[ES] Las características espaciales de la exhibición en museos puede limitar en los visitantes la experiencia de los artefactosque se presentan. En el caso de artefactos de gran tamaño, la limitación de espacio puede afectar su visualización completa o inhibir la visualización de los detalles más lejanos al observador. En otros casos, el almacenamiento de artefactos en sitios lejanos y apartados (museos o zonas arqueológicas) puede influir en su proceso de conocimiento o en su análisis comparativo. Es más, el precario estado de conservación de algunos artefactos, con partes dañadas o perdidas, hace difícil percibir su aspecto original. Para superar estas limitaciones, proponemos un enfoque integrado de modelos 3D y realidad aumentada (RA) que mejore el disfrute de los artefactos, mejorando su visualización, análisis y conocimiento personal/compartido, incluso sobrepasando las limitaciones de espacio y tiempo. La aplicación final es una herramienta fácilmente accesible para la mayoría de usuarios mediante un portátil, que se use dentro, pero también fuera de los museos, abriendo nuevas perspectivas de disfrute. El enfoque promueve el uso de software libre y gratuito y herramientas estándar, con vistas a maximizar su amplia distribución y reivindicar las potencialidades de dichas tecnologías, que son superiores a su actual uso en el campo del patrimonio cultural. Se proponen casos de estudio seleccionados para testear y validar el enfoque integrado, a partir de algunos artefactos Romanos encontrados en la zona de Módena (Italia). El primero es un suelo de mosaico Romano, encontrado en Savignano sul Panaro (cerca de Módena) en 2011, que conserva menos de la mitad de sus 4.5 x 6.9 m de superficie originales. Los otros son dos esculturas funerarias romanas de león: el primero es uno de los dos leones que flanquean la puerta principal de la Catedral de Módena, y el segundo, bien conservado pero dañado, se almacena en el Museo Lapidario Estense de Módena. Finalmente, la aplicación se prueba por expertos del museo y visitantes dentro y fuera del museo, y se evalúa positivamente.Gherardini, F. (2019). Enhancing heritage fruition through 3D virtual models and augmented reality: an application to Roman artefacts. Virtual Archaeology Review. 10(21):67-79. https://doi.org/10.4995/var.2019.11918SWORD67791021Antinucci, F. (2014). Comunicare nel Museo. Roma-Bari, Italy: Laterza, pp. 117-124.Bici, M., Guachi, R., Colacicchi, O., D'Ercoli, G., & Campana, F. (2019). Posture evaluation for fragment re-alignment of ancient bronze statues: The case study of the Principe Ellenistico. In F. Cavas-Martínez, B. Eynard, F. Fernández Cañavate, D. Fernández-Pacheco, P. Morer, & V. Nigrelli (Eds.), Advances on mechanics, design engineering and manufacturing II. Lecture notes in mechanical engineering (pp 323-335). Cham: Springer. https://doi.org/10.1007/978-3-030-12346-8_32Black, G. (2005). The Engaging Museum: Developing Museums for Visitor Involvement (pp. 179-210). London, UK:Routledge.Blanco-Pons, S., Carrión-Ruiz, B., & Lerma, J.L. (2019) Augmented reality application assessment for disseminating rock art. Multimedia Tools and Applications, 78(8), 10265-10286. https://doi.org/10.1007/s11042-018-6609-xBrawne, M. (1965). The New Museum: Architecture and Display. New York, US: Praeger.Caspani, S., Brumana, R., Oreni, D., & Previtali, M. (2017). Virtual museums as digital storytellers for dissemination of built environment: Possible narratives and outlooks for appealing and rich encounters with the past. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XLII-2/W5, 113-119. https://doi.org/10.5194/isprs-archives-XLII-2-W5-113-2017Castagnetti, C., Giannini, M., & Rivola, R. (2017). Image-based virtual tours and 3d modeling of past and current ages for the enhancement of archaeological parks: The visual versilia 3d project. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XLII-5/W1, 639-645. https://doi.org/10.5194/isprs-rchives-XLII-5-W1-639-2017Charitos, D., Lepouras, G., Vassilakis, C., Katifori, V., & Halatsi, L. (2000). A method for designing and implementing virtual museums. In R. Hollands (Ed.), Proceedings 7th International UK VR-SIG Conference, Glasgow, UK.Corti, C. (2001). Atlante dei Beni Archeologici della Provincia di Modena. III, Collina e Alta Pianura. Book 1, (pp. 190-191). Sesto Fiorentino (Firenze), Italy: All'Insegna del Giglio.Crespellani, A. (1899). Scavi nel Modenese (1896-1897). AMDST Province Modenesi Serie IV, IX, pp 269-288.Fazio, L., Lo Brutto, M., & Dardanelli, G. (2019). Survey and virtual reconstruction of ancient Roman floors in an archaeological context. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XLII-2/W11, 511-518. https://doi.org/10.5194/isprs-archives-XLII-2-W11-511-2019Fields, N. (2018). Mutina 43 BC: Mark Antony's Struggle for Survival. Oxford, UK: Osprey Publishing.Gherardini, F., Santachiara, M., & Leali, F. (2018). 3D virtual reconstruction and augmented reality visualization of damaged stone sculptures. IOP Conference Series: Materials Science and Engineering, 364(1), 012018. https://doi.org/10.1088/1757-899X/364/1/012018Giordani, N., & Paolozzi Strozzi, G. (2005). Il Museo Lapidario Estense, Catalogo generale. Venezia, Italy: Marsilio Editore.Glover J. (2018). Unity 2018 Augmented Reality Projects: Build four immersive and fun AR applications using ARKit, ARCore, and Vuforia. Birmingham, UK: Packt Publishing Limited.Guidi, G., Trocchianesi, R., Pils, G., Morlando, G., & Seassaro, A. (2010). A Virtual Museum for Design: New forms of interactive fruition. 16th International Conference on Virtual Systems and Multimedia, VSMM 2010 (pp. 242-249). https://doi.org/10.1109/VSMM.2010.5665977Guidi, G., & Remondino, F. (2012). 3D modelling from real data. In C. Alexandru (Ed.), Modeling and Simulation in Engineering (pp. 69-102). https://doi.org/10.5772/30323ICOM, International Council of Museums (n.d.). Museum Definition. Retrieved from https://icom.museum/en/activities/standards-guidelines/museum-definitionKleinhenz, C. (2004). Routledge Revivals: Medieval Italy (2004): An Encyclopedia (Vol. II). New York: Routledge.Krukar, J. (2014). Walk, look, remember: The influence of the gallery's spatial layout on human memory for an art exhibition. Behavioral Sciences, 4(3), 181-201. https://doi.org/10.3390/bs4030181Lanham, M. (2018). Learn ARCore - Fundamentals of Google ARCore: Learn to build augmented reality apps for Android, Unity, and the web with Google ARCore 1.0. Birmingham, UK: Packt Publishing Limited.Linowes, J. & Babilinski, K. (2017). Augmented Reality for Developers. Birmingham, UK: Packt Publishing Limited.Lo Brutto, M., Garraffa, A., Pellegrino, L., & Di Natale, B. (2015). 3D Mosaic documentation using close range photogrammetry. Proceedings of the 1st International Conference on Metrology for Archaeology (pp. 82-87). Benevento, Italy.Ludovico A., (2012). Augmented Art. In Communication Strategies Lab (Ed.), Realtà aumentate: esperienze, strategie e contenuti per l'Augmented Reality (pp. 109-133). Milano, Italy: Apogeo Education.Luigini, A., Brusaporci, S., Vattano, S., & Tata, A. (2019). 3D digital models for a widespread museum: the Renon's "Bauernhöfe". International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XLII-2/W9, 447-453. https://doi.org/10.5194/isprs-archives-XLII-2-W9-447-2019Maícas, J. M., & Viñals, M. J. (2017). Design of a virtual tour for the enhancement of Llíria's architectural and urban heritage and its surroundings. Virtual Archaeology Review, 8(17), 42-48. https://doi.org/10.4995/var.2017.5845Maioli, M. G. (2013). I mosaici di Savignano Il mosaico ritrovato. Indagini archeologiche a Savignano sul Panaro. Firenze, Italy: All'insegna del Giglio. pp 55-65.Malmusi, C. (1930). Museo Lapidario Modenese. Modena, Italy: Franco Cosimo Panini.Noh, Z., Shahrizal, M., & Pan, Z. (2009). A review on augmented reality for virtual heritage system. In M. Chang, R. Kuo et al. (Eds.), Learning by playing. game-based education system design and development (pp. 50-61). Heidelberg, Germany: Springer.Ouimet, C., Gregga, J., Kretz, S., Chandler, C. A., & Hayes, J. (2015). Documentation and dissemination of the sculptural elements of Canada's parliamentary buildings: methodology development and evolution, a case study. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XL-5/W7, 347-352. https://doi.org/10.5194/isprsarchives-XL-5-W7-347-2015Parras, D., Cavas-Martínez, F., Nieto, J., Cañavate, F.J.F., & Fernández-Pacheco, D.G. (2018). Reconstruction by low cost software based on photogrammetry as a reverse engineering process. Lecture Notes in Computer Science, Vol. 10909, 145-154. https://doi.org/10.1007/978-3-319-91581-4_11Polycarpou, C. (2018). The ViMM Definition of a Virtual Museum. Retrieved from https://www.vi-mm.eu/2018/01/10/thevimm-definition-of-a-virtual-museum/Psarra, S. (2005). Spatial culture, way-finding and the educational message: the impact of layout on the spatial, social and educational experiences of visitors to museums and galleries. In S. Macleod (Ed.), Reshaping museum space (pp. 78-94). London, UK: Imprint Routledge.Purnomo, F. A., Santosa, P. I., Hartanto, R., Pratisto, E. H., & Purbayu, A. (2018). Implementation of augmented reality technology in Sangiran Museum with Vuforia. IOP Conference Series: Materials Science and Engineering, 333(1). https://doi.org/10.1088/1757-899X/333/1/012103Rebecchi, F. (1984). Il reimpiego di materiale antico nel Duomo di Modena. In Lanfranco e Wiligelmo. Il Duomo di Modena, catalogo della mostra (luglio 1984, febbraio 1985)(pp. 319-353). Modena, Italy:Edizioni Panini.Rees Lehay, H. (2005). Producing a public for art: gallery space in the twenty-first century. In S. Macleod (Ed.), Reshaping museum space (pp. 9-25). London, UK: Imprint Routledge.Rojas-Sola, J. I., & de la Morena-de la Fuente, E. (2018). Digital 3D reconstruction of Betancourt's historical heritage: The dredging machine in the port of Kronstadt. Virtual Archaeology Review, 9(18), 44-56. https://doi.org/10.4995/var.2018.7946Sandonnini, T. (1983). Cronaca dei restauri del Duomo di Modena (1897-1925) (pp. 145-149). Modena, Italy: Aedes Muratoriana.Santachiara, M., Gherardini, F., & Leali, F. (2018). An augmented reality application for the visualization and the pattern analysis of a Roman mosaic. IOP Conference Series: Materials Science and Engineering, 364(1), 012094. https://doi.org/10.1088/1757-899X/364/1/012094Santos, P., Ritz, M., Fuhrmann, C., & Fellner, D. (2017). 3D mass digitization: A milestone for archaeological documentation. Virtual Archaeology Review, 8(16), 1-11. https://doi.org/10.4995/var.2017.6321Scopigno, R., Corsini, M., Callieri, M., & Dellepiane, M. (2011). Using digital 3D models for study and restoration of cultural heritage artifacts. In F. Stanco, S. Battiato et al (Eds.), Digital imaging for cultural heritage preservation: Analysis, restoration, and reconstruction of ancient artworks (pp. 353-384). Boca Raton, US: CRC Press. 13Screven, C. (1986). Exhibitions and information centers: some principles and approaches. Curator, 29(2), 109-37. https://doi.org/10.1111/j.2151-6952.1986.tb01433.xShackley, M. (1999). Visitor management. In A. Leaske, & I. Yeoman (Eds.), Heritage visitor attractions: An operations management perspective (pp. 69-82). London, UK: Cassells.Skolnick, L. H. (2005). Towards a new museum architecture: Narrative and representation. In S. Macleod (Ed.), Reshaping Museum Space (pp. 118-130). London, UK: Imprint Routledge.Sylaiou, S., Liarokapis, F., Kotsakis, K., & Petros, P. (2009). Virtual museums, a survey and some issues for consideration. Journal of Cultural Heritage, 10(4), 520-528. https://doi.org/10.1016/j.culher.2009.03.003Tsichritzis, D., & Gibbs, S. (1991). Virtual museums and virtual realities. Proceedings of the International Conference on Hypermedia and Interactivity in Museums. Pittsburgh, US.Tucci, G., Cini, D., & Nobile, A. (2011). Effective 3D digitization of archaeological artifacts for interactive virtual museum. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XXXVIII-5W16, 413-420. https://doi.org/10.5194/isprsarchives-XXXVIII-5-W16-413-2011Tucci, G., Bonora, V., Conti, A., & Fiorini, L. (2017). High-quality 3D models and their use in a cultural heritage conservation project. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XLII-2W5, 687-693. https://doi.org/10.5194/isprs-archives-XLII-2-W5-687-2017V-MUST, Virtual Museum Transnational Network. (n.d.) What is a Virtual Museum? Retrieved from http://www.vmust.net/virtual-museums/what-virtual-museumValente, R., Brumana, R., Oreni, D., Banfi, F., Barazzetti, L., & Previtali, M. (2017). Object-oriented approach for 3D archaeological documentation. International Archives of the Photogrammetry, Remote Sensing and Spatial Information ciences, XLII-2-W5, 707-712. https://doi.org/10.5194/isprs-archives-XLII-2-W5-707-2017Vergo, P. (1989). Introduction. In P. Vergo (Ed.), The New Museology (pp.1-5). London, UK: Reaktion Books.Volpe, Y., Furferi, R., Governi, L., & Tennirelli, G. (2014). Computer-based methodologies for semi-automatic 3D model generation from paintings. International Journal of Computer Aided Engineering and Technology, 6(1), 88-112. https://doi.org/10.1504/IJCAET.2014.058012Wilson, P. F., Stott, J., Warnett, J. M., Attridge, A., Smith, M. P., & Williams, M. A. (2018). Evaluation of touchable 3Dprinted replicas in museums. Curator: The Museum Journal, 60(4), 445-465. https://doi.org/10.1111/cura.12244Younes, G., Kahil, R., Jallad, M., Asmar, D., Elhajj, I., Turkiyyah, G., & Al-Harithy, H. (2017). Virtual and augmented reality for rich interaction with cultural heritage sites: A case study from the Roman Theater at Byblos. Digital Applications in Archaeology and Cultural Heritage, 5, 1-9. https://doi.org/10.1016/j.daach.2017.03.00