Sketched Reality: Sketching Bi-Directional Interactions Between Virtual and Physical Worlds with AR and Actuated Tangible UI

This paper introduces Sketched Reality, an approach that combines AR sketching and actuated tangible user interfaces (TUI) for bidirectional sketching interaction. Bi-directional sketching enables virtual sketches and physical objects to "affect" each other through physical actuation and digital computation. In the existing AR sketching, the relationship between virtual and physical worlds is only one-directional -- while physical interaction can affect virtual sketches, virtual sketches have no return effect on the physical objects or environment. In contrast, bi-directional sketching interaction allows the seamless coupling between sketches and actuated TUIs. In this paper, we employ tabletop-size small robots (Sony Toio) and an iPad-based AR sketching tool to demonstrate the concept. In our system, virtual sketches drawn and simulated on an iPad (e.g., lines, walls, pendulums, and springs) can move, actuate, collide, and constrain physical Toio robots, as if virtual sketches and the physical objects exist in the same space through seamless coupling between AR and robot motion. This paper contributes a set of novel interactions and a design space of bi-directional AR sketching. We demonstrate a series of potential applications, such as tangible physics education, explorable mechanism, tangible gaming for children, and in-situ robot programming via sketching.Comment: UIST 202

RealitySketch: Embedding Responsive Graphics and Visualizations in AR through Dynamic Sketching

We present RealitySketch, an augmented reality interface for sketching interactive graphics and visualizations. In recent years, an increasing number of AR sketching tools enable users to draw and embed sketches in the real world. However, with the current tools, sketched contents are inherently static, floating in mid air without responding to the real world. This paper introduces a new way to embed dynamic and responsive graphics in the real world. In RealitySketch, the user draws graphical elements on a mobile AR screen and binds them with physical objects in real-time and improvisational ways, so that the sketched elements dynamically move with the corresponding physical motion. The user can also quickly visualize and analyze real-world phenomena through responsive graph plots or interactive visualizations. This paper contributes to a set of interaction techniques that enable capturing, parameterizing, and visualizing real-world motion without pre-defined programs and configurations. Finally, we demonstrate our tool with several application scenarios, including physics education, sports training, and in-situ tangible interfaces.Comment: UIST 202

THE SEDUCTION OF THE SIMULATION. 3D MODELLING AND STORYTELLING OF UNREALIZED PERUGIA RAIL STATION

Abstract. The study aims at enhancing and requalifying the area of Fontivegge in Perugia. It seeks to instil in the population a sense of identity and belonging to the place through the digital reconstruction of the first design hypothesis of the station itself, transmitting the cultural heritage of the place with new participatory methods, such as serious games and virtual reality. Starting from the project drawings preserved at the academy of San Luca in Rome, conceived by the architect Antonio Cipolla, the project is reconstructed philologically following historical and archival studies, by interpreting the data collected. Following the reconstruction of the 3D model, evocative mook up images and a virtual reality are created, making the intangible tangible and explorable. Through the virtual simulation of the place and its visualisation, new forms of participation are sought, mending the relationship between territory and population, laying the foundations for a different reading of the area and a cultural heritage accessible and open to all

A Pipeline for the Implementation of Immersive Experience in Cultural Heritage Sites in Sicily

Modern digital technologies allow potentially to explore Cultural Heritage sites in immersive virtual environments. This is surely an advantage for the users that can better experiment and understand a specific site, also before a real visit. This specific approach has gained increasing attention during the extreme conditions of the recent COVID-19 pandemic. In this work, we present the processes that lead to the implementation of an immersive app for different kinds of low and highcost devices, which have been attained in the context of the 3DLab-Sicilia project. 3DLab-Sicilia’s main objective is to sponsor the creation, development, and validation of a sustainable infrastructure that interconnects three main Sicilian centers specialized in augmented and virtual reality. The project gives great importance to the cultural heritage, as well as to the tourism related areas. Despite the presentation of the case study of the Santa Maria La Vetere church, the process of the final app implementation guided by the general pipeline here presented is general and can be applied to other cultural heritage sites

Virtual Community Heritage:An Immersive Approach to Community Heritage

   Our relationship with cultural heritage has been transformed by digital technologies. Opportunities have emerged to preserve and access cultural heritage material while engaging an audience at both regional and global level. Accessibility of technology has enabled audiences to participate in digital heritage curation process. Participatory practices and co-production methodologies have created new relationships between museums and communities, as they are engaged to become active participants in the co-design and co-creation of heritage material. Audiences are more interested in experiences vs services nowadays and museums and heritage organisations have potential to entertain while providing engaging experiences beyond their physical walls. Mixed reality is an emerging method of engagement that has allowed enhanced interaction beyond traditional 3D visualisation models into fully immersive worlds. There is potential to transport audiences to past worlds that enhance their experience and understanding of cultural heritage

From digital drawing to dissemination of the collected data, reflections on the virtual creative process

[EN] The research presented here focuses on the understanding of the act of drawing not only as a means of restitution of the architectural survey but also considering the modernization of the representation techniques. This must be considered as a first step for the creation of virtual environments capable of offering advanced methodologies for the dissemination of the collected data. Expanding the techniques that can be referred to drawing will not only be a necessary process, and one that is already underway, but it will have to strengthen the intrinsic potential of representation. The aim is to communicate and record information with common codes facilitating reading and dissemination. Central to this is the use of drawing as a means of connection between different figures researching a topic. Drawing must be interpreted as the restitution and result of processes of elaboration and codification of the information collected with the survey, that leads us to the modeling and realization of exhaustive elaborations, containing the greatest amount of information useful for the dissemination of the existing architectural heritage and beyond. Its digital evolution, an opportunity to increase the potential and strength of the scientific dissemination of knowledge.This research has been developed as a personal subject by the authors, tools and software used for it are licensed or property of the Didalabs system, Dipartimento di Architettura, University of Florence. The authors would like to thank Professor Giorgio Verdiani for his valuable support, professor Alessandro Camiz and Tom Rankin for sharing their contents, research and experience.Ricci, Y.; Pasquali, A.; Giraudeau, S. (2021). From digital drawing to dissemination of the collected data, reflections on the virtual creative process. EGE Revista de Expresión Gráfica en la Edificación. 0(14):114-124. https://doi.org/10.4995/ege.2021.15650OJS114124014Addison, A.C. 2000. "Emerging trends in virtual heritage." IEEE multimedia, vol. 7(2), pp. 22-25. https://doi.org/10.1109/93.848421Baecker, R.M., Grudin, J., Buxton, W.A.S., Greenberg, S. 1995. Readings in Human-Computer Interaction: Toward the Year 2000, Second Edition, Morgan Kaufmann Publishers, Burlington, Massachusetts, USA.Cioli, F., Ricci, Y. 2020. "L'Officina Profumo-Farmaceutica di Santa Maria Novella. From the point cloud to the virtual reality." Proceedings of the 42th International Conference of Representation Disciplines Teachers. https://doi.org/10.3280/oa-548.107Fazal, S., 2008. GIS Basics, New Age International Pvt Ltd Publishers, India.Guidi G., Russo M., Beraldin J.A. 2010. Acquisizione 3D e modellazione poligonale. Milano: Mcgraw Hill.Pescarmona G. 2020. Augmented Reality and Renaissance Painting - An AR Experience for the Fitzwilliam Museum in Cambridge. Kultur und Informatik: Extended Reality.Ricci Y., Pasquali A., Giorgio Verdiani G. 2019. "A Petrified Petrifying Eyesight: A Story for the Medusa's Heads from Istanbul, Turkey." Proceedings of the 23rd International Conference on Cultural Heritage and New Technologies 2018.Rickman G., 1971. Roman granaries and store buildings. Cambridge University Press.Sutherland, I.E. 1980. Sketchpad: A Man-Machine Graphical Communication System, Garland Publishers, New York, USA.Verdiani G. 2010. Il ritorno all'immagine, nuove procedure image based per il Cultural Heritage. Lulu.com, USA

FROM 3D SURVEY TO DIGITAL REALITY OF A COMPLEX ARCHITECTURE: A DIGITAL WORKFLOW FOR CULTURAL HERITAGE PROMOTION

In recent years, the digitalization and dissemination of historical heritage have become crucial nodes in the preservation and valorization of Cultural Heritage (CH). Technologies such as Unmanned Aerial Vehicle (UAV) and terrestrial photogrammetry, Terrestrial Laser Scanning (TLS) and handheld Simultaneous Localisation and Mapping (SLAM) laser scanning allow the generation of digital models of architecture that can be explored through interactive web platforms, such as those based on WebGL graphic library. These are considered one of the most promising innovations for digitizing and sharing CH site due to their application in a wide range of contexts, promoting new forms of interaction with architecture at different scales. Additionally, the use of geomatic tools allows for a more complete 3D reconstruction and evaluation of the results by comparing different techniques. The article focuses on digitization as a tool for documenting and sharing CH assets, with the aim of developing a replicable prototype platform for an immersive Virtual Tour (VT) of an art collection and the architectural complex in which it is resided. In addition, this paper presents the results of a case study conducted at the Ricci Oddi Gallery of Modern Art in Piacenza, Italy. The source code of the implemented application is available on GitHub to permit replicability for other case studies

Beyond Interactivity and Immersion. A Kinetic Reconceptualization for Virtual Reality and Video Games

peer reviewedIn this paper, we aim to propose a theory for understanding and analyzing the experience of meaning in video games and virtual reality. By focusing on the viewer’s movements on the interface and their transformation into movements within virtual worlds, we will attempt to overcome assumptions based on the concepts of interaction and immersion that are at the core of academic thinking. It is not clear what distinguishes the interaction of digital media from the interpretive interaction of traditional media. Similarly, the concept of immersion risks generating confusion between sensory involvement and emotional and narrative involvement. For this reason, we will understand video games and VR as movement-images: experiences that build their meaning and storytelling according to two interrelated sets of qualities. On the one hand, there are the visual qualities that are widely studied in semiotics and in art history, while on the other hand, there are the kinetic qualities, those concerning movement. By postulating the existence of kinetic diagrams, i.e. relations between movements on the interface and virtual movements, we will identify the way in which video games and virtual reality build emerging narratives, combining visual worlds with figurative, abstract or thematic movements. Some examples of virtual experiences — including the cases of Tetris, Half-Life: Alyx, and Carne y Arena — will allow us to show how kinetic syntax and visual syntax are respectively articulated in video games and virtual reality