Virtual Reality Games for Motor Rehabilitation

This paper presents a fuzzy logic based method to track user satisfaction without the need for devices to monitor users physiological conditions. User satisfaction is the key to any product’s acceptance; computer applications and video games provide a unique opportunity to provide a tailored environment for each user to better suit their needs. We have implemented a non-adaptive fuzzy logic model of emotion, based on the emotional component of the Fuzzy Logic Adaptive Model of Emotion (FLAME) proposed by El-Nasr, to estimate player emotion in UnrealTournament 2004. In this paper we describe the implementation of this system and present the results of one of several play tests. Our research contradicts the current literature that suggests physiological measurements are needed. We show that it is possible to use a software only method to estimate user emotion

HBIM, dibujo 3D y realidad virtual aplicados a sitios arqueológicos y ruinas antiguas

[EN] Data collection, documentation and analysis of the traces of ancient ruins and archaeological sites represent an inestimable value to be handed down to future generations. Thanks to the development of new technologies in the field of computer graphics, Building Information Modelling (BIM), Virtual Reality (VR) and three-dimensional (3D) digital survey, this research proposes new levels of interactivity between users and virtual environments capable of communicating the tangible and intangible values of remains of ancient ruins. In this particular field of development, 3D drawing and digital modelling are based on the application of new Scan-to-HBIM-to-VR specifications capable of transforming simple points (point clouds) into mathematical models and digital information. Thanks to the direct application of novel grades of generation (GOG) and accuracy (GOA) it has been possible to go beyond the creation of complex models for heritage BIM (HBIM) and explore the creation of informative 3D representation composed by subelements (granular HBIM objects) characterized by a further level of knowledge. The value of measurement, 3D drawing and digital modelling have been investigated from the scientific point of view and oriented to the generation of a holistic model able to relate both with architects, engineers, and surveyors but also with archaeologists, restorers and virtual tourists.[ES] La captura de datos, la documentación y el análisis de los restos de las ruinas antiguas y  de  los sitios arqueológicos representan una herencia inestimabile que debe ser transferida a las generaciones futúras. Gracias al desarrollo de las nuevas tecnologías en el campo de los gráficos por ordenador, el modelado de información de la construción (BIM), la realidad virtual (RV) y el levantamiento  digital tridimensional (3D), esta investigación propone nuevos niveles de interacción entre los usuarios y los entornos digitales que pueden comunicar los valores tangibiles e intangibles de los restos de las ruinas antiguas. En este particular ámbito de desarrollo, el dibujo 3D y la modelización digital se basan en la aplicación de las nuevas especificaciones escaneado-a-HBIM-a-RV, capaces de transformar puntos simples (nubes de puntos) en modelos matemáticos e informacción digital. Gracias a la aplicación directa de los GOG (grados of generación) y GOA (grados de exactitud) ha sido posible ir más allá de la creacción de los complejos BIM patrimoniales (HBIM) y explorar la creacción de representaciones 3D, formada por sub-elementos (objetos HBIM granulares) caracterizados por un mayor nivel de conocimiento. El valor de la medición, el dibujo 3D y el modelado digital ha sido investigado desde un enfoque científico y orientado a la generación de un modelo holístico capaz de relacionar tanto a arquitectos, ingenieros y aparejadores con arqueológos, restauradores y turistas virtuales.Banfi, F. (2020). HBIM, 3D drawing and virtual reality for archaeological sites and ancient ruins. Virtual Archaeology Review. 11(23):16-33. https://doi.org/10.4995/var.2020.12416OJS16331123Alby, E., Vigouroux, E., & Elter, R. (2019). Implementation of survey and three-dimensional monitoring of archaeological excavations of the Khirbat al-Dusaq site, Jordan. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XLII-2/W15, 41-47. https://doi.org/10.5194/isprs-archives-XLII-2-W15-41-2019Alia, A., & Cuomo, L. (2017). Bajardo 360: Strategie di rigenerazione per un borgo dell'entroterra ligure (Master's thesis Politecnico di Milano ICAR/21 Urbanistica).Antonopoulou, S., & Bryan, P. (Eds.). (2017). Historic England BIM for Heritage: Developing a Historic Building Information Model. Swindon: Historic England. Retrieved March 10, 2019, from https://historicengland.org.uk/imagesbooks/publications/bim-for-heritage/heag-154-bim-for-heritage/Anzani, A., Baila, A., Penazzi, D., & Binda, L. (2004). Vulnerability study in seismic areas: the role of on-site and archives investigation. In IV International Seminar on Structural Analysis of Historical Constructions (Vol. 2, pp. 1051-1059).Arayici, Y., Counsell, J., Mahdjoubi, L., Nagy, G. A., Hawas, S., & Dweidar, K. (Eds.) (2017). Heritage building information modelling. Abingdon: Routledge. Taylor & Francis. https://doi.org/10.4324/9781315628011Banfi, F. (2019). HBIM generation: extending geometric primitives and bim modelling tools for heritage structures and complex vaulted systems. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XLII-2/W15, 139-148. https://doi.org/10.5194/isprs-archives-XLII-2-W15-139-2019Banfi, F. (2017). BIM orientation: grades of generation and information for different type of analysis and management process. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XLII(2/W5), 57-64. https://doi.org/10.5194/isprs-archives-XLII-2-W5-57-2017Banfi, F., Brumana, R., & Stanga, C. (2019). Extended reality and informative models for the architectural heritage: from scan-to-BIM process to virtual and augmented reality. Virtual Archaeology Review, 10(21), 14-30. https://doi.org/10.4995/var.2019.11923Barba, S., Barbarella, M., Di Benedetto, A., Fiani, M., & Limongiello, M. (2019). Quality assessment of UAV photogrammetric archaeological survey. International Archives of the Photogrammetry, Remote Sensing & Spatial Information Sciences, XLII-2/W9, 93-100. https://doi.org/10.5194/isprs-archives-XLII-2-W9-93-2019Barazzetti, L., Banfi, F., Brumana, R., Gusmeroli, G., Previtali, M., & Schiantarelli, G. (2015). Cloud-to-BIM-to-FEM: Structural simulation with accurate historic BIM from laser scans. Simulation Modelling Practice and Theory, 57, 71-87. https://doi.org/10.1016/j.simpat.2015.06.004Binda, L., Anzani, A., Baila, A., & Penazzi, D. (2004). Indagine conoscitiva, per l'analisi di vulnerabilità, di due centri storici liguri. In XI Cong. Naz. L'Ingegneria Sismica in Italia (pp. 1-8). Padova: Servizi Grafici Editoriali.Bolognesi, C., & Aiello, D. (2019). The secrets of s. Maria delle Grazie: virtual fruition of an iconic milanese architecture. International Archives of the Photogrammetry, Remote Sensing & Spatial Information Sciences, XLII-2/W15, 185-192. https://doi.org/10.5194/isprs-archives-XLII-2-W15-185-2019Brumana, R., Banfi, F., Cantini, L., Previtali, M., & Della Torre, S. (2019). HBIM level of detail-geometry and survey analysis for architectural preservation. International Archives of the Photogrammetry, Remote Sensing & Spatial Information Sciences, XLII-2/W11, 293-299. https://doi.org/10.5194/isprs-archives-XLII-2-W11-293-2019Brumana, R., Condoleo, P., Grimoldi, A., Banfi, F., Landi, A. G., & Previtali, M. (2018). HR LOD based HBIM to detect influences on geometry and shape by stereotomic construction techniques of brick vaults. Applied Geomatics, 10(4), 529-543. https://doi.org/10.1007/s12518-018-0209-3Biagini, C., Capone, P., Donato, V., & Facchini, N. (2016). Towards the BIM implementation for historical building restoration sites. Automation in Construction, 71, 74-86. https://doi.org/10.1016/j.autcon.2016.03.003Böhler, W., & Marbs, A. (2004). 3D scanning and photogrammetry for heritage recording: a comparison. In S. Anders Brandt (Ed.), Proceedings of 12th International Conference on Geoinformatics (pp. 291-298). Gävle, Sweden.Caballero Zoreda, L. (2010). Experiencia metodológica en Arqueología de la Arquitectura de un grupo de investigación. In Actas del congreso Arqueología aplicada al estudio e interpretación de edificios históricos. Últimas tendencias metodológicas (pp. 103-119). Madrid: Ministerio de Cultura.Chiabrando, F., Lo Turco, M., & Rinaudo, F. (2017). Modeling the decay in an HBIM starting from 3D point clouds. a followed approach for cultural heritage knowledge. International Archives of the Photogrammetry, Remote Sensing & Spatial Information Sciences, XLII-2/W5, 605-612. https://doi:10.5194/isprs-archives-XLII-2-W5-605-2017Cogima, C. K., Paiva, P. V. V., Dezen-Kempter, E., Carvalho, M. A. G., & Soibelman, L. (2019). The role of knowledge-based information on BIM for built heritage. In Advances in Informatics and Computing in Civil and Construction Engineering (pp. 27-34). Cham: Springer. https://doi.org/10.1007/978-3-030-00220-6_4Cuca, B., & Barazzetti, L. (2018). Damages from extreme flooding events to cultural heritage and landscapes: water component estimation for Centa River (Albenga, Italy). Advances in Geosciences, 45, 389-395. https://doi.org/10.5194/adgeo-45-389-2018Della Torre, S. (2012). Renovation and post-intervention management. Annales, Series Historia et Sociologia, 22(2), 533-538.Diara, F., & Rinaudo, F. (2019). From reality to parametric models of cultural heritage assets for HBIM. International Archives of the Photogrammetry, Remote Sensing & Spatial Information Sciences, XLII-2/W15, 413-419, https://doi.org/10.5194/isprs-archives-XLII-2-W15-413-2019Dore, C., Murphy, M., McCarthy, S., Brechin, F., Casidy, C., & Dirix, E. (2015). Structural simulations and conservation analysis-historic building information model (HBIM). International Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences, XL-5/W4, 351-357. https://doi:10.5194/isprsarchives-XL-5-W4-351-2015Fai, S., & Rafeiro, J. (2014). Establishing an appropriate level of detail (LoD) for a building information model (BIM)-West Block, Parliament Hill, Ottawa, Canada. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, II-5, 123-130. https://doi:10.5194/isprsannals-II-5-123-2014Fazio, L., & Lo Brutto, M. (2019). 3D Survey for the archaeological study and virtual reconstruction of the "Sanctuary of Isis" in the ancient Lilybaeum (Italy). Virtual Archaeology Review, 11(22), 1-14. https://doi.org/10.4995/var.2020.11928Garagnani, S., Gaucci, A., & Gruška, B. (2016). From the archaeological record to ArchaeoBIM: the case study of the Etruscan temple of Uni in Marzabotto. Virtual Archaeology Review, 7(15), 77-86. https://doi.org/10.4995/var.2016.5846Georgopoulos, A., (2018a). Contemporary Digital Technologies at the Service of Cultural Heritage. In B. Chanda, S. Chaudhuri, S. Chaudhury (Eds.), Heritage Preservation (pp. 1-20). Singapore: Springer. https://doi.org/10.1007/978-981-10-7221-5_1Georgopoulos, A., Ioannidis, C., Soile, S., Tapeinaki, S., Chliverou, R., Moropoulou, A., Tsilimantou, E., & Lampropoulos, K. (2018b). The role of Digital Geometric Documentation in the Rehabilitation of the Tomb of Christ. In 3rd International Congress & Expo Digital Heritage 2018. https://10.1109/DigitalHeritage.2018.8810044Grussenmeyer, P., Landes, T., Voegtle, T., & Ringle, K. (2008). Comparison Methods of Terrestrial Laser Scanning, Photogrammetry and Tacheometry Data for Recording of Cultural Heritage Buildings. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 37(B5): 213-218. https://www.isprs.org/proceedings/XXXVII/congress/5_pdf/38.pdfIoannides, M., Magnenat-Thalmann, N., & Papagiannakis, G. (2017). Mixed Reality and Gamification for Cultural Heritage. Cham: Springer. https://doi.org/10.1007/978-3-319-49607-8Khalil, A., & Stravoravdis, S. (2019). H-BIM and the domains of data investigations of heritage buildings current state of the art. International Archives of the Photogrammetry, Remote Sensing & Spatial Information Sciences, XLII-2/W11, 661-667. https://doi.org/10.5194/isprs-archives-XLII-2-W11-661-2019Korumaz, M., Betti, M., Conti, A., Tucci, G., Bartoli, G., Bonora, V., ... & Fiorini, L. (2017). An integrated Terrestrial Laser Scanner (TLS), Deviation Analysis (DA) and Finite Element (FE) approach for health assessment of historical structures. A minaret case study. Engineering Structures, 153, 224-238. https://doi.org/10.1016/j.engstruct.2017.10.026Kuo, C. L., Cheng, Y. M., Lu, Y. C., Lin, Y. C., Yang, W. B., & Yen, Y. N. (2018). A Framework for Semantic Interoperability in 3D Tangible Cultural Heritage in Taiwan. In Euro-Mediterranean Conference (pp. 21-29). Cham: Springer. https://doi.org/10.1007/978-3-030-01765-1_3Kumar, S. S., & Cheng, J. C. (2015). A BIM-based automated site layout planning framework for congested construction sites. Automation in Construction, 59, 24-37. https://doi.org/10.1016/j.autcon.2015.07.008Lerma, J. L., Navarro, S., Cabrelles, M., & Villaverde, V. (2010). Terrestrial laser scanning and close range photogrammetry for 3D archaeological documentation: the Upper Palaeolithic Cave of Parpalló as a case study. Journal of Archaeological Science, 37(3), 499-507. https://doi.org/10.1016/j.jas.2009.10.011López, F. J., Lerones, P. M., Llamas, J., Gómez-García-Bermejo, J., & Zalama, E. (2018). Linking HBIM graphical and semantic information through the Getty AAT: Practical application to the Castle of Torrelobatón. In IOP Conference Series: Materials Science and Engineering (Vol. 364, No. 1, p. 012100). IOP Publishing. https://doi.org/10.1088/1757-899X/364/1/012100Masiero, A., Chiabrando, F., Lingua, A. M., Marino, B. G., Fissore, F., Guarnieri, A., & Vettore, A. (2019). 3D modeling of Girifalco Fortress. International Archives of the Photogrammetry, Remote Sensing & Spatial Information Sciences, XLII-2/W9, 473-478, https://doi.org/10.5194/isprs-archives-XLII-2-W9-473-2019Nieto Julián, J. E., & Moyano Campos, J. J. (2013). La necesidad de un modelo de información aplicado al patrimonio arquitectónico. In 1er Congreso Nacional BIM-EUBIM. Valencia, Spain. https://pdfs.semanticscholar.org/4979/bf843da620460cdaa4c3520acd5d5ad8a23c.pdfNieto Julián, J., & Moyano Campos, J. (2014). The paramental study on the model of information of historic building or "HBIM Project". Virtual Archaeology Review, 5(11), 73-85. https://doi.org/10.4995/var.2014.4183Parrinello, S., Bercigli, M., & Bursich, D. (2017). From survey to 3D model and from 3D model to "videogame". The virtual reconstruction of a Roman Camp in Masada, Israel. DISEGNARECON, 10(19), 11.1-11.19.Penna, A., Calderini, C., Sorrentino, L., Carocci, C. F., Cescatti, E., Sisti, R., ... & Prota, A. (2019). Damage to churches in the 2016 central Italy earthquakes. Bulletin of Earthquake Engineering, 17(10), 5763-5790. https://doi.org/10.1007/s10518-019-00594-4Piegl, L., & Tiller, W. (2012). The NURBS book. Springer Science & Business Media. Cham: Springer.Previtali, M., Barazzetti, L., Banfi, F., & Roncoroni, F. (2019). Informative content models for infrastructure load testing management: the Azzone Visconti Bridge In Lecco. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XLII-2/W11, 995-100. https://doi.org/10.5194/isprs-Archives-XLII-2-W11-995-2019Pybus, C., Graham, K., Doherty, J., Arellano, N., & Fai, S. (2019). New Realities for Canada's Parliament: a Workflow for Preparing Heritage Bim for Game Engines and Virtual Reality. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XLII-2/W15, 945-952. https://doi.org/10.5194/isprs-archives-XLII-2-W15-945-2019Reina Ortiz, M., Yang, C., Weigert, A., Dhanda, A., Min, A., Gyi, M., ... & Santana Quintero, M. (2019). Integrating heterogeneous datasets in HBIM of decorated surfaces. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XLII-2/W15, 981-988. https://doi.org/10.5194/isprs-archives-XLII-2-W15-981-2019Riveiro, B., & Lindenbergh, R. (Eds.) (2020). Laser Scanning: An Emerging Technology in Structural Engineering. CRC Press. London: Taylor & Francis Group. https://doi.org/10.1201/9781351018869Rossi, C. (2019). Aristotle's mirror: combining digital and material culture. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XLII-2/W11, 1025-1029. https://doi.org/10.5194/isprs-archives-XLII-2-W11-1025-2019, 2019.Russo, M., Remondino, F., & Guidi, G. (2011). Principali tecniche e strumenti per il rilievo tridimensionale in ambito archeologico. Archeologia e calcolatori, 22, 169-198.Rua, H., & Alvito, P. (2011). Living the past: 3D models, virtual reality and game engines as tools for supporting archaeology and the reconstruction of cultural heritage-the case-study of the Roman villa of Casal de Freiria. Journal of Archaeological Science, 38(12), 3296-3308. https://doi.org/10.1016/j.jas.2011.07.015Scianna, A., Gristina, S., & Paliaga, S. (2014). Experimental BIM applications in archaeology: a work-flow. In Euro-Mediterranean Conference (pp. 490-498). Cham: Springer. https://doi.org/10.1007/978-3-319-13695-0_48Stampouloglou, M., Toska, O., Tapinaki, S., Kontogianni, G., Skamantzari, M., & Georgopoulos, A. (2019). 3D documentation and virtual archaeological restoration of Macedonian tombs. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XLII-2/W11, 1073-1080, https://doi.org/10.5194/isprs-archives-XLII-2-W11-1073-2019Saglietto G. (ND). Breve guida illustrata di Bajardo (Imperia). Municipality of Bajardo.Stanga, C., Spinelli, C., Brumana, R., Oreni, D., Valente, R., & Banfi, F. (2017). A n-d virtual notebook about the basilica of S. Ambrogio in Milan: information modeling for the communication of historical phases subtraction process. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XLII-2/W5, 653-660. https://doi.org/10.5194/isprs-archives-XLII-2-W5-653-2017Solarino, S. (2007). Il terremoto del 23 Febbraio 1887 in Liguria Occidentale, Descrizioni, considerazioni e prevenzione 120 anni dopo il grande evento, in "Memoria in occasione della mostra Terremoti: conoscerli per difendersi" . Retrieved from https://docplayer.it/18977788-Il-terremoto-del-23-febbraio-1887-in-liguria-occidentale-descrizioni-considerazioni-e-prevenzione-120-anni-dopo-il-grande-evento.htmlTrizio, I., Savini, F., Giannangeli, A., Boccabella, R., & Petrucci, G. (2019). The Archaeological Analysis of Masonry for the Restoration Project in HBIM. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XLII-2/W9, 715-722. https://doi.org/10.5194/isprs-archives-XLII-2-W9-715-2019Tucci, G., Conti, A., Fiorini, L., Corongiu, M., Valdambrini, N., & Matta, C. (2019). M-BIM: a new tool for the Galleria dell'Accademia di Firenze. Virtual Archaeology Review, 10(21), 40-55. https://doi.org/10.4995/var.2019.11943Quattrini, R., Clementi, F., Lucidi, A., Giannetti, S., & Santoni, A. (2019). From TLS to FE analysis: points cloud exploitation for structural behaviour definition. The San Ciriaco's bell tower. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XLII-2/W15, 957-964. https://doi.org/10.5194/isprs-archives-XLII-2-W15-957-2019Valente, R., Brumana, R., Oreni, D., Banfi, F., Barazzetti, L., & Previtali, M. (2017). Object-oriented approach for 3D archaeological documentation. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XLII-2/W5, 707-712. https://doi.org/10.5194/isprs-archives-XLII-2-W5-707-2017Yang, X., Koehl, M., & Grussenmeyer, P. Mesh-to-BIM: from segmented mesh elements to BIM model with limited parameters. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XLII-2, 1213-1218. https://doi.org/10.5194/isprs-archives-XLII-2-1213-201

Analysis domain model for shared virtual environments

The field of shared virtual environments, which also encompasses online games and social 3D environments, has a system landscape consisting of multiple solutions that share great functional overlap. However, there is little system interoperability between the different solutions. A shared virtual environment has an associated problem domain that is highly complex raising difficult challenges to the development process, starting with the architectural design of the underlying system. This paper has two main contributions. The first contribution is a broad domain analysis of shared virtual environments, which enables developers to have a better understanding of the whole rather than the part(s). The second contribution is a reference domain model for discussing and describing solutions - the Analysis Domain Model

Interaction in an immersive virtual Beijing courtyard house

Courtyard housing had been a standard dwelling type in China for more than 3000 years, which integrated tightly with local customs, aesthetics, philosophy, and natural conditions. As the representative of Chinese courtyard housing, Beijing\u27s style has its unique features including structure, plan layout, and urban form. How to present these features effectively is of great importance to understand Beijing courtyard housing. The current major visualization methods in architecture include physical model, digital imaging, and hand drawing. All of them have two common limitations--small dimensions and non-interaction. As an alternative, VR owns two advantages--immersion and interactivity. In a full-immersive VR environment, such as the C6, users can examine virtual buildings at full-scale and operate models interactively at real-time. Thus, this project attempts to implement an interactive simulation of Beijing courtyard house in C6, and find out if architectural knowledge can be presented through this environment. The methodological steps include VR modeling, interaction planning, and C6 implementation. A four-yard house in Beijing was used as the prototype of VR modeling. By generating the model into six versions with different nodes and textures, it was found that the fewer nodes a model has, the quicker it is in C6. The main interaction mechanism is to demonstrate the main hall\u27s structure interactively through menu selection. The sequence to show the structure is based on its constructional process. Each menu item uses the name of structural components, and by clicking a menu item, the corresponding constructional step is shown in C6. There were five viewers invited to see the simulation and comment on the functionality of full-immersion and interactivity in this product. Overall, the results are positive that the full-immersive and interactive VR environment is potentially effective to present architectural knowledge. A major suggestion from the viewers is that more details can be added in the simulation, such as characters and furniture. Upon the accomplishment of this project, a method to implement architectural simulations efficiently in C6 could be found. In the future, this study could involve more complex interactions such as virtual inhabitants, as a means to show the Chinese culture vividly

HBIM and Virtual Tools: A New Chance to Preserve Architectural Heritage

Nowadays, architectural heritage is increasingly exposed to dangers due to natural disasters or human invasive actions. However, management and conservation represent crucial phases within the life cycle of historical buildings. Unfortunately, the complexity of conservation practices and the lack of knowledge of historic buildings are the cause of an inefficient recovering process in case of emergencies. To overcome this problem, this research aims to ensure the preservation of relevant information through the use of building information modeling (BIM) methodology. By developing historic building information models (HBIMs), it is possible to enhance the architectural heritage. This represents an opportunity to incorporate digital media into the global heritage conservation field. To achieve this goal, a historical castle was selected as a case study; this unique piece of architecture is located in the Piedmont Region, close to city of Turin (Italy). The results show a direct relation between a historical digital model, finalized to the management of architectural and system components, and visualization tools. To conclude, the adoption of this strategy is an effective way to preserve and consult information using advanced visualization techniques based on augmented and virtual reality (AR and VR)

Computer-Assisted Interactive Documentary and Performance Arts in Illimitable Space

This major component of the research described in this thesis is 3D computer graphics, specifically the realistic physics-based softbody simulation and haptic responsive environments. Minor components include advanced human-computer interaction environments, non-linear documentary storytelling, and theatre performance. The journey of this research has been unusual because it requires a researcher with solid knowledge and background in multiple disciplines; who also has to be creative and sensitive in order to combine the possible areas into a new research direction. [...] It focuses on the advanced computer graphics and emerges from experimental cinematic works and theatrical artistic practices. Some development content and installations are completed to prove and evaluate the described concepts and to be convincing. [...] To summarize, the resulting work involves not only artistic creativity, but solving or combining technological hurdles in motion tracking, pattern recognition, force feedback control, etc., with the available documentary footage on film, video, or images, and text via a variety of devices [....] and programming, and installing all the needed interfaces such that it all works in real-time. Thus, the contribution to the knowledge advancement is in solving these interfacing problems and the real-time aspects of the interaction that have uses in film industry, fashion industry, new age interactive theatre, computer games, and web-based technologies and services for entertainment and education. It also includes building up on this experience to integrate Kinect- and haptic-based interaction, artistic scenery rendering, and other forms of control. This research work connects all the research disciplines, seemingly disjoint fields of research, such as computer graphics, documentary film, interactive media, and theatre performance together.Comment: PhD thesis copy; 272 pages, 83 figures, 6 algorithm