Exploitation of Novel Multiplayer Gesture-based Interaction and Virtual Puppetry for Digital Storytelling to Develop Children’s Narrative Skills

In recent years, digital storytelling has demonstrated powerful pedagogical functions by improving creativity, collaboration and intimacy among young children. Saturated with digital media technologies in their daily lives, the young generation demands natural interactive learning environments which offer multimodalities of feedback and meaningful immersive learning experiences. Virtual puppetry assisted storytelling system for young children, which utilises depth motion sensing technology and gesture control as the Human-Computer Interaction (HCI) method, has been proved to provide natural interactive learning experience for single player. In this paper, we designed and developed a novel system that allows multiple players to narrate, and most importantly, to interact with other characters and interactive virtual items in the virtual environment. We have conducted one user experiment with four young children for pedagogical evaluation and another user experiment with five postgraduate students for system evaluation. Our user study shows this novel digital storytelling system has great potential to stimulate learning abilities of young children through collaboration tasks

Exploitation of multiplayer interaction and development of virtual puppetry storytelling using gesture control and stereoscopic devices

With the rapid development of human-computer interaction technologies, the new media generation demands novel learning experiences with natural interaction and immersive experience. Considering that digital storytelling is a powerful pedagogical tool for young children, in this paper, we design an immersive storytelling environment that allows multiple players to use naturally interactive hand gestures to manipulate virtual puppetry for assisting narration. A set of multimodal interaction techniques is presented for a hybrid user interface that integrates existing 3D visualization and interaction devices including head-mounted displays and depth motion sensor. In this system, the young players could intuitively use hand gestures to manipulate virtual puppets to perform a story and interact with props in a virtual stereoscopic environment. We have conducted a user experiment with four young children for pedagogical evaluation, as well as system acceptability and interactivity evaluation by postgraduate students. The results show that our framework has great potential to stimulate learning abilities of young children through collaboration tasks. The stereoscopic head-mounted display outperformed the traditional monoscopic display in a comparison between the two

Semantic framework for interactive animation generation and its application in virtual shadow play performance.

Designing and creating complex and interactive animation is still a challenge in the field of virtual reality, which has to handle various aspects of functional requirements (e.g. graphics, physics, AI, multimodal inputs and outputs, and massive data assets management). In this paper, a semantic framework is proposed to model the construction of interactive animation and promote animation assets reuse in a systematic and standardized way. As its ontological implementation, two domain specific ontologies for the hand-gesture-based interaction and animation data repository have been developed in the context of Chinese traditional shadow play art. Finally, prototype of interactive Chinese shadow play performance system using deep motion sensor device is presented as the usage example

ASSESSMENT OF MEDIA LEARNING BASED ON LEARNING VIRTUAL REALITY IN INDUSTRIAL WORK PRACTICES IN SMK

Purpose of Study: The development of science and technology as it is today will greatly affect the learning process and the preparation of vocational school graduate workforce. Law Number 20 Year 2003 on National Education System explains that the purpose of vocational education is to prepare learners to be ready to work in certain fields. Virtual reality-based learning media can help grade XI SMK students to know the world of industrial work practices so that students of class XI SMK are better prepared in carrying out industrial work practices. Based on the above reasons, the research problem can be formulated as follows: (Baroughi & Zarei, 2013) what is the definition of virtual reality-based learning ?; (Chao liu. 2012) how are the procedures and steps to create virtual reality ?; (Citra parameswari. 2008) what are the advantages and disadvantages of virtual reality-based learning media ?; and (Matandare, m. A. 2018) how is the tendency of utilization of virtual reality-based learning media to industry practice in SMK? This study aims to: (Baroughi & Zarei, 2013) know the meaning of virtual reality; (Chao liu. 2012) know the procedures and steps to create virtual reality; (Citra parameswari. 2008) analyzing the advantages and disadvantages of virtual reality-based learning media; and (Matandare, 2018) to analyze the tendency of utilization of virtual reality-based learning media on industry practice in SMK. Methodology: Research carried out through this literature study yields the following conclusions: (Baroughi & Zarei, 2013) virtual reality is a technology that allows users to interact with a computer-simulated environment; (Chao liu. 2012) procedures and steps to create virtual reality is to record the atmosphere and activities of industrial workplaces and processed into virtual reality-based learning media; (Citra parameswari. 2008) the advantages of virtual reality-based learning media can provide more interesting and interactive learning, while the weakness of virtual reality-based learning media is costly and requires a long time to create; and (Matandare, 2018) the use of virtual reality-based learning media shows an increasing trend in the implementation of industrial work practices. Results: To make a video into a VR Box, prepare a camera with a minimum quality of 8 megapixels. Record the location that will be the object. The location we want to take is the location where the world of work. Implications/Applications: The advantages of using VR to teach educational purposes are similar in many ways with the advantages of using computers or interactive simulations, especially three-dimensional computer simulations. Computer-based simulations have been used for many years in computer-assisted instruction (CAI)

“Masters of Laoting Shadow Play” Experiencing traditional art practice in virtual space

As a child growing up in China, traditional art forms such as shadow plays and paper cutting were popular and spiritual arts enjoyed by everyone; they were also entertainment and decorative art. After a few years studying abroad, I found that such cultural treasures could hardly be found in public or during major festivals; everything seemed to be replaced by digital entertainment. As traditional Chinese art forms are a part of me, it would be sad to see those intangible cultural practices vanish. For this thesis project, (1) I will use field research to interview and understand the current landscape for one living artisan in China who still practices an art form at the brink of extinction, and (2) produce an immersive 360 documentary short that retells his/her stories and recreate a viewer’s experience of these forms from an observer's perspective, as a way of preserving these important traditional arts

The Effects of Intertrial Intervals on Receptive Tasks for Young Children with Autism

A small number of studies have evaluated the effects of different lengths of intertrial intervals on the speed of learning and found that shorter durations of intertrial intervals appear to support faster learning.The purpose of this study was to conduct a systematic replication of the effects of intertrial intervals on receptive labeling by three children (ages 4 to 7 years old) diagnosed on the autism spectrum. A parallel treatments design was used to compare the effects of short intertrial intervals (5-10 seconds) to longer inter-trial intervals (15-20 seconds) during discrete trial teaching. The results were mixed. One participant learned all of the pairs in roughly the same number of trials using both lengths of inter-intervals. The other two participants sometimes learned a pair of pictures with fewer trials using the short intertrial intervals and sometimes using the long intertrial intervals. While participants appeared to learn the tasks in a similar number of teaching trials, all participants learned the tasks in a shorter amount of total teaching time when the short intertrial intervals were used

La realidad virtual como herramienta versátil de investigación, difusión y mediación en las humanidades

[EN] The VESPACE project aims to revive an evening of theatre at the Foire Saint-Germain in Paris in the 18th century, by recreating spaces, atmospheres and theatrical entertainment in virtual reality. The venues of this fair have disappeared without leaving any archaeological traces, so their digital reconstruction requires the use of many different sources, including the expertise of historians, historians of theatre and literature. In this article, we present how we have used video game creation tools to enable the use of virtual reality in three key stages of research in the human sciences and particularly in history or archaeology: preliminary research, scientific dissemination and mediation with the general public. In particular, we detail the methodology used to design a three-dimensional (3D) model that is suitable for both research and virtual reality visualization, meets the standards of scientific work regarding precision and accuracy, and the requirements of a real-time display. This model becomes an environment in which experts can be immersed within their fields of research and expertise, and thus extract knowledge reinforcing the model created –through comments, serendipity and new perspectives– while enabling a multidisciplinary workflow. We also present our tool for annotating and consulting sources, relationships and hypotheses in immersion, called PROUVÉ. This tool is designed to make the virtual reality experience go beyond a simple image and to convey scientific information and theories in the same way an article or a monograph does. Finally, this article offers preliminary feedback on the use of our solutions with three target audiences: the researchers from our team, the broader theatre expert community and the general public.[ES] El proyecto VESPACE tiene como objetivo revivir una velada de teatro en la Foire Saint-Germain de París en el siglo XVIII, recreando espacios, atmósferas y entretenimiento teatral en realidad virtual. Las sedes de esta feria han desaparecido sin dejar ningún rastro arqueológico, por lo que su reconstrucción digital requiere el uso de muy diversas fuentes, entre las que se encuentran expertos historiadores, historiadores del teatro y la literatura. En este artículo presentamos cómo hemos utilizado las herramientas de creación de videojuegos que posibilitan el uso de la realidad virtual en tres etapas clave de la investigación en las ciencias humanas y particularmente en historia o arqueología: investigación preliminar, divulgación científica y mediación con el público en general. En particular, detallamos la metodología utilizada para diseñar un modelo tridimensional (3D) que sea adecuado tanto para la investigación como para la visualización mediante realidad virtual, que cumpla con los estándares del trabajo científico en cuanto a precisión y exactitud, y los requisitos de una visualización en tiempo real. Este modelo se convierte en un entorno en el que los expertos pueden sumergirse dentro de sus campos de investigación y especialización, y así extraer conocimiento reforzando el modelo creado –a través de comentarios, serendipia y nuevas perspectivas– al tiempo que posibilita un flujo de trabajo multidisciplinar. También presentamos nuestra herramienta de anotación y consulta de fuentes, relaciones e hipótesis en inmersión, denominada PROUVÉ. Esta herramienta está diseñada para hacer que la experiencia de la realidad virtual vaya más allá de una simple imagen y que transmita información científica y teorías de la misma manera que lo hace un artículo o una monografía. Finalmente, este artículo ofrece comentarios preliminares sobre el uso de nuestras soluciones con tres públicos objetivo: los investigadores de nuestro equipo, la comunidad de expertos en teatro (más amplia) y el público en general.This research was funded in France by the Agence Nationale de la Recherche, with the CIRESFI project, Grant No. ANR 14-CE31-0017, and the ReSeed project, Grant No. ANR 16-CE38-0005, and a grant from RFI Pays de la Loire "Ouest Industries Créatives" and the École Centrale de Nantes for the AAP 2017 VESPACE project. In the United States, the project is supported by the National Endowment for Humanities, Grant No. NEH HAA-266501-19.François, P.; Leichman, J.; Laroche, F.; Rubellin, F. (2021). Virtual reality as a versatile tool for research, dissemination and mediation in the humanities. Virtual Archaeology Review. 12(25):1-15. https://doi.org/10.4995/var.2021.14880OJS1151225Abiven, M.-M., Laubé, S., Querrec, R., Garlatti, S., Rohou, B., Courgeon, M., Vourch, R., & Roué, J.-P. (2018). Histoire des paysages portuaires et humanités numériques. Retrieved Jan 3, 2021, from https://hal.archives-ouvertes.fr/CFV/hal-01797296v1Barreau, J.-B., Esnault, E., Foucher, J., Six, M., & Le Faou, C. (2020). 3D modelling of a 15th century city gate of Rennes : Portes Mordelaises. Virtual Archaeology Review, 11(22), 41-55. https://doi.org/10.4995/var.2020.12653Barreau, J.-B., Nouviale, F., Gaugne, R., Bernard, Y., Llinares, S., & Gouranton, V. (2015). An immersive virtual sailing on the 18th-century ship Le Boullongne. Presence: Teleoperators and Virtual Environments, 24(3), 201-219. https://doi.org/10.1162/PRES_a_00231Bertino, E., Franzoni, S., Mazzoleni, P., Mussio, P., & Valtoline, S. (2006). Integration of virtual reality and database systems for cultural heritage dissemination. International Journal of Computational Science and Engineering, 2(5), 307-316. https://doi.org/10.1504/IJCSE.2006.014775Brooke, J. (1986). SUS - A quick and dirty usability scale. Redhatch Consulting Ltd. https://hell.meiert.org/core/pdf/sus.pdfBruno, F., Bruno, S., De Sensi, G., Luchi, M.-L., Mancuso, S., Muzzupappa, M. (2010). From 3D reconstruction to virtual reality: A complete methodology for digital archaeological exhibition. Journal of Cultural Heritage, 11(1), 42-49. https://doi.org/10.1016/j.culher.2009.02.006Christou, C., Angus, C., Loscos, C., Dettori, A., & Roussou, M. (2006). A Versatile Large-Scale Multimodal VR System for Cultural Heritage Visualization. VRST'06. VRST'06, Limassol, 133-140. https://doi.org/10.1145/1180495.1180523Clarke, J. (2017). Back to Black : Variable Lighting Levels on the Seventeenth-Century French Stage, Lavoisier and the Enigma of La Pierre philosophale. Insights, 10(2), 2-13.Clergeaud, D., & Guitton, P. (2017). Design of an annotation system for taking notes in virtual reality. 3DTV Conference 2017: Research and Applications in Future 3D Media. 3DTV Conference 2017: Research and Applications in Future 3D Media, Copenhague, 4-8. https://doi.org/10.1109/3DTV.2017.8280398Cotteleer, M. J., Trouton, S., & Dobner, E. (2016). 3D opportunity and the digital thread. Deloitte University Press.Delevoie, C., & Vergnieux, R. (2015). Virtual Retrospect 2013. Ausonius Éditions.Delgado, J. M. D., Oyedele, L., Demian, P., & Beach, T. (2020). A research agenda for augmented and virtual reality in architecture, engineering and construction. Advanced Engineering Informatics, 45, 1-21. https://doi.org/10.1016/j.aei.2020.101122Dominic, J., & Robb, A. (2020). Exploring Effects of Screen-Fixed and World-Fixed Annotation on Navigation in Virtual Reality. IEEE Conference on Virtual Reality and 3D User Interfaces. IEEE Conference on Virtual Reality and 3D User Interfaces, 607-615. https://doi.org/10.1109/VR46266.2020.00083Falconer, L., Burden, D., Cleal, R., Hoyte, R., Phelps, P., Slawson, N., Snashall, N., & Welham, K. (2020). Virtual Avebury : exploring sense of place in a virtual archaeology simulation. Virtual Archaeology Review, 11(23), 50-62. https://doi.org/10.4995/var.2020.12924François, P. (2021). La Foire Saint-Germain vers 1770 : Images et mutations des lieux de spectacle. Actes de la journée d'études Permanences et mutations des espaces du spectacle vivant dans la ville (de 1750 à nos jours). Permanences et mutations des espaces du spectacle vivant dans la ville (de 1750 à nos jours), Bordeaux. In press.François, P., Laroche, F., Rubellin, F., & Leichman, J. (2019). Remonter le temps pour comprendre le passé: L'immersion virtuelle au service des historiens. Conférence Extraction et Gestion de Connaissances 2019 EGC 2019. Conférence Extraction et Gestion de Connaissances EGC 2019, Metz. https://hal.archives-ouvertes.fr/hal-01989653v1Gaugne, R., Gouranton, V., Dumont, G., Chauffaut, A., & Arnaldi, B. (2014). Immersia, an open immersive infrastructure: Doing archaeology in virtual reality. Archeologia e Calcolatori, 5, 180-189.Guerreiro, J., Medeiros, D., Mendes, D., Sousa, M., Jorge, J., Raposo, A., & Santos, I. (2014). Beyond Post-It : Structured Multimedia Annotations for Collaborative VEs. International Conference on Artificial Reality and Telexistence. Eurographics Symposium on Virtual Environments, 55-62. https://doi.org/10.2312/ve.20141365Hardesty, J. L. (2014). Exhibiting library collections online: Omeka in context. New Library World, 115(3/4), 75-86. https://doi.org/10.1108/NLW-01-2014-0013Hvass, J., Larsen, O., Vendelbo, K., Nilsson, N., Nordahl, R., & Serafin, S. (2017). Visual Realism and Presence in a Virtual Reality Game. 3DTV Conference 2017: Research and Applications in Future 3D Media. 3DTV Conference 2017: Research and Applications in Future 3D Media, Copenhague. https://doi.org/10.1109/3DTV.2017.8280421ICOMOS (2017). Principles of Seville. International Principles of Virtual Archaeology. https://icomos.es/wp-content/uploads/2020/06/Seville-Principles-IN-ES-FR.pdfKilteni, K., Groten, R., & Slater, M. (2012). The Sense of Embodiment in Virtual Reality. PRESENCE: Teleoperators and Virtual Environments, 21(4), 373-387. https://doi.org/10.1162/PRES_a_00124Krasniewicz, L. (2000). Immersive imaging technologies for archaeological research. Virtual Reality in Archaeology, 843, 163 169.Laroche, F. (2017). KLM for Heritage [Mémoire d'Habilitation à Diriger des Recherches]. Université de Nantes. https://hal.archives-ouvertes.fr/tel-01712318Laubé, S., Querrec, R., Abiven, M. M., & Garlatti, S. (2018). Lab In Virtuo: Un environnement Virtuel Intelligent (EVI) participatif dédié à l'histoire et aux patrimoines des paysages culturels industriels. DHNord2018 : Matérialités de la recherche en sciences humaines et sociales, Lille, France. https://hal.archives-ouvertes.fr/hal-01950624Mack, Ch. A. (2018). How to write a Good Scientific Paper. SPIE. https://doi.org/10.1117/3.2317707Manuel, A. (2016). Annotation sémantique 2D/3D d'images spatialisées pour la documentation et l'analyse d'objets patrimoniaux. PhD thesis under the supervision of De Luca, L. Arts et Métiers ParisTech. https://hal.archives-ouvertes.fr/tel-01304776Montagud, M., Segura-Garcia, J., De Rus, A., & Fayos Jordán, R. (2020). Towards an Immersive and Accessible Virtual Reconstruction of Theaters from the Early Modern: Bringing Back Cultural Heritage from the Past. In International Conference on Interactive Media Experiences (pp. 143-147). Cornella. https://doi.org/10.1145/3391614.3399390Montandon, A. (1994). Pour une Histoire des traités de savoir-vivre en Europe. Université Blaise Pascal.Martin, I. (2002). Le théâte de la Foire, des tréteaux aux boulevards. Voltaire Foundation.Morris, R. V. (2001). How teachers can conduct historical reenactments in their own schools. Childhood Education, 4(77), 196-203. https://doi.org/10.1080/00094056.2001.10522164Mileto, C., Vegas, F., Cristini, V., & Cabezos Bernal, P. M. (2021). Burial architecture. 3D dissemination study for a selection of Byzantine graves. Virtual Archaeology Review, 12(24), 90-98. https://doi.org/10.4995/var.2021.13187Patte, P. (1782). Essai sur l'architecture théâtrale. Moutard.Reverdy-Médélice, I. (2012). La Valorisation du patrimoine: problèmes méthodologiques, limites et enjeux de la restitution archéologique et historique. PhD thesis under the supervision of Lamboley, J.-L. Université de Grenoble. https://tel.archives-ouvertes.fr/tel-01124042/Rua, H., & Alvito, P. (2011). Living the past: 3D models, virtual reality and game engines as tools for supporting archaeology and the reconstruction of cultural heritage in the case-study of the Roman villa of Casal de Freiria. Journal of Archaeological Science, 38, 3296-3308. https://doi.org/10.1016/j.jas.2011.07.015Shehade, M., & Stylianou-Lambert, T. (2020). Virtual Reality in Museums: Exploring the Experiences of Museum Professionals. Applied Sciences, 10(11), 4031-4051. https://doi.org/10.3390/app10114031Sierra, A., de Prado, G., Ruiz Soler, I., & Codina, F. (2017). Virtual Reality and Archaeological Reconstruction: Be There, Back Then. MW17: Museums and the Web 2017. MW17: Museums and the Web 2017, Cleveland.Takeuchi, H., & Nonaka, I. 2004. Hitotsubashi on Knowledge Management. John Wiley and Sons.Viaud-Delmon, I. (2007). Corps, action et cognition : la réalité virtuelle au défi des sciences cognitives. Intellectica-La revue de l'Association pour la Recherche sur les sciences de la Cognition (ARCo), 45, 37-58. https://doi.org/10.3406/intel.2007.126

Investigating User Experience Using Gesture-based and Immersive-based Interfaces on Animation Learners

Creating animation is a very exciting activity. However, the long and laborious process can be extremely challenging. Keyframe animation is a complex technique that takes a long time to complete, as the procedure involves changing the poses of characters through modifying the time and space of an action, called frame-by-frame animation. This involves the laborious, repetitive process of constantly reviewing results of the animation in order to make sure the movement-timing is accurate. A new approach to animation is required in order to provide a more intuitive animating experience. With the evolution of interaction design and the Natural User Interface (NUI) becoming widespread in recent years, a NUI-based animation system is expected to allow better usability and efficiency that would benefit animation. This thesis investigates the effectiveness of gesture-based and immersive-based interfaces as part of animation systems. A practice-based element of this research is a prototype of the hand gesture interface, which was created based on experiences from reflective practices. An experimental design is employed to investigate the usability and efficiency of gesture-based and immersive-based interfaces in comparison to the conventional GUI/WIMP interface application. The findings showed that gesture-based and immersive-based interfaces are able to attract animators in terms of the efficiency of the system. However, there was no difference in their preference for usability with the two interfaces. Most of our participants are pleasant with NUI interfaces and new technologies used in the animation process, but for detailed work and taking control of the application, the conventional GUI/WIMP is preferable. Despite the awkwardness of devising gesture-based and immersive-based interfaces for animation, the concept of the system showed potential for a faster animation process, an enjoyable learning system, and stimulating interest in a kinaesthetic learning experience

A Functional Analysis of Three Treatments for Pica in a Naturalistic Play Environment

Pica is an eating disorder involving the ingestion of non-food items. In the current study, a functional analysis (FA) screening was conducted to confirm the pica of two subjects diagnosed with autism spectrum disorder (ASD) was maintained by automatic reinforcement. Subsequently, three behavior-analytic treatments (i.e., verbal reprimand, noncontingent reinforcement [NCR], and functional communication training [FCT]) were systematically evaluated in an unstructured play environment using a multi-element design. Findings showed all three treatments decreased pica compared to pre-treatment levels; however, FCT was observed to consistently reduce the rate of pica for both subjects to near-zero levels. Results of this study support the assertion that behavior-analytic treatments for pica might be effective when applied during unstructured play for some subjects