We describe recent developments in the CaveUT software, which supports immersive virtual reality installations based on the Unreal Tournament game engine. CaveUT implements several high-end VR features such as real-time stereoscopy with head and hand tracking. We demonstrate the use of CaveUT in the SAS Cube™, a PC-based CAVE™-like immersive four-screen display. One of the main advantages of the system is to support fully immersive VR while retaining the game engine’s advanced features for interaction and behavioral (or AI) systems. We illustrate the use of CaveUT on two installations: an artistic VR installation and an immersive interactive storytelling system

Cavazza, Marc

Jacobson, Jeffrey

Le Renard, Marc

Lugrin, Jean-Luc

English

We describe recent developments in the CaveUT software, which supports immersive virtual reality installations based on the Unreal Tournament game engine. CaveUT implements several high-end VR features such as real-time stereoscopy with head and hand tracking. We demonstrate the use of CaveUT in the SAS Cube™, a PC-based CAVE™-like immersive four-screen display. One of the main advantages of the system is to support fully immersive VR while retaining the game engine's advanced features for interaction and behavioral (or AI) systems. We illustrate the use of CaveUT on two installations: an artistic VR installation and an immersive interactive storytelling system

Teeside University's Research Repository

The CaveUT System: Immersive Entertainment  Based on a Game Engine  Jeffrey Jacobson1, Marc Le Renard 2, Jean-Luc Lugrin 3 and Marc Cavazza 3 1 Department of Information Sciences, University of Pittsburgh 135, North Bellefield, PA 15260  Jeff@planetjeff.net 2 CLARTE, 4 Rue de l’Ermitage, 53000 Laval, France. Lerenard@esiea-ouest.fr  3 School of Computing, University of Teesside, TS1 3BA Middlesbrough, United Kingdom. j-l.lugrin@tees.ac.uk, m.o.cavazza@tees.ac.uk ABSTRACT We describe recent developments in the CaveUT software, which supports immersive virtual reality installations based on the Unreal Tournament game engine. CaveUT implements several high-end VR features such as real-time stereoscopy with head and hand tracking. We demonstrate the use of CaveUT in the SAS Cube™, a PC-based CAVE™-like immersive four-screen display. One of the main advantages of the system is to support fully immersive VR while retaining the game engine’s advanced features for interaction and behavioral (or AI) systems. We illustrate the use of CaveUT on two installations: an artistic VR installation and an immersive interactive storytelling system.  Categories and Subject Descriptors H5.1 [Multimedia Information Systems] Artificial, Augmented and Virtual Reality - Virtual Reality for Art and Entertainment. General Terms Design and Experimentation. Keywords  Game engine, Immersive Displays, Digital Arts, and Intelligent Virtual Environments. 1. INTRODUCTION Sensory immersion combined with real-time interaction (Virtual Reality, or "VR") has always had great promise for innovative game design, but application development software and libraries for VR research are intended for very diverse applications.  They generally do not provide the animation support, optimized graphics and real-time Physics provided by most game engines. CaveUT1 solves this problem by adding a VR interface to the Unreal Engine2, which preserves the engine's built-in advantages, allows for the re-use of existing game content, and allows for the creation of new content using standard methods.   CaveUT follows the principles developed for the original CAVE™ system [6], supports a variety of immersive display strategies, from low-tech to fully stereoscopic multi-screen display. The invention of  CaveUT is part of a general trend among researchers to take advantage of new technologies developed by the game industry [11], including support for immersive displays with full tracking and stereographic imaging. A game engine can display in a CAVE™-like [6] enclosure through direct modification of the engine itself3, or the combination of modified graphics drivers and top-level code to synchronize the views.  CaveUT takes the latter approach, which avoids many licensing and distribution issues and allows straightforward upgrades when new versions of the game engine are released. CaveUT is a set of open-source freeware modifications, which allows the player to interact with Unreal Tournament, so that s/he sees a unified view across multiple screens, and each screen can be in any orientation to the user. It has been available to the public since 2001 ([7], [8], [9]) and is detailed at its online distribution site1.  The latest version is CaveUT 2004 which takes advantage of the latest release of Unreal Tournament and provides better synchronization between screens.   Recently, in collaboration with members of the ALTERNE project [3], CaveUT has been extended to support stereoscopic display and real-time tracking of head and hand position. These capabilities have been incorporated into CaveUT 2003 and we will soon add them to CaveUT 2004. CaveUT 2003 v2.0 is available by request (jeff@planetjeff.net ) and will be available as part of the AlterneTM software platform for VR Art4  [3]. In further sections, we will illustrate the use of CaveUT through some of the artistic VR installations developed as part of the ALTERNE project (figure 1).                                                                   1 CaveUT. (2004). CaveUT, http://planetjeff.net/ut/CaveUT.html 2 EpicGames.(2004).UnrealTournament, http://www.unrealtournament.com 3 Vizbox. (2004). Vizbox, Inc., http://www.visbox.com/ 4 http://www.alterne.info  Permission to make digital or hard copies of all or part of this work for  Copyright 2004 ACM 1-58113-000-0/00/0004…$5.00.    Figure 1: A Immersive Display: The SAS-CUBETM 2. A CaveUT Primer The development of CaveUT was made possible by the fact that Unreal Tournament (UT) is partially open-source.  UT uses the proprietary Unreal Engine, which handles graphics rendering, animation, physics, networking and a byte-code interpreter, which supports Unreal Script, a Java-like programming language.  CaveUT is a package of original code written in Unreal Script with extensive documentation for its proper use. A multi-screen display based on CaveUT requires a server computer connected by a standard LAN to a number of client computers, at least one for each screen in the display. On each client, the CaveUT code rotates the view according to parameters defined in a configuration file, so that each screen is showing the part of the composite view it is supposed to.  For example, the view in the SAS-Cube™ shown in figure two is produced in part by having one client “look” forward, one look ninety-degrees left, another look ninety-degrees right and the last one look down. CaveUT preserves these rotations relative to the server player's view, so the operator can navigate using standard game controls attached to the server. However, the perspective correction on each screen must be adjusted so that the ideal viewing point for each screen is located at the same point in physical space.  The perspective correction is introduced by a modified OpenGL wrapper library, called VRGL (by Willem de Jonge), which rests “between” OpenGL and the Unreal engine.  VRGL and CaveUT share the same configuration file, which provides the parameters needed for the perspective correction.   3. Rendering Synchronization  In a network environment, load may fall unevenly on each of the client computers, depending on which part of the scene it is rendering, affecting the quality of rendering and the user experience. CaveUT 2.0 solves this problem with the addition of a simple swaplock server, which runs on the server computer, in parallel with the UT game server. The swaplock server starts by broadcasting a “ready” message to the client computers. The signal instructs the VRGL on each computer to wait for a “render” message before displaying the current rendered frame. Each computer then sends the swaplock server a “ready” signal.  The server will wait until it has received a signal from all of the clients, before sending them the “render” signal.  Figure 2: Immersive Visualization In the SAS-CUBETM 4. Tracking CaveUT supports real-time tracking in physical space, using the Intersense™ IS900 system5 or any similar devices. Tracking the player's head allows CaveUT to generate a stable view of the virtual world, allowing the player to move around freely inside the display. CaveUT uses another freeware package, Virtual Reality Peripheral Network6 (released by the Department of Computer at the University of North Carolina at Chapel Hill) to handle input from all control peripherals such as joysticks, buttons, gamepads and the tracking system itself. The VRPN server normalizes data from the control peripherals sends it to the CaveUT code in the UT game server, via a UDP port. The VRPN server broadcasts the user's new location to each one of the UT clients, and the information is received by a VRPN client. Then, the VRPN client sends the tracking information via another UDP port to the VRGL code attached to the UT Client. VRGL uses this information to adjust the perspective correction, in real-time, preserving the perspective illusion of depth. 5. Stereographic Display  CaveUT 2.0 supports stereographic display by using two computers per screen, one to render the left eye view and one to render the right eye view, with an average frame rate of 60 frames/sec per eye in most experiments reported here. The camera view can be offset from the viewer's default configuration by a set value equal to half the inter-pupillary distance. Active stereo requires a single stereographic projector which will alternate between the left and right eye views at 120 frames per second.  The player wears “shutter glasses”, on where each lens alternate between black and clear, also at 120 frames per second. The glasses switch in time with the display, and the result is that each eye gets the view it is supposed to at 60fps—the left view for the left eye and the right view for the right eye.  All of the screens in the composite display must also switch view at exactly the same time, a desirable state called “genlock”.  The original CaveUT system does not have software support for generating an active stereographic image from a single machine.                                                                   5 Intersense. (2004). http://www.isense.com/products/prec/is900 6 http://www.cs.unc.edu/Research/vrpn/ The CaveUT 2.0 installation in the ALTERNE SAS-Cube™ platform uses two computers for each screen, one for each eye view, and uses the DVG (video) cards in their ORAD7 (PC) cluster to mix the two and send it to a single stereographic projector. The DVG cards also handle the genlock synchronization across all screens of the composite display (see figure 3).   6. CaveUT in action: Artistic Installations The stereoscopic version of CaveUT described in this paper has been developed in the course of the ALTERNE project, which is a VR Art project [3] aimed at developing a re-usable platform for the creation of VR Art installation. The main objective of ALTERNE is to define a technological platform for the design of “alternative reality” environments, i.e. virtual worlds whose fundamental behavior can be entirely re-defined in terms, for instance, of laws of Physics. The ALTERNE software platform is thus organized around three main components: i) the UT 2003™ engine for visualization and basic object interaction mechanisms, ii) CaveUT to support immersive stereoscopic visualization and tracking and iii) the “alternative reality” engine, which contains qualitative simulation systems overriding the native Physics engine for specific object categories. ALTERNE installations have provided the first test beds for the stereoscopic version of CaveUT described in this paper. Several artistic briefs have been implemented, which define immersive virtual worlds in which the user experiences novel forms of interaction with the environment.   “Ego.Geo.Graphies”, by Alok Nandi [3] was developed as part of the ALTERNE project and illustrates the use of CaveUT for art installations. The user navigates in an organic world populated by spheres which originate in determinate areas of the environment. The spheres’ behavior depends on the perceived “empathy” of the user, which is a function of her navigation patterns, unknown to her. This behavior manifests itself essentially through the effects that follow collision between spheres, which range from soft sphere merging to explosions propagating to the environment. These effects are under the control of the alternative reality engine, which intercepts collision events and computes alternative forms of causality.  This installation makes uses most of CaveUT’s features, from tracking and object interaction to stereoscopic visualization. User navigation brings the user close to geometrical structures which acquire their full dimension as real stereo 3D objects, requiring her to navigate around them. The spheres traverse the SAS Cube™ volume as floating 3D objects, conferring a high level of realism to the user interaction.  In addition, the ultrasonic tracking implemented in CaveUT supports direct physical interaction with the spheres through the SAS Cube™ gamepad, which can be attracted or pushed back by the user (Figure 4).                                                                   7 http://www.orad.tv  Figure 3: "Ego.geo.Graphies" in The SAS-CUBETM More recently, we have investigated how work developed with UT 2003 could be ported to an immersive context using CaveUT. Interactive storytelling is one of the applications that epitomize what future entertainment systems could look like. In particular, the “Holodeck™” system popularized by the Star Trek series has became a model for research in future immersive interactive storytelling system [12] [13]. Such a system is characterized by the full immersion of the user in a 3D stage, populated by virtual actors, in which the plot unfolds around the user herself. It has thus seemed a natural extension of our research in Interactive Storytelling [2] to adapt it to a fully immersive platform to explore the “Holodeck™” concept. Our storytelling system is developed on top of the Unreal Tournament engine, which made CaveUT a natural choice to port it to a fully immersive context. In the next sections, we briefly summarize the main features of the storytelling system, and discuss how the immersive implementation affects basic interactive storytelling concepts. We then comment upon the technical issues which had to be solved in the context of our CaveUT implementation.  We have named our approach “character-based interactive storytelling” [2] to reflect the specific stance taken with respect to relations between characters and plot. The baseline plot for the interactive narrative (in the example supporting our experiment, the plot consists in a Sitcom-like episode about a group of friends organizing a party) is projected onto individual roles for the virtual actors, formalized as HTN plans. Each virtual actor will thus act independently according to its baseline role in the virtual world. The dynamic interaction between actors is the key principle behind the generation of multiple narratives from a basic storyline. The technical basis for dynamic story generation is that each actor’s role is formalized as a plan to be executed in the virtual stage; several actors will be competing for resources shared in the same environment, these resources being objects or other actors. These conflicts for resources result in plan failure and re-planning, hence creating humorous situations and driving the narrative forward. The same mechanisms support various forms of user intervention in the narrative, which will affect the characters’ actions by failing or modifying some of their goals. Detailed technical descriptions can be found in [5].  Immersion changes the paradigm for user involvement in the interactive narrative. We have originally developed our system for an “interactive TV” philosophy in which the user would influence the story from a god-mode perspective rather than as an actor. We then explored a mixed-reality approach in which the user was simultaneously actor and spectator [4]. In the “Holodeck™” paradigm implemented here through CaveUT (figure 5), the user interacts from a first-person perspective, as a member of the cast. The stereoscopic display confers an increased feeling of realism, as 3D actors traverse the SAS Cube™ space, avoiding the user whose tracked position generates a bounding box. Full immersion and hand/wand tracking supports improved physical intervention on the virtual stage, i.e. the user can influence the story unfolding by removing/hiding key narrative objects (in addition to other forms of interaction, such as influencing actors through speech recognition).  Figure 4:The  “Holodeck™” in The SAS-CUBETM 7. Conclusions Through the implementation of CaveUT, we have shown that the advanced features provided by game engines for both visualization and interaction could be adapted to the context and specific requirements of immersive virtual environments. This has significant implications for entertainment technologies, as it opens the way for the exploration of gameplay in VR and perhaps the search for game genres better suited for the VR setting. One of these new genres in undoubtedly interactive storytelling, whose immersive form endeavors to implement the Holodeck concept. Existing forms of digital entertainment such as VR Art, whose installations used to be developed through custom-made, high-cost, VR systems, can greatly benefit from this approach, which combines sophisticated features within greater accessibility of the software platform to developers. Finally, immersive VR based on game engines constitutes yet another example of the increasing use of game technologies in non-gaming applications. 8. ACKNOWLEDGMENTS Part of the work presented here has been funded by the European Commission through the ALTERNE (IST-38575) Project. The “Ego.Geo Graphies” installation has been authored by Alok Nandi as part of the same project. The authors acknowledge the assistance of Epic Ltd through “PublicVR”. The opinions expressed in this article are the authors’ only and this does not imply any endorsement by Epic. 9. REFERENCES [1] Bowman, D. A. Principles for the Design of Performance-oriented Interaction Techniques. In K. M. Stanney (Ed.), Handbook of Virtual Environments, Mahwa, New Jersey: Lawrence Erlbaum Associates, Inc., Publishers, 2002.   [2] Cavazza, M., Charles, F. and Mead, S.J.. Character-based Interactive Storytelling. IEEE Intelligent Systems, 17, 4, 2002, 17-24. [3] Cavazza, M., Lugrin, J. L., Hartly, S., Libardi, P., Barnes, M. J., LeBras, M., Le Renard, M., Bec, L., Nandi, A. New Ways of Worldmaking: the Alterne Platform for VR Art. ACM Multimedia 2004, New York, USA, 2004. [4] Charles F., Martin O., Cavazza M., Mead S.J., Nandi A. and Marichal X. Compelling Experiences in Mixed Reality Interactive Storytelling. International Conference on Advances in Computer Entertainment Technology, ACE 2004, ACM Press, Singapore, 2004, 32-41. [5] Charles, F. and Cavazza, M. Exploring the Scalability of Character-based Storytelling. Third ACM Joint Conference on Autonomous Agents and Multi-Agent Systems, ACM Press, New York, USA, 2004, 872-879. [6] Cruz-Neira, C., Sandin, D.J. and DeFanti, T.A. Surround-Screen Projection-Based Virtual Reality: The Design and Implementation of the CAVE. Proceedings of the ACM SIGGRAPH 1993 Conference, 1993, 135-142. [7] Jacobson, J. Configuring Multiscreen Immersive Displays With Existing Computer Equipment. Human Factors and Ergonomics Society 46th Annual Meeting, Baltimore, 2002. [8] Jacobson, J. Using CaveUT to Build Immersive Displays With the Unreal Tournament Engine and a PC Cluster, ACM SIGGRAPH 2003 Symposium on Interactive 3D Graphics, Monterrey, California, (April 2003).  [9] Jacobson, J., and Hwang, Z. Unreal Tournament for Immersive Interactive Theater, Communications of the ACM, 45 , 2002, 39-42. [10] Jacobson, J., Redfern, M. S., Furman, J. M., Whiney, L. W., Sparto, P. J., Wilson, J. B., Hodges, L. F.. Balance NAVE; A Virtual Reality Facility for Research and Rehabilitation of Balance Disorders. ACM Virtual Reality Software and Technology (VRST), Banff, Canada, 2001. [11] Lewis, M., and Jacobson, J. Game Engines In Scientific Research. Communications of the ACM, 45, 2002, 27-31. [12] Murray, J.H. Hamlet on the Holodeck: The Future of Narrative in Cyberspace, MIT Press, Cambridge, 1997. [13] Swartout, W., Hill, R., Gratch, J., Johnson, W.L., Kyriakakis, C., LaBore, C.,Lindheim, R., Marsella, S., Miraglia, D., Moore, B., Morie, J., Rickel, J.,Thiebaux, M., Tuch, L., Whitney, R., Douglas, J. Toward the Holodeck: Integrating graphics, sound, character and story. Proceedings 5th International Conference on Autonomous Agents (Agents-01), ACM Press, New York, 2001, 4 

The CaveUT system:Immersive entertainment based on a game engine

https://research.tees.ac.uk/ws/files/6410196/100293.pdf

The CaveUT system:Immersive entertainment based on a game engine

Abstract

Similar works

Full text

Available Versions

Teeside University's Research Repository

Teeside University's Research Repository