Fidelity metrics for virtual environment simulations based on spatial memory awareness states

This paper describes a methodology based on human judgments of memory awareness states for assessing the simulation fidelity of a virtual environment (VE) in relation to its real scene counterpart. To demonstrate the distinction between task performance-based approaches and additional human evaluation of cognitive awareness states, a photorealistic VE was created. Resulting scenes displayed on a headmounted display (HMD) with or without head tracking and desktop monitor were then compared to the real-world task situation they represented, investigating spatial memory after exposure. Participants described how they completed their spatial recollections by selecting one of four choices of awareness states after retrieval in an initial test and a retention test a week after exposure to the environment. These reflected the level of visual mental imagery involved during retrieval, the familiarity of the recollection and also included guesses, even if informed. Experimental results revealed variations in the distribution of participants’ awareness states across conditions while, in certain cases, task performance failed to reveal any. Experimental conditions that incorporated head tracking were not associated with visually induced recollections. Generally, simulation of task performance does not necessarily lead to simulation of the awareness states involved when completing a memory task. The general premise of this research focuses on how tasks are achieved, rather than only on what is achieved. The extent to which judgments of human memory recall, memory awareness states, and presence in the physical and VE are similar provides a fidelity metric of the simulation in question

Cognitive transfer of spatial awareness states from immersive virtual environments to reality.

An individual's prior experience will influence how new visual information in a scene is perceived and remembered. Accuracy of memory performance per se is an imperfect reflection of the cognitive activity (awareness states) that underlies performance in memory tasks. The aim of this research is to investigate the effect of varied visual fidelity of training environments on the transfer of training to the real-world after exposure to immersive simulations representing a real-world scene. A between groups experiment was carried out to explore the effect of rendering quality on measurements of location-based recognition memory for objects and associated states of awareness. The immersive simulation, consisted of one room that was either rendered flat-shaded or using radiosity rendering. The simulation was displayed on a stereo head-tracked Head Mounted Display. Post exposure to the synthetic simulation, participants completed a memory recognition task conducted in a real-world scene by physically arranging objects in their physical form in a real world room. Participants also reported one of four states of awareness following object recognition. They were given several options of awareness states that reflected the level of visual mental imagery involved during retrieval, the familiarity of the recollection and related guesses. The scene incorporated objects that 'fitted' into the specific context of the real-world scene, referred to as consistent objects, and objects which were not related to the specific context of the real-world scene, referred to as inconsistent objects. A follow-up study was conducted a week after the initial test. Interestingly, results revealed a higher proportion of correct object recognition associated with mental imagery when participants were exposed to low fidelity flat-shaded training scenes rather than the radiosity rendered ones. Memory psychology indicates that awareness states based on visual imagery require stronger attentional processing in the first instance than those based on familiarity. A tentative claim would therefore be that those immersive environments that are distinctive because of their variation from 'real', such as flat-shaded environments, recruit stronger attentional resources. This additional attentional processing may bring about a change in participants' subjective experiences of 'remembering' when they later transfer the training from that environment into a real-world situation

Fidelity metrics for animation

In this talk, the problem of evaluating the fidelity of animations will be addressed. The concept of plausible simulation has recently be receiving much attention, and I will present a review of this field and discuss how perceptual metrics are necessary to ensure that such animations are truly perceived as real. Then, our recent work in the development of such metrics will be presented. This includes investigations into the perception of collisions and, more recently, psychophysical experiments that examined human sensitivity to dynamic anomalies, leading to the first steps to developing a metric to evaluate the visual fidelity of physically - based animations. Finally, the efforts to develop perceptual metrics for other types of animation, including multi-modal systems and character animation, will also be discusse

Three levels of metric for evaluating wayfinding

Three levels of virtual environment (VE) metric are proposed, based on: (1) users’ task performance (time taken, distance traveled and number of errors made), (2) physical behavior (locomotion, looking around, and time and error classification), and (3) decision making (i.e., cognitive) rationale (think aloud, interview and questionnaire). Examples of the use of these metrics are drawn from a detailed review of research into VE wayfinding. A case study from research into the fidelity that is required for efficient VE wayfinding is presented, showing the unsuitability in some circumstances of common metrics of task performance such as time and distance, and the benefits to be gained by making fine-grained analyses of users’ behavior. Taken as a whole, the article highlights the range of techniques that have been successfully used to evaluate wayfinding and explains in detail how some of these techniques may be applied

Research and Education in Computational Science and Engineering

Over the past two decades the field of computational science and engineering (CSE) has penetrated both basic and applied research in academia, industry, and laboratories to advance discovery, optimize systems, support decision-makers, and educate the scientific and engineering workforce. Informed by centuries of theory and experiment, CSE performs computational experiments to answer questions that neither theory nor experiment alone is equipped to answer. CSE provides scientists and engineers of all persuasions with algorithmic inventions and software systems that transcend disciplines and scales. Carried on a wave of digital technology, CSE brings the power of parallelism to bear on troves of data. Mathematics-based advanced computing has become a prevalent means of discovery and innovation in essentially all areas of science, engineering, technology, and society; and the CSE community is at the core of this transformation. However, a combination of disruptive developments---including the architectural complexity of extreme-scale computing, the data revolution that engulfs the planet, and the specialization required to follow the applications to new frontiers---is redefining the scope and reach of the CSE endeavor. This report describes the rapid expansion of CSE and the challenges to sustaining its bold advances. The report also presents strategies and directions for CSE research and education for the next decade.Comment: Major revision, to appear in SIAM Revie

Sensory Manipulation as a Countermeasure to Robot Teleoperation Delays: System and Evidence

In the field of robotics, robot teleoperation for remote or hazardous environments has become increasingly vital. A major challenge is the lag between command and action, negatively affecting operator awareness, performance, and mental strain. Even with advanced technology, mitigating these delays, especially in long-distance operations, remains challenging. Current solutions largely focus on machine-based adjustments. Yet, there's a gap in using human perceptions to improve the teleoperation experience. This paper presents a unique method of sensory manipulation to help humans adapt to such delays. Drawing from motor learning principles, it suggests that modifying sensory stimuli can lessen the perception of these delays. Instead of introducing new skills, the approach uses existing motor coordination knowledge. The aim is to minimize the need for extensive training or complex automation. A study with 41 participants explored the effects of altered haptic cues in delayed teleoperations. These cues were sourced from advanced physics engines and robot sensors. Results highlighted benefits like reduced task time and improved perceptions of visual delays. Real-time haptic feedback significantly contributed to reduced mental strain and increased confidence. This research emphasizes human adaptation as a key element in robot teleoperation, advocating for improved teleoperation efficiency via swift human adaptation, rather than solely optimizing robots for delay adjustment.Comment: Submitted to Scientific Report

Knowledge Equivalence in Digital Twins of Intelligent Systems

A digital twin contains up-to-date data-driven models of the physical world being studied and can use simulation to optimise the physical world. However, the analysis made by the digital twin is valid and reliable only when the model is equivalent to the physical world. Maintaining such an equivalent model is challenging, especially when the physical systems being modelled are intelligent and autonomous. The paper focuses in particular on digital twin models of intelligent systems where the systems are knowledge-aware but with limited capability. The digital twin improves the acting of the physical system at a meta-level by accumulating more knowledge in the simulated environment. The modelling of such an intelligent physical system requires replicating the knowledge-awareness capability in the virtual space. Novel equivalence maintaining techniques are needed, especially in synchronising the knowledge between the model and the physical system. This paper proposes the notion of knowledge equivalence and an equivalence maintaining approach by knowledge comparison and updates. A quantitative analysis of the proposed approach confirms that compared to state equivalence, knowledge equivalence maintenance can tolerate deviation thus reducing unnecessary updates and achieve more Pareto efficient solutions for the trade-off between update overhead and simulation reliability.Comment: 35 pages, 16 figures. Under review. Submitted to ACM Transactions on Modeling and Computer Simulation (TOMACS

Developing knowledge for real world problem scenarios : using 3D gaming technology within a problem-based learning framework

Problem-based learning is an instructional strategy that emphasises active and experiential learning through problem-solving activity. Using gaming technologies to embed this approach in a three-dimensional (3D) simulation environment provides users with a dynamic, responsive, visually engaging, and cost effective learning experience. Representing real world problems in 3D simulation environments develops knowledge and skills that are applicable to their resolution. The Simulation, User, and Problem-based Learning (SUPL) Design Framework was developed to inform the design of learning environments which develop problem-solving knowledge for real world application. This framework identifies design factors relative to the user, the problem-solving task, and the 3D simulation environment which facilitate the transfer, development, and application of problem-solving knowledge. To assess the validity of the SUPL Design Framework, the Fires in Underground Mines Evacuation Simulator (FUMES) was developed to train mining personnel in emergency evacuation procedures at the Challenger gold mine in South Australia. Two groups of participants representing experienced and novice personnel were utilised to ascertain the effectiveness of FUMES as a training platform in this regard. Findings demonstrated that FUMES accurately represented emergency evacuation scenarios in the Challenger mine. Participants were able to utilise existing real world knowledge in FUMES to resolve emergency evacuation problem-solving tasks and develop new knowledge. The effectiveness of the SUPL Design Framework was also demonstrated, as was the need to design learning environments to meet the learning needs of users rather than merely as static simulations of real world problems. A series of generalisable design guidelines were also established from these findings which could be applied to design problem-based learning simulations in other training contexts