Perceptual Manipulations for Hiding Image Transformations in Virtual Reality

Users of a virtual reality make frequent gaze shifts and head movements to explore their surrounding environment. Saccades are rapid, ballistic, conjugate eye movements that reposition our gaze, and in doing so create large-field motion on our retina. Due to the high speed motion on the retina during saccades, the brain suppresses the visual signals from the eye, a perceptual phenomenon known as the saccadic suppression. These moments of visual blindness can help hide the display graphical updates in a virtual reality. In this dissertation, I investigated how the visibility of various image transformations differed, during combinations of saccade and head rotation conditions. Additionally, I studied how hand and gaze interaction, affected image change discrimination in an inattentional blindness task. I conducted four psychophysical experiments in desktop or head-mounted VR. In the eye tracking studies, users viewed 3D scenes, and were triggered to make a vertical or horizontal saccade. During the saccade an instantaneous translation or rotation was applied to the virtual camera used to render the scene. Participants were required to indicate the direction of these transitions after each trial. The results showed that type and size of the image transformation affected change detectability. During horizontal or vertical saccades, rotations along the roll axis were the most detectable, while horizontal and vertical translations were least noticed. In a second similar study, I added a constant camera motion to simulate a head rotation, and in a third study, I compared active head rotation with a simulated rotation or a static head. I found less sensitivity to transsaccadic horizontal compared to vertical camera shifts during simulated or real head pan. Conversely, during simulated or real head tilt observers were less sensitive to transsaccadic vertical than horizontal camera shifts. In addition, in my multi-interactive inattentional blindness experiment, I compared sensitivity to sudden image transformations when a participant used their hand and gaze to move and watch an object, to when they only watched it move. The results confirmed that when involved in a primary task that requires focus and attention with two interaction modalities (gaze and hand), a visual stimuli can better be hidden than when only one sense (vision) is involved. Understanding the effect of continuous head movement and attention on the visibility of a sudden transsaccadic change can help optimize the visual performance of gaze-contingent displays and improve user experience. Perceptually suppressed rotations or translations can be used to introduce imperceptible changes in virtual camera pose in applications such as networked gaming, collaborative virtual reality and redirected walking. This dissertation suggests that such transformations can be more effective and more substantial during active or passive head motion. Moreover, inattentional blindness during an attention-demanding task provides additional opportunities for imperceptible updates to a visual display

Perceptual Manipulations for Hiding Image Transformations in Virtual Reality

Users of a virtual reality make frequent gaze shifts and head movements to explore their surrounding environment. Saccades are rapid, ballistic, conjugate eye movements that reposition our gaze, and in doing so create large-field motion on our retina. Due to the high speed motion on the retina during saccades, the brain suppresses the visual signals from the eye, a perceptual phenomenon known as the saccadic suppression. These moments of visual blindness can help hide the display graphical updates in a virtual reality. In this dissertation, I investigated how the visibility of various image transformations differed, during combinations of saccade and head rotation conditions. Additionally, I studied how hand and gaze interaction, affected image change discrimination in an inattentional blindness task. I conducted four psychophysical experiments in desktop or head-mounted VR. In the eye tracking studies, users viewed 3D scenes, and were triggered to make a vertical or horizontal saccade. During the saccade an instantaneous translation or rotation was applied to the virtual camera used to render the scene. Participants were required to indicate the direction of these transitions after each trial. The results showed that type and size of the image transformation affected change detectability. During horizontal or vertical saccades, rotations along the roll axis were the most detectable, while horizontal and vertical translations were least noticed. In a second similar study, I added a constant camera motion to simulate a head rotation, and in a third study, I compared active head rotation with a simulated rotation or a static head. I found less sensitivity to transsaccadic horizontal compared to vertical camera shifts during simulated or real head pan. Conversely, during simulated or real head tilt observers were less sensitive to transsaccadic vertical than horizontal camera shifts. In addition, in my multi-interactive inattentional blindness experiment, I compared sensitivity to sudden image transformations when a participant used their hand and gaze to move and watch an object, to when they only watched it move. The results confirmed that when involved in a primary task that requires focus and attention with two interaction modalities (gaze and hand), a visual stimuli can better be hidden than when only one sense (vision) is involved. Understanding the effect of continuous head movement and attention on the visibility of a sudden transsaccadic change can help optimize the visual performance of gaze-contingent displays and improve user experience. Perceptually suppressed rotations or translations can be used to introduce imperceptible changes in virtual camera pose in applications such as networked gaming, collaborative virtual reality and redirected walking. This dissertation suggests that such transformations can be more effective and more substantial during active or passive head motion. Moreover, inattentional blindness during an attention-demanding task provides additional opportunities for imperceptible updates to a visual display

Multimodality with Eye tracking and Haptics: A New Horizon for Serious Games?

The goal of this review is to illustrate the emerging use of multimodal virtual reality that can benefit learning-based games. The review begins with an introduction to multimodal virtual reality in serious games and we provide a brief discussion of why cognitive processes involved in learning and training are enhanced under immersive virtual environments. We initially outline studies that have used eye tracking and haptic feedback independently in serious games, and then review some innovative applications that have already combined eye tracking and haptic devices in order to provide applicable multimodal frameworks for learning-based games. Finally, some general conclusions are identified and clarified in order to advance current understanding in multimodal serious game production as well as exploring possible areas for new applications

Multimodal interactions in virtual environments using eye tracking and gesture control.

Multimodal interactions provide users with more natural ways to interact with virtual environments than using traditional input methods. An emerging approach is gaze modulated pointing, which enables users to perform virtual content selection and manipulation conveniently through the use of a combination of gaze and other hand control techniques/pointing devices, in this thesis, mid-air gestures. To establish a synergy between the two modalities and evaluate the affordance of this novel multimodal interaction technique, it is important to understand their behavioural patterns and relationship, as well as any possible perceptual conflicts and interactive ambiguities. More specifically, evidence shows that eye movements lead hand movements but the question remains that whether the leading relationship is similar when interacting using a pointing device. Moreover, as gaze modulated pointing uses different sensors to track and detect user behaviours, its performance relies on users perception on the exact spatial mapping between the virtual space and the physical space. It raises an underexplored issue that whether gaze can introduce misalignment of the spatial mapping and lead to users misperception and interactive errors. Furthermore, the accuracy of eye tracking and mid-air gesture control are not comparable with the traditional pointing techniques (e.g., mouse) yet. This may cause pointing ambiguity when fine grainy interactions are required, such as selecting in a dense virtual scene where proximity and occlusion are prone to occur. This thesis addresses these concerns through experimental studies and theoretical analysis that involve paradigm design, development of interactive prototypes, and user study for verification of assumptions, comparisons and evaluations. Substantial data sets were obtained and analysed from each experiment. The results conform to and extend previous empirical findings that gaze leads pointing devices movements in most cases both spatially and temporally. It is testified that gaze does introduce spatial misperception and three methods (Scaling, Magnet and Dual-gaze) were proposed and proved to be able to reduce the impact caused by this perceptual conflict where Magnet and Dual-gaze can deliver better performance than Scaling. In addition, a coarse-to-fine solution is proposed and evaluated to compensate the degradation introduced by eye tracking inaccuracy, which uses a gaze cone to detect ambiguity followed by a gaze probe for decluttering. The results show that this solution can enhance the interaction accuracy but requires a compromise on efficiency. These findings can be used to inform a more robust multimodal inter- face design for interactions within virtual environments that are supported by both eye tracking and mid-air gesture control. This work also opens up a technical pathway for the design of future multimodal interaction techniques, which starts from a derivation from natural correlated behavioural patterns, and then considers whether the design of the interaction technique can maintain perceptual constancy and whether any ambiguity among the integrated modalities will be introduced

Design, Control, and Evaluation of a Human-Inspired Robotic Eye

Schulz S. Design, Control, and Evaluation of a Human-Inspired Robotic Eye. Bielefeld: Universität Bielefeld; 2020.The field of human-robot interaction deals with robotic systems that involve humans and robots closely interacting with each other. With these systems getting more complex, users can be easily overburdened by the operation and can fail to infer the internal state of the system or its ”intentions”. A social robot, replicating the human eye region with its familiar features and movement patterns, that are the result of years of evolution, can counter this. However, the replication of these patterns requires hard- and software that is able to compete with the human characteristics and performance. Comparing previous systems found in literature with the human capabili- ties reveal a mismatch in this regard. Even though individual systems solve single aspects, the successful combination into a complete system remains an open challenge. In contrast to previous work, this thesis targets to close this gap by viewing the system as a whole — optimizing the hard- and software, while focusing on the replication of the human model right from the beginning. This work ultimately provides a set of interlocking building blocks that, taken together, form a complete end-to-end solution for the de- sign, control, and evaluation of a human-inspired robotic eye. Based on the study of the human eye, the key driving factors are identified as the success- ful combination of aesthetic appeal, sensory capabilities, performance, and functionality. Two hardware prototypes, each based on a different actua- tion scheme, have been developed in this context. Furthermore, both hard- ware prototypes are evaluated against each other, a previous prototype, and the human by comparing objective numbers obtained by real-world mea- surements of the real hardware. In addition, a human-inspired and model- driven control framework is developed out, again, following the predefined criteria and requirements. The quality and human-likeness of the motion, generated by this model, is evaluated by means of a user study. This frame- work not only allows the replication of human-like motion on the specific eye prototype presented in this thesis, but also promotes the porting and adaption to less equipped humanoid robotic heads. Unlike previous systems found in literature, the presented approach provides a scaling and limiting function that allows intuitive adjustments of the control model, which can be used to reduce the requirements set on the target platform. Even though a reduction of the overall velocities and accelerations will result in a slower motion execution, the human characteristics and the overall composition of the interlocked motion patterns remain unchanged

Development of Visualization Tools for Dynamic Networks and Evaluation of Visual Stability Characteristics

Das dynamische Graphenzeichnen ist das Mittel der Wahl, wenn es um die Analyse und Visualisierung dynamischer Netzwerke geht. Diese Zeichnungen werden oft durch Festhalten aufeinanderfolgender Datenreihen oder “Snapshots” des untersuchten Netzwerkes erzeugt. Für jede von diesen Zeichnungen wird mit Hilfe des ausgewählten Algorithmus eine unabhängige Graphenzeichnung berechnet und die resultierende Sequenz wird dem Benutzer in einer vorbestimmten Reihenfolge präsentiert. Trotz der Einfachheit dieser Methode tauchen bei der dynamischen Graphenzeichnung mit der vorher genannten Strategie Probleme auf. Teilnehmer, Verbindungen und Muster können während der Untersuchung des dynamischen Netzwerkes ihre Position auf der Darstellung verändern. Außerdem neigen dynamische Graphenzeichnungen dazu, fortlaufend Elemente ohne vorhergehende Information hinzuzufügen und zu entfernen. Als Konsequenz ergibt sich die Schwierigkeit, die Entwicklung der Mitglieder des Netzwerkes zu beobachten. Es wurden verschiedene Techniken zur Anpassung von Layouts entwickelt, welche das Ziel haben, die Änderungen der dynamischen Graphenzeichnung zu minimieren. Einige von ihnen schlagen vor, dass die Grundstruktur der Zeichnung jederzeit beibehalten werden muss. Andere wiederum, dass jeder Teilnehmer und jede Beziehung einer fixen Position im Euklidischen Raum zugeordnet werden soll. Eine neu entwickelte Technik schlägt eine Alternative vor: Mehrere Teilnehmer können gleichzeitig einen Knotenpunkt im Euklidischen Raum beanspruchen, solange sie nicht zum selben Zeitpunkt erscheinen. Mehrere Beziehungen können unter den vorgenannten Bedingungen dementsprechend denselben Eckpunkt im Euklidischen Raum beanspruchen. Daraus folgt, dass die dynamische Graphenzeichnung ihre Veränderungen minimiert bis hin zu einem Zustand, in dem es als "visuell stabil" angesehen werden kann. Diese Arbeit zeigt inwieweit die visuelle Stabilität einer dynamischen Graphenzeichnung die Benutzererfahrung und die Effektivität der visuellen Suche beim Verfolgen der Mitglieder oder Netzwerkeigenschaften beeinflusst. Zu diesem Zweck wurde ein Framework zur Unterstützung flexibler Visualisierungstechniken entwickelt. Es diente als Plattform, um existierende Techniken zu bewerten. Solche Bewertungen kombinieren den Gebrauch von Fragebögen, um Informationen über die Nutzererfahrung zu sammeln, ein Eye-Tracking System, um die Augenbewegungen zu erfassen sowie ein neues mathematisches Modell zur Quantifizierung der visuellen Stabilität einer dynamischen Graphenzeichnung. Die daraus folgenden Resultate ergeben, dass es einen Zielkonflikt zwischen der Benutzererfahrung und der Effizienz der visuellen Suche gibt, welche von der visuellen Stabilität der dynamischen Graphenzeichnung abhängt. Einerseits bieten dynamische Graphenzeichnungen mit einem höheren Niveau an visueller Stabilität eine bessere Benutzererfahrung bei Verfolgungsaufgaben, aber eine schlechtere Effizienz bei der visuellen Suche. Andererseits bieten dynamische Graphenzeichnungen mit einer geringeren visuellen Stabilität eine nicht zufriedenstellende Benutzererfahrung, jedoch im Austausch eine Verbesserung der Effizienz der visuellen Suche. Dieses Ergebnis wird genutzt, um visuell stabile Beschreibungen zu entwickeln, die darauf abzielen, die Netzwerkeigenschaften über einen gewissen Zeitraum zu untersuchen. Solche Beschreibungen und Empfehlungen bedienen sich Merkmalen wie Skalierung und Hervorhebung, um die Effizienz der visuellen Suche zu verbessern.Dynamic graph drawings are the metaphor of choice when it comes to the analysis and visualization of dynamic networks. These drawings are often created by capturing a successive sequence of states or “snapshots” from the network under study. Then, for each one of them, a graph drawing is independently computed with the layout algorithm of preference and the resulting sequence is presented to the user in a predefined order. Despite the simplicity of the method, dynamic graph drawings created with the pre- vious strategy possess some problems. Actors, relations or patterns can change their position on the canvas as the dynamic network is explored. Furthermore, dynamic graph drawings tend to constantly add and remove elements without prior information. As a consequence, it is very difficult to observe how the members of the network evolve over time. The scientific community has developed a series of layout adjustment techniques, which aim at minimizing the changes in a dynamic graph drawing. Some of them suggest that the “shape” of the drawing must be maintained at all time. Others that every actor and relation must be assigned to a fixed position in the Euclidean Space. However, a recently developed technique proposes an alternative. Multiple actors can occupy the same node position in the Euclidean Space, as long as they do not appear at the same point in time. Likewise, multiple relations can occupy the same edge position in the Euclidean Space following the principle aforementioned. As the result, a dynamic graph drawing minimizes its changes to a point where it can be perceived as visually stable. This thesis presents how the visual stability of a dynamic graph drawing affects the user experience and the efficiency of the visual search when tracking actors or network attributes over time. For this purpose, a framework to support flexible visualization techniques was developed. It served as the platform to evaluate existing layout ad- justment techniques. Such an evaluation combined the use of questionnaires to gather information about the user experience; an eye-tracking device to record the eye move- ments and a new mathematical model to appropriately quantify the visual stability of dynamic graph drawings. The results obtained suggest that there is a trade-off between the user experience and the efficiency of the visual search, which depends on the visual stability of a dynamic graph drawing. On the one hand, dynamic graph drawings with higher levels of visual stability provide a satisfying user experience in tracking tasks. Nonetheless, they are inefficient in terms of the visual search. On the other hand, dynamic graph drawings with lower levels of visual stability, do not provide a satisfying user experience in tracking tasks but considerably improve the efficiency of the visual search. These findings were used to develop visually stable metaphors, aiming at exploring network attributes over time. Such metaphors rely on features like scaling or highlighting to improve the efficiency of the visual search

Eye tracking and avatar-mediated communication in immersive collaborative virtual environments

The research presented in this thesis concerns the use of eye tracking to both enhance and understand avatar-mediated communication (AMC) performed by users of immersive collaborative virtual environment (ICVE) systems. AMC, in which users are embodied by graphical humanoids within a shared virtual environment (VE), is rapidly emerging as a prevalent and popular form of remote interaction. However, compared with video-mediated communication (VMC), which transmits interactants’ actual appearance and behaviour, AMC fails to capture, transmit, and display many channels of nonverbal communication (NVC). This is a significant hindrance to the medium’s ability to support rich interpersonal telecommunication. In particular, oculesics (the communicative properties of the eyes), including gaze, blinking, and pupil dilation, are central nonverbal cues during unmediated social interaction. This research explores the interactive and analytical application of eye tracking to drive the oculesic animation of avatars during real-time communication, and as the primary method of experimental data collection and analysis, respectively. Three distinct but interrelated questions are addressed. First, the thesis considers the degree to which quality of communication may be improved through the use of eye tracking, to increase the nonverbal, oculesic, information transmitted during AMC. Second, the research asks whether users engaged in AMC behave and respond in a socially realistic manner in comparison with VMC. Finally, the degree to which behavioural simulations of oculesics can both enhance the realism of virtual humanoids, and complement tracked behaviour in AMC, is considered. These research questions were investigated over a series of telecommunication experiments investigating scenarios common to computer supported cooperative work (CSCW), and a further series of experiments investigating behavioural modelling for virtual humanoids. The first, exploratory, telecommunication experiment compared AMC with VMC in a three-party conversational scenario. Results indicated that users employ gaze similarly when faced with avatar and video representations of fellow interactants, and demonstrated how interaction is influenced by the technical characteristics and limitations of a medium. The second telecommunication experiment investigated the impact of varying methods of avatar gaze control on quality of communication during object-focused multiparty AMC. The main finding of the experiment was that quality of communication is reduced when avatars demonstrate misleading gaze behaviour. The final telecommunication study investigated truthful and deceptive dyadic interaction in AMC and VMC over two closely-related experiments. Results from the first experiment indicated that users demonstrate similar oculesic behaviour and response in both AMC and VMC, but that psychological arousal is greater following video-based interaction. Results from the second experiment found that the use of eye tracking to drive the oculesic behaviour of avatars during AMC increased the richness of NVC to the extent that more accurate estimation of embodied users’ states of veracity was enabled. Rather than directly investigating AMC, the second series of experiments addressed behavioural modelling of oculesics for virtual humanoids. Results from the these experiments indicated that oculesic characteristics are highly influential to the perceived realism of virtual humanoids, and that behavioural models are able to complement the use of eye tracking in AMC. The research presented in this thesis explores AMC and eye tracking over a range of collaborative and perceptual studies. The overall conclusion is that eye tracking is able to enhance AMC towards a richer medium for interpersonal telecommunication, and that users’ behaviour in AMC is no less socially ‘real’ than that demonstrated in VMC. However, there are distinct differences between the two communication mediums, and the importance of matching the characteristics of a planned communication with those of the medium itself is critical

Torque #2: The Act of Reading

This book is the result of a 12 month research process, including a residency at Tate Liverpool, an exhibition in London's Furtherfield Gallery, and a symposium and performance event taking place at FACT in Liverpool in 2015. The book is aimed as a popular primer on reading discourse from a range of disciplines, but also as a performative document on what interdisciplinary brings to such discourse. It features new material from several world-leading practitioners from a range of disciplines, including media theorists N. Katherine Hayles and Soenke Zehle, literary theorist Garrett Stewart, political theorists Esther Leslie and Nina Power, the poet Charles Bernstein, artists Tim Etchells and Erica Scourti, and clinical neuroscientist Alex Leff, along with a selection of early career academics. Each of these contributors were chosen according to the prescience of their current work to the questions: what does means to read today? and, how is reading changing as a result of current political and technological conditions? The content for the book was developed with the contributors in a process that included discussion at the symposium and supplementary e-correspondence, resulting in a number of cross disciplinary observations being made by the authors - for example between Garrett Stewart's close readings of literary sonification, and Alex Leff's work on the role of the eye saccade in reading disorders. The book was designed by Mark Simmonds, and has a central insert of yellow pages that features a number of creative submissions, further contributing to the range ways that the book itself can interrogate reading. In an introduction (c. 1500 words) inter and cross-disciplinarity is framed as integral to the approach of the editors, and essential to properly understand the context of reading as itself "hybrid": "recycling ... more innate neuronal networks such as object recognition and memory". In a comment published on the reverse of the book, Professor Steven Connor notes the effectiveness of the book's approach to its subject: "Reading inquisitively over each others' shoulders, the poems, meditations, analyses and experiences in this volume response with audacity and adventure to the challenge of characterising what reading … has been and may yet become" Nathan Jones was research leader on this project, co-editing and co-authoring the introduction with Sam Skinner. He also contributes a 10-page creative-theoretical chapter in the book. The book sold out its edition of 500 copies, and is available freely as PDF and EPUB on the publisher's website. Further public impact and relevance to the project can be seen in the Jones and Skinner's paper "Absorbing Text: Rereading Speed Reading" (APRJA_Machine Research 2016), and contribution to the Transmediale Festival of that year. The material and research partnerships in the book also resulted in an exhibition and new artworks by Jones and Skinner, "Re-learning to Read" (Grundy Art Gallery, 2017). The book and project was supported by a grant from Arts Council England, and by the institutions Foundation for Art and Creative Technology (FACT), Tate Liverpool and Furtherfield Gallery in London

Predictive attenuation of tactile sensation.

It has been proposed that, in order to enhance sensitivity to novel information, the brain removes predictable components of sensory input. This thesis describes a series of psychophysical and behavioural studies investigating predictive filtering in the perception of touch. Using a novel force-matching paradigm, we demonstrate that self-generated tactile sensations are perceived as weaker than the same stimuli externally imposed. This attenuation is shown to be temporally tuned to the expected time of contact and modulated by the certainty with which a sensation can be attributed to self-action. We confirm experimentally that this attenuation results from a predictive, rather than postdictive, mechanism. Such a mechanism may predict the sensory consequences of action based on an internal model of the environment and an efference copy of the motor command. We investigate how prediction is acquired in a new environment and the coordinate systems in which the new environment is internally represented. Using a novel protocol of transcranial magnetic stimulation, we find evidence to suggest that the efference copy signal underlying the prediction arises upstream of primary motor cortex. Patients with schizophrenia are found to show less attenuation than healthy controls, consistent with models of the disease that propose an underlying deficit in sensory prediction. These experimental findings are discussed in relation to potential neural mechanisms of sensory filtering, and the many proposed roles for predictive mechanisms in human sensory and motor systems are reviewed

Aerial Vehicles

This book contains 35 chapters written by experts in developing techniques for making aerial vehicles more intelligent, more reliable, more flexible in use, and safer in operation.It will also serve as an inspiration for further improvement of the design and application of aeral vehicles. The advanced techniques and research described here may also be applicable to other high-tech areas such as robotics, avionics, vetronics, and space