4,174 research outputs found
Real-time Immersive human-computer interaction based on tracking and recognition of dynamic hand gestures
With fast developing and ever growing use of computer based technologies, human-computer interaction (HCI) plays an increasingly pivotal role. In virtual reality (VR), HCI technologies provide not only a better understanding of three-dimensional shapes and spaces, but also sensory immersion and physical interaction. With the hand based HCI being a key HCI modality for object manipulation and gesture based communication, challenges are presented to provide users a natural, intuitive, effortless, precise, and real-time method for HCI based on dynamic hand gestures, due to the complexity of hand postures formed by multiple joints with high degrees-of-freedom, the speed of hand movements with highly variable trajectories and rapid direction changes, and the precision required for interaction between hands and objects in the virtual world.
Presented in this thesis is the design and development of a novel real-time HCI system based on a unique combination of a pair of data gloves based on fibre-optic curvature sensors to acquire finger joint angles, a hybrid tracking system based on inertia and ultrasound to capture hand position and orientation, and a stereoscopic display system to provide an immersive visual feedback. The potential and effectiveness of the proposed system is demonstrated through a number of applications, namely, hand gesture based virtual object manipulation and visualisation, hand gesture based direct sign writing, and hand gesture based finger spelling.
For virtual object manipulation and visualisation, the system is shown to allow a user to select, translate, rotate, scale, release and visualise virtual objects (presented using graphics and volume data) in three-dimensional space using natural hand gestures in real-time. For direct sign writing, the system is shown to be able to display immediately the corresponding SignWriting symbols signed by a user using three different signing sequences and a range of complex hand gestures, which consist of various combinations of hand postures (with each finger open, half-bent, closed, adduction and abduction), eight hand orientations in horizontal/vertical plans, three palm facing directions, and various hand movements (which can have eight directions in horizontal/vertical plans, and can be repetitive, straight/curve, clockwise/anti-clockwise). The development includes a special visual interface to give not only a stereoscopic view of hand gestures and movements, but also a structured visual feedback for each stage of the signing sequence. An excellent basis is therefore formed to develop a full HCI based on all human
gestures by integrating the proposed system with facial expression and body posture recognition methods. Furthermore, for finger spelling, the system is shown to be able to recognise five vowels signed by two hands using the British Sign Language in real-time
An Immersive Telepresence System using RGB-D Sensors and Head Mounted Display
We present a tele-immersive system that enables people to interact with each
other in a virtual world using body gestures in addition to verbal
communication. Beyond the obvious applications, including general online
conversations and gaming, we hypothesize that our proposed system would be
particularly beneficial to education by offering rich visual contents and
interactivity. One distinct feature is the integration of egocentric pose
recognition that allows participants to use their gestures to demonstrate and
manipulate virtual objects simultaneously. This functionality enables the
instructor to ef- fectively and efficiently explain and illustrate complex
concepts or sophisticated problems in an intuitive manner. The highly
interactive and flexible environment can capture and sustain more student
attention than the traditional classroom setting and, thus, delivers a
compelling experience to the students. Our main focus here is to investigate
possible solutions for the system design and implementation and devise
strategies for fast, efficient computation suitable for visual data processing
and network transmission. We describe the technique and experiments in details
and provide quantitative performance results, demonstrating our system can be
run comfortably and reliably for different application scenarios. Our
preliminary results are promising and demonstrate the potential for more
compelling directions in cyberlearning.Comment: IEEE International Symposium on Multimedia 201
Designing 3D scenarios and interaction tasks for immersive environments
In the world of today, immersive reality such as virtual and mixed reality, is one of the
most attractive research fields. Virtual Reality, also called VR, has a huge potential
to be used in in scientific and educational domains by providing users with real-time
interaction or manipulation. The key concept in immersive technologies to provide a
high level of immersive sensation to the user, which is one of the main challenges in
this field. Wearable technologies play a key role to enhance the immersive sensation
and the degree of embodiment in virtual and mixed reality interaction tasks.
This project report presents an application study where the user interacts with
virtual objects, such as grabbing objects, open or close doors and drawers while wearing
a sensory cyberglove developed in our lab (Cyberglove-HT). Furthermore, it presents
the development of a methodology that provides inertial measurement unit(IMU)-based
gesture recognition.
The interaction tasks and 3D immersive scenarios were designed in Unity 3D.
Additionally, we developed an inertial sensor-based gesture recognition by employing
an Long short-term memory (LSTM) network. In order to distinguish the effect of
wearable technologies in the user experience in immersive environments, we made an
experimental study comparing the Cyberglove-HT to standard VR controllers (HTC
Vive Controller). The quantitive and subjective results indicate that we were able
to enhance the immersive sensation and self embodiment with the Cyberglove-HT. A
publication resulted from this work [1] which has been developed in the framework
of the R&D project Human Tracking and Perception in Dynamic Immersive Rooms
(HTPDI
GIFT: Gesture-Based Interaction by Fingers Tracking, an Interaction Technique for Virtual Environment
Three Dimensional (3D) interaction is the plausible human interaction inside a Virtual Environment (VE). The rise of the Virtual Reality (VR) applications in various domains demands for a feasible 3D interface. Ensuring immersivity in a virtual space, this paper presents an interaction technique where manipulation is performed by the perceptive gestures of the two dominant fingers; thumb and index. The two fingertip-thimbles made of paper are used to trace states and positions of the fingers by an ordinary camera. Based on the positions of the fingers, the basic interaction tasks; selection, scaling, rotation, translation and navigation are performed by intuitive gestures of the fingers. Without keeping a gestural database, the features-free detection of the fingers guarantees speedier interactions. Moreover, the system is user-independent and depends neither on the size nor on the color of the users’ hand. With a case-study project; Interactions by the Gestures of Fingers (IGF) the technique is implemented for evaluation. The IGF application traces gestures of the fingers using the libraries of OpenCV at the back-end. At the front-end, the objects of the VE are rendered accordingly using the Open Graphics Library; OpenGL. The system is assessed in a moderate lighting condition by a group of 15 users. Furthermore, usability of the technique is investigated in games. Outcomes of the evaluations revealed that the approach is suitable for VR applications both in terms of cost and accuracy
Navigation and interaction in a real-scale digital mock-up using natural language and user gesture
This paper tries to demonstrate a very new real-scale 3D system and sum up some firsthand and cutting edge results concerning multi-modal navigation and interaction interfaces. This work is part of the CALLISTO-SARI collaborative project. It aims at constructing an immersive room, developing a set of software tools and some navigation/interaction interfaces. Two sets of interfaces will be introduced here: 1) interaction devices, 2) natural language (speech processing) and user gesture. The survey on this system using subjective observation (Simulator Sickness Questionnaire, SSQ) and objective measurements (Center of Gravity, COG) shows that using natural languages and gesture-based interfaces induced less cyber-sickness comparing to device-based interfaces. Therefore, gesture-based is more efficient than device-based interfaces.FUI CALLISTO-SAR
Real-time immersive human-computer interaction based on tracking and recognition of dynamic hand gestures
With fast developing and ever growing use of computer based technologies, human-computer interaction (HCI) plays an increasingly pivotal role. In virtual reality (VR), HCI technologies provide not only a better understanding of three-dimensional shapes and spaces, but also sensory immersion and physical interaction. With the hand based HCI being a key HCI modality for object manipulation and gesture based communication, challenges are presented to provide users a natural, intuitive, effortless, precise, and real-time method for HCI based on dynamic hand gestures, due to the complexity of hand postures formed by multiple joints with high degrees-of-freedom, the speed of hand movements with highly variable trajectories and rapid direction changes, and the precision required for interaction between hands and objects in the virtual world. Presented in this thesis is the design and development of a novel real-time HCI system based on a unique combination of a pair of data gloves based on fibre-optic curvature sensors to acquire finger joint angles, a hybrid tracking system based on inertia and ultrasound to capture hand position and orientation, and a stereoscopic display system to provide an immersive visual feedback. The potential and effectiveness of the proposed system is demonstrated through a number of applications, namely, hand gesture based virtual object manipulation and visualisation, hand gesture based direct sign writing, and hand gesture based finger spelling. For virtual object manipulation and visualisation, the system is shown to allow a user to select, translate, rotate, scale, release and visualise virtual objects (presented using graphics and volume data) in three-dimensional space using natural hand gestures in real-time. For direct sign writing, the system is shown to be able to display immediately the corresponding SignWriting symbols signed by a user using three different signing sequences and a range of complex hand gestures, which consist of various combinations of hand postures (with each finger open, half-bent, closed, adduction and abduction), eight hand orientations in horizontal/vertical plans, three palm facing directions, and various hand movements (which can have eight directions in horizontal/vertical plans, and can be repetitive, straight/curve, clockwise/anti-clockwise). The development includes a special visual interface to give not only a stereoscopic view of hand gestures and movements, but also a structured visual feedback for each stage of the signing sequence. An excellent basis is therefore formed to develop a full HCI based on all human gestures by integrating the proposed system with facial expression and body posture recognition methods. Furthermore, for finger spelling, the system is shown to be able to recognise five vowels signed by two hands using the British Sign Language in real-time.EThOS - Electronic Theses Online ServiceGBUnited Kingdo
Defining Interaction within Immersive Virtual Environments
PhDThis thesis is concerned with the design of Virtual Environments (YEs) -
in particular with the tools and techniques used to describe interesting and
useful environments. This concern is not only with respect to the appearance
of objects in the VE but also with their behaviours and their reactions to
actions of the participants. The main research hypothesis is that there are
several advantages to constructing these interactions and behaviours whilst
remaining immersed within the VE which they describe. These advantages
include the fact that editing is done interactively with immediate effect and
without having to resort to the usual edit-compile-test cycle. This means
that the participant doesn't have to leave the VE and lose their sense of
presence within it, and editing tasks can take advantage of the enhanced
spatial cognition and naturalistic interaction metaphors a VE provides.
To this end a data flow dialogue architecture with an immersive virtual
environment presentation system was designed and built. The data flow
consists of streams of data that originate at sensors that register the body
state of the participant, flowing through filters that modify the streams and
affect the yE.
The requirements for such a system and the filters it should contain are
derived from two pieces of work on interaction metaphors, one based on
a desktop system using a novel input device and the second a navigation
technique for an immersive system. The analysis of these metaphors highlighted
particular tasks that such a virtual environment dialogue architecture
(VEDA) system might be used to solve, and illustrate the scope of interactions
that should be accommodated.
Initial evaluation of the VEDA system is provided by moderately sized
demonstration environments and tools constructed by the author. Further
evaluation is provided by an in-depth study where three novice VE designers
were invited to construct VEs with the VEDA system. This highlighted the
flexibility that the VEDA approach provides and the utility of the immersive
presentation over traditional techniques in that it allows the participant to
use more natural and expressive techniques in the construction process. In
other words the evaluation shows how the immersive facilities of VEs can be
exploited in the process of constructing further VEs
- …