1,567 research outputs found
A multi-modal dance corpus for research into real-time interaction between humans in online virtual environments
We present a new, freely available, multimodal corpus for research into, amongst other areas, real-time realistic interaction between humans in online virtual environments. The specific corpus scenario focuses on an online dance class application scenario where students, with avatars driven by whatever 3D capture technology are locally available to them, can learn choerographies with teacher guidance in an online virtual ballet studio. As the data corpus is focused on this scenario, it consists of student/teacher dance choreographies concurrently captured at two different sites using a variety of media modalities, including synchronised audio rigs, multiple cameras, wearable inertial measurement devices and depth sensors. In the corpus, each of the several dancers perform a number of fixed choreographies, which are both graded according to a number of specific evaluation criteria. In addition, ground-truth dance choreography annotations are provided. Furthermore, for unsynchronised sensor modalities, the corpus also includes distinctive events for data stream synchronisation. Although the data corpus is tailored specifically for an online dance class application scenario, the data is free to download and used for any research and development purposes
Architecture and applications of the FingerMouse: a smart stereo camera for wearable computing HCI
In this paper we present a visual input HCI system for wearable computers, the FingerMouse. It is a fully integrated stereo camera and vision processing system, with a specifically designed ASIC performing stereo block matching at 5Mpixel/s (e.g. QVGA 320ร240at 30fps) and a disparity range of 47, consuming 187mW (78mW in the ASIC). It is button-sized (43mmร18mm) and can be worn on the body, capturing the user's hand and processing in real-time its coordinates as well as a 1-bit image of the hand segmented from the background. Alternatively, the system serves as a smart depth camera, delivering foreground segmentation and tracking, depth maps and standard images, with a processing latency smaller than 1ms. This paper describes the FingerMouse functionality and its applications, and how the specific architecture outperforms other systems in size, latency and power consumptio
Requirement analysis and sensor specifications โ First version
In this first version of the deliverable, we make the following contributions: to design the
WEKIT capturing platform and the associated experience capturing API, we use a
methodology for system engineering that is relevant for different domains such as: aviation,
space, and medical and different professions such as: technicians, astronauts, and medical
staff. Furthermore, in the methodology, we explore the system engineering process and how
it can be used in the project to support the different work packages and more importantly
the different deliverables that will follow the current.
Next, we provide a mapping of high level functions or tasks (associated with experience
transfer from expert to trainee) to low level functions such as: gaze, voice, video, body
posture, hand gestures, bio-signals, fatigue levels, and location of the user in the
environment. In addition, we link the low level functions to their associated sensors.
Moreover, we provide a brief overview of the state-of-the-art sensors in terms of their
technical specifications, possible limitations, standards, and platforms.
We outline a set of recommendations pertaining to the sensors that are most relevant for
the WEKIT project taking into consideration the environmental, technical and human
factors described in other deliverables. We recommend Microsoft Hololens (for Augmented
reality glasses), MyndBand and Neurosky chipset (for EEG), Microsoft Kinect and Lumo Lift
(for body posture tracking), and Leapmotion, Intel RealSense and Myo armband (for hand
gesture tracking). For eye tracking, an existing eye-tracking system can be customised to
complement the augmented reality glasses, and built-in microphone of the augmented
reality glasses can capture the expertโs voice. We propose a modular approach for the design
of the WEKIT experience capturing system, and recommend that the capturing system
should have sufficient storage or transmission capabilities.
Finally, we highlight common issues associated with the use of different sensors. We
consider that the set of recommendations can be useful for the design and integration of the
WEKIT capturing platform and the WEKIT experience capturing API to expedite the time
required to select the combination of sensors which will be used in the first prototype.WEKI
์ธ๊ฐ ๊ธฐ๊ณ ์ํธ์์ฉ์ ์ํ ๊ฐ๊ฑดํ๊ณ ์ ํํ ์๋์ ์ถ์ ๊ธฐ์ ์ฐ๊ตฌ
ํ์๋
ผ๋ฌธ(๋ฐ์ฌ) -- ์์ธ๋ํ๊ต๋ํ์ : ๊ณต๊ณผ๋ํ ๊ธฐ๊ณํญ๊ณต๊ณตํ๋ถ, 2021.8. ์ด๋์ค.Hand-based interface is promising for realizing intuitive, natural and accurate human machine interaction (HMI), as the human hand is main source of dexterity in our daily activities.
For this, the thesis begins with the human perception study on the detection threshold of visuo-proprioceptive conflict (i.e., allowable tracking error) with or without cutantoues haptic feedback, and suggests tracking error specification for realistic and fluidic hand-based HMI. The thesis then proceeds to propose a novel wearable hand tracking module, which, to be compatible with the cutaneous haptic devices spewing magnetic noise, opportunistically employ heterogeneous sensors (IMU/compass module and soft sensor) reflecting the anatomical properties of human hand, which is suitable for specific application (i.e., finger-based interaction with finger-tip haptic devices).
This hand tracking module however loses its tracking when interacting with, or being nearby, electrical machines or ferromagnetic materials. For this, the thesis presents its main contribution, a novel visual-inertial skeleton tracking (VIST) framework, that can provide accurate and robust hand (and finger) motion tracking even for many challenging real-world scenarios and environments,
for which the state-of-the-art technologies are known to fail due to their respective fundamental limitations (e.g., severe occlusions for tracking purely with vision sensors; electromagnetic interference for tracking purely with IMUs (inertial measurement units) and compasses; and mechanical contacts for tracking purely with soft sensors).
The proposed VIST framework comprises a sensor glove with multiple IMUs and passive visual markers as well as a head-mounted stereo camera; and a tightly-coupled filtering-based visual-inertial fusion algorithm to estimate the hand/finger motion and auto-calibrate hand/glove-related kinematic parameters simultaneously while taking into account the hand anatomical constraints.
The VIST framework exhibits good tracking accuracy and robustness, affordable material cost, light hardware and software weights, and ruggedness/durability even to permit washing.
Quantitative and qualitative experiments are also performed to validate the advantages and properties of our VIST framework, thereby, clearly demonstrating its potential for real-world applications.์ ๋์์ ๊ธฐ๋ฐ์ผ๋ก ํ ์ธํฐํ์ด์ค๋ ์ธ๊ฐ-๊ธฐ๊ณ ์ํธ์์ฉ ๋ถ์ผ์์ ์ง๊ด์ฑ, ๋ชฐ์
๊ฐ, ์ ๊ตํจ์ ์ ๊ณตํด์ค ์ ์์ด ๋ง์ ์ฃผ๋ชฉ์ ๋ฐ๊ณ ์๊ณ , ์ด๋ฅผ ์ํด ๊ฐ์ฅ ํ์์ ์ธ ๊ธฐ์ ์ค ํ๋๊ฐ ์ ๋์์ ๊ฐ๊ฑดํ๊ณ ์ ํํ ์ถ์ ๊ธฐ์ ์ด๋ค.
์ด๋ฅผ ์ํด ๋ณธ ํ์๋
ผ๋ฌธ์์๋ ๋จผ์ ์ฌ๋ ์ธ์ง์ ๊ด์ ์์ ์ ๋์ ์ถ์ ์ค์ฐจ์ ์ธ์ง ๋ฒ์๋ฅผ ๊ท๋ช
ํ๋ค. ์ด ์ค์ฐจ ์ธ์ง ๋ฒ์๋ ์๋ก์ด ์ ๋์ ์ถ์ ๊ธฐ์ ๊ฐ๋ฐ ์ ์ค์ํ ์ค๊ณ ๊ธฐ์ค์ด ๋ ์ ์์ด ์ด๋ฅผ ํผํ์ ์คํ์ ํตํด ์ ๋์ ์ผ๋ก ๋ฐํ๊ณ , ํนํ ์๋ ์ด๊ฐ ์ฅ๋น๊ฐ ์์๋ ์ด ์ธ์ง ๋ฒ์์ ๋ณํ๋ ๋ฐํ๋ค.
์ด๋ฅผ ํ ๋๋ก, ์ด๊ฐ ํผ๋๋ฐฑ์ ์ฃผ๋ ๊ฒ์ด ๋ค์ํ ์ธ๊ฐ-๊ธฐ๊ณ ์ํธ์์ฉ ๋ถ์ผ์์ ๋๋ฆฌ ์ฐ๊ตฌ๋์ด ์์ผ๋ฏ๋ก, ๋จผ์ ์๋ ์ด๊ฐ ์ฅ๋น์ ํจ๊ป ์ฌ์ฉํ ์ ์๋ ์ ๋์ ์ถ์ ๋ชจ๋์ ๊ฐ๋ฐํ๋ค.
์ด ์๋ ์ด๊ฐ ์ฅ๋น๋ ์๊ธฐ์ฅ ์ธ๋์ ์ผ์ผ์ผ ์ฐฉ์ฉํ ๊ธฐ์ ์์ ํํ ์ฌ์ฉ๋๋ ์ง์๊ธฐ ์ผ์๋ฅผ ๊ต๋ํ๋๋ฐ, ์ด๋ฅผ ์ ์ ํ ์ฌ๋ ์์ ํด๋ถํ์ ํน์ฑ๊ณผ ๊ด์ฑ ์ผ์/์ง์๊ธฐ ์ผ์/์ํํธ ์ผ์์ ์ ์ ํ ํ์ฉ์ ํตํด ํด๊ฒฐํ๋ค.
์ด๋ฅผ ํ์ฅํ์ฌ ๋ณธ ๋
ผ๋ฌธ์์๋, ์ด๊ฐ ์ฅ๋น ์ฐฉ์ฉ ์ ๋ฟ ์๋๋ผ ๋ชจ๋ ์ฅ๋น ์ฐฉ์ฉ / ํ๊ฒฝ / ๋ฌผ์ฒด์์ ์ํธ์์ฉ ์์๋ ์ฌ์ฉ ๊ฐ๋ฅํ ์๋ก์ด ์ ๋์ ์ถ์ ๊ธฐ์ ์ ์ ์ํ๋ค.
๊ธฐ์กด์ ์ ๋์ ์ถ์ ๊ธฐ์ ๋ค์ ๊ฐ๋ฆผ ํ์ (์์ ๊ธฐ๋ฐ ๊ธฐ์ ), ์ง์๊ธฐ ์ธ๋ (๊ด์ฑ/์ง์๊ธฐ ์ผ์ ๊ธฐ๋ฐ ๊ธฐ์ ), ๋ฌผ์ฒด์์ ์ ์ด (์ํํธ ์ผ์ ๊ธฐ๋ฐ ๊ธฐ์ ) ๋ฑ์ผ๋ก ์ธํด ์ ํ๋ ํ๊ฒฝ์์ ๋ฐ์ ์ฌ์ฉํ์ง ๋ชปํ๋ค.
์ด๋ฅผ ์ํด ๋ง์ ๋ฌธ์ ๋ฅผ ์ผ์ผํค๋ ์ง์๊ธฐ ์ผ์ ์์ด ์๋ณด์ ์ธ ํน์ฑ์ ์ง๋๋ ๊ด์ฑ ์ผ์์ ์์ ์ผ์๋ฅผ ์ตํฉํ๊ณ , ์ด๋ ์์ ๊ณต๊ฐ์ ๋ค ์์ ๋์ ์์ง์์ ๊ฐ๋ ์ ๋์์ ์ถ์ ํ๊ธฐ ์ํด ๋ค์์ ๊ตฌ๋ถ๋์ง ์๋ ๋ง์ปค๋ค์ ์ฌ์ฉํ๋ค.
์ด ๋ง์ปค์ ๊ตฌ๋ถ ๊ณผ์ (correspondence search)๋ฅผ ์ํด ๊ธฐ์กด์ ์ฝ๊ฒฐํฉ (loosely-coupled) ๊ธฐ๋ฐ์ด ์๋ ๊ฐ๊ฒฐํฉ (tightly-coupled ๊ธฐ๋ฐ ์ผ์ ์ตํฉ ๊ธฐ์ ์ ์ ์ํ๊ณ , ์ด๋ฅผ ํตํด ์ง์๊ธฐ ์ผ์ ์์ด ์ ํํ ์ ๋์์ด ๊ฐ๋ฅํ ๋ฟ ์๋๋ผ ์ฐฉ์ฉํ ์ผ์๋ค์ ์ ํ์ฑ/ํธ์์ฑ์ ๋ฌธ์ ๋ฅผ ์ผ์ผํค๋ ์ผ์ ๋ถ์ฐฉ ์ค์ฐจ / ์ฌ์ฉ์์ ์ ๋ชจ์ ๋ฑ์ ์๋์ผ๋ก ์ ํํ ๋ณด์ ํ๋ค.
์ด ์ ์๋ ์์-๊ด์ฑ ์ผ์ ์ตํฉ ๊ธฐ์ (Visual-Inertial Skeleton Tracking (VIST)) ์ ๋ฐ์ด๋ ์ฑ๋ฅ๊ณผ ๊ฐ๊ฑด์ฑ์ด ๋ค์ํ ์ ๋/์ ์ฑ ์คํ์ ํตํด ๊ฒ์ฆ๋์๊ณ , ์ด๋ VIST์ ๋ค์ํ ์ผ์ํ๊ฒฝ์์ ๊ธฐ์กด ์์คํ
์ด ๊ตฌํํ์ง ๋ชปํ๋ ์ ๋์ ์ถ์ ์ ๊ฐ๋ฅ์ผ ํจ์ผ๋ก์จ, ๋ง์ ์ธ๊ฐ-๊ธฐ๊ณ ์ํธ์์ฉ ๋ถ์ผ์์์ ๊ฐ๋ฅ์ฑ์ ๋ณด์ฌ์ค๋ค.1 Introduction 1
1.1. Motivation 1
1.2. Related Work 5
1.3. Contribution 12
2 Detection Threshold of Hand Tracking Error 16
2.1. Motivation 16
2.2. Experimental Environment 20
2.2.1. Hardware Setup 21
2.2.2. Virtual Environment Rendering 23
2.2.3. HMD Calibration 23
2.3. Identifying the Detection Threshold of Tracking Error 26
2.3.1. Experimental Setup 27
2.3.2. Procedure 27
2.3.3. Experimental Result 31
2.4. Enlarging the Detection Threshold of Tracking Error by Haptic Feedback 31
2.4.1. Experimental Setup 31
2.4.2. Procedure 32
2.4.3. Experimental Result 34
2.5. Discussion 34
3 Wearable Finger Tracking Module for Haptic Interaction 38
3.1. Motivation 38
3.2. Development of Finger Tracking Module 42
3.2.1. Hardware Setup 42
3.2.2. Tracking algorithm 45
3.2.3. Calibration method 48
3.3. Evaluation for VR Haptic Interaction Task 50
3.3.1. Quantitative evaluation of FTM 50
3.3.2. Implementation of Wearable Cutaneous Haptic Interface
51
3.3.3. Usability evaluation for VR peg-in-hole task 53
3.4. Discussion 57
4 Visual-Inertial Skeleton Tracking for Human Hand 59
4.1. Motivation 59
4.2. Hardware Setup and Hand Models 62
4.2.1. Human Hand Model 62
4.2.2. Wearable Sensor Glove 62
4.2.3. Stereo Camera 66
4.3. Visual Information Extraction 66
4.3.1. Marker Detection in Raw Images 68
4.3.2. Cost Function for Point Matching 68
4.3.3. Left-Right Stereo Matching 69
4.4. IMU-Aided Correspondence Search 72
4.5. Filtering-based Visual-Inertial Sensor Fusion 76
4.5.1. EKF States for Hand Tracking and Auto-Calibration 78
4.5.2. Prediction with IMU Information 79
4.5.3. Correction with Visual Information 82
4.5.4. Correction with Anatomical Constraints 84
4.6. Quantitative Evaluation for Free Hand Motion 87
4.6.1. Experimental Setup 87
4.6.2. Procedure 88
4.6.3. Experimental Result 90
4.7. Quantitative and Comparative Evaluation for Challenging Hand Motion 95
4.7.1. Experimental Setup 95
4.7.2. Procedure 96
4.7.3. Experimental Result 98
4.7.4. Performance Comparison with Existing Methods for Challenging Hand Motion 101
4.8. Qualitative Evaluation for Real-World Scenarios 105
4.8.1. Visually Complex Background 105
4.8.2. Object Interaction 106
4.8.3. Wearing Fingertip Cutaneous Haptic Devices 109
4.8.4. Outdoor Environment 111
4.9. Discussion 112
5 Conclusion 116
References 124
Abstract (in Korean) 139
Acknowledgment 141๋ฐ
A gaze-contingent framework for perceptually-enabled applications in healthcare
Patient safety and quality of care remain the focus of the smart operating room of the future. Some of the most influential factors with a detrimental effect are related to suboptimal communication among the staff, poor flow of information, staff workload and fatigue, ergonomics and sterility in the operating room. While technological developments constantly transform the operating room layout and the interaction between surgical staff and machinery, a vast array of opportunities arise for the design of systems and approaches, that can enhance patient safety and improve workflow and efficiency.
The aim of this research is to develop a real-time gaze-contingent framework towards a "smart" operating suite, that will enhance operator's ergonomics by allowing perceptually-enabled, touchless and natural interaction with the environment. The main feature of the proposed framework is the ability to acquire and utilise the plethora of information provided by the human visual system to allow touchless interaction with medical devices in the operating room. In this thesis, a gaze-guided robotic scrub nurse, a gaze-controlled robotised flexible endoscope and a gaze-guided assistive robotic system are proposed. Firstly, the gaze-guided robotic scrub nurse is presented; surgical teams performed a simulated surgical task with the assistance of a robot scrub nurse, which complements the human scrub nurse in delivery of surgical instruments, following gaze selection by the surgeon. Then, the gaze-controlled robotised flexible endoscope is introduced; experienced endoscopists and novice users performed a simulated examination of the upper gastrointestinal tract using predominately their natural gaze. Finally, a gaze-guided assistive robotic system is presented, which aims to facilitate activities of daily living. The results of this work provide valuable insights into the feasibility of integrating the developed gaze-contingent framework into clinical practice without significant workflow disruptions.Open Acces
Recommended from our members
An Experimental Hybrid User Interface for Collaboration
We present EMMIE (Environment Management for Multi-user Information Environments), an experimental user interface to a collaborative augmented environment. Users share a 3D virtual space and manipulate virtual objects representing information to be discussed. This approach not only allows for cooperation in a shared physical space, but also addresses tele-collaboration in physically separate but virtually shared spaces. We refer to EMMIE as a hybrid user interface because it combines a variety of different technologies and techniques, including virtual elements such as 3D widgets, and physical objects such as tracked displays and input devices. See-through head-worn displays overlay the virtual environment on the physical environment. Our research prototype includes additional 2D and 3D displays, ranging from palm-sized to wall-sized, allowing the most appropriate one to be used for any task. Objects can be moved among displays (including across dimensionalities) through drag & drop. In analogy to 2D window managers, we describe a prototype implementation of a shared 3Denvironment manager that is distributed across displays, machines, and operating systems. We also discuss two methods we are exploring for handling information privacy in such an environment
Software techniques for improving head mounted displays to create comfortable user experiences in virtual reality
Head Mounted Displays (HMDs) allow users to experience Virtual Reality (VR) with a great level of immersion. Advancements in hardware technologies have led to a reduction in cost of producing good quality VR HMDs bringing them out from research labs to consumer markets. However, the current generation of HMDs suffer from a few fundamental problems that can deter their widespread adoption. For this thesis, we explored two techniques to overcome some of the challenges of experiencing VR when using HMDs.
When experiencing VR with HMDs strapped to your head, even simple physical tasks like drinking a beverage can be difficult and awkward. We explored mixed reality renderings that selectively incorporate the physical world into the virtual world for interactions with physical objects. We conducted a user study comparing four rendering techniques that balance immersion in the virtual world with ease of interaction with the physical world.
Users of VR systems often experience vection, the perception of self-motion in the absence of any physical movement. While vection helps to improve presence in VR, it often leads to a form of motion sickness called cybersickness. Prior work has discovered that changing vection (changing the perceived speed or moving direction) causes more severe cybersickness than steady vection (walking at a constant speed or in a constant direction). Based on this idea, we tried to reduce cybersickness caused by character movements in a First Person Shooter (FPS) game in VR. We propose Rotation Blurring (RB), uniformly blurring the screen during rotational movements to reduce cybersickness. We performed a user study to evaluate the impact of RB in reducing cybersickness and found that RB led to an overall reduction in sickness levels of the participants and delayed its onset. Participants who experienced acute levels of cybersickness benefited significantly from this technique
- โฆ