Stable haptic rendering in virtual environment

Haptics refers to the science of perception and manipulation of objects in virtual environments. Its applications spread rapidly from a human-computer interface to manufacturing, scientific discovery, medical training, etc. In a complex dynamic virtual environment, it is important to have smooth and realistic haptic feedback. In this project, we focus on research and development of stable haptic rendering methods and algorithms to provide continuous force and torque feedback in dynamic virtual environments. In haptic rendering, many algorithms and methods were proposed such as the god-object method, spring-damper method, virtual proxy method, Voxmap Point Shell (VPS) method, constraint-based method, Quasi-Static Approximation (QSA) method, etc. Currently, for six degrees-of-freedom (6-DOF) haptic rendering, the direct haptic rendering methods only support geometric rendering without physically based dynamic simulation. Virtual coupling based methods separate the haptic device from the virtual tool. It enables high stable force feedback and supports dynamic simulation of the virtual objects with physical properties. Although these algorithms have greatly improved performance of haptic rendering, there are still unsolved and challenging problems as follows. 1) Buzzing. If a virtual tool has physically based properties (for example, mass), the buzzing would appear as continuous high frequency vibrations. 2) Inaccurate manipulation. When the virtual tool has a large mass value, the displacement would become larger because of the gravity. This large displacement would introduce an inaccurate movement during the haptic manipulation that can cause accuracy problems. 3) Discontinuous force update. When there are complex models and/or deformable models, the physical simulation may produce a low update rate of force which causes discontinuous force output on the haptic device. The aim of the research is to propose general haptic rendering algorithms to improve stability of haptic rendering systems. To improve stability of haptic rendering, we propose new stable haptic rendering algorithms based on virtual coupling. The algorithms can be used for various static or dynamic applications to provide stable haptic force and torque feedback. First, we propose a stable dynamic algorithm based on virtual coupling for 6-DOF haptic rendering. It can overcome the “buzzing” problem when a virtual tool with small mass values is used. The novelty is that a nonlinear force/torque algorithm is proposed to calculate the haptic interaction when the collision happens between the virtual tool and virtual objects. The force/torque magnitude is automatically saturated to the maximum force/torque value of the haptic device. The algorithm is tested on the standard benchmarks and outperforms available algorithms such as spring-damper algorithm and QSA algorithm. Experimental results show that this algorithm is capable to provide stable 6-DOF haptic rendering for dynamic rigid virtual objects with physical properties. Second, we propose an adaptive haptic rendering algorithm based on virtual coupling to overcome the inaccurate manipulation problem caused by the large mass values of the virtual tool. The algorithm can automatically adjust virtual coupling parameters according to the mass values of the simulated virtual tools. In addition, the force/torque magnitude is saturated to the maximum force/torque values of the haptic device when large interaction force is generated. The algorithm is tested on the standard haptic rendering benchmarks. Compared to other algorithms, the adaptive algorithm supports more accurate haptic manipulation. Third, we propose a new prediction algorithm for smooth haptic rendering to overcome the low update rate of the force during physical simulation of complex and/or deformable models. We propose to use a prediction method combined with an interpolation method to calculate smooth haptic interaction force. An auto-regressive model is used to predict the force value from the previous haptic force calculation. We introduce a spline function to interpolate smooth force values for the haptic force output. The proposed method can provide smooth and continuous haptic force feedback in a high update rate during the virtual manipulation of complex and/or deformable objects. It outperforms other force estimation/prediction methods. A haptic enabled molecular docking system HMolDock is developed to find the correct docking positions between ligand and receptor. Here, a stable haptic rendering algorithm is implemented at the application level of the system to enable stable haptic manipulation of large molecules. HMolDock can help the drug designer to find the correct docking positions between molecular systems. For medical applications, we develop a haptic-based serious game "T Puzzle" and an EEG-enabled haptic-based serious game “Basket”. In the game "T Puzzle", virtual blocks are assigned with small mass values, and the stable dynamic algorithm is implemented to provide stable haptic manipulation in virtual environments. This game can be used for intellectual development and post stroke rehabilitation exercises. The EEG-enabled haptic based post stroke rehabilitation serious game “Basket” is developed to help patients to perform rehabilitation activities. In the game, the haptic device is used to manipulate various virtual objects and move them into the basket. The adaptive haptic rendering algorithm is implemented to guarantee an accurate haptic manipulation of the virtual objects with different mass values. The EEG based emotion recognition algorithm is implemented to recognize emotions of the patient and automatically adjust the difficulty level of the game. The proposed haptic rendering algorithms are also integrated in CHAI 3D library.DOCTOR OF PHILOSOPHY (EEE

Stable dynamic algorithm based on virtual coupling for 6-DOF haptic rendering

In this paper, a new stable dynamic algorithm based on virtual coupling was proposed for 6-Degrees-of-Freedom (DOF) haptic rendering. It allows stable haptic manipulation of virtual objects when a virtual tool has physical property such as mass. In the haptic rendering process, we consider the dynamic property such as rotation inertia in each haptic frame. The main contribution of the stable dynamic algorithm is that it could overcome the "buzzing" problem appeared in the haptic rendering process. A nonlinear force/torque algorithm is proposed to calculate the haptic interaction when the collision happens between the virtual tool and virtual objects. The force/torque magnitude could saturate to the maximum force/torque value of the haptic device. The implemented algorithm was tested with peg-in-hole and Stanford bunny benchmarks. The experimental results showed that our algorithm was capable to provide stable 6-DOF haptic rendering for dynamic rigid virtual objects with physical property such as mass

Research on Applicability of Hot-Bulb Anemometer under Low Pressure

In Mars and other deep space exploration missions, the planetary atmosphere makes the difference in heat transfer characteristics on the planetary surface and on-orbit environment. In order to achieve the purposes of thermal model correction and spacecraft verification in extreme environment, Mars rover needs to be tested in a simulated Mars environment including low pressure, solar heat flux, wind speed and background temperature. Thus, wind speed should be measured at multiple points in the Mars rover thermal test. In a general Mars rover thermal balance test, the requirement for wind speed control and measurement is 0-15m/s under 700Pa pressure. The current anemometer for industrial use is mainly based on the dynamic pressure, heat or ultrasound. They have a small signal and need to be recalibrated at low pressures. In this paper, a constant heat flux hot-bulb anemometer model has been built using dimensionless number analysis method, with which the anemometer response under low pressure has been calculated. A series of calibration test has been employed to verify the model in space environment chamber. The two methods above reached a similar result, which demonstrates the effectiveness of the analysis

CogniMeter: EEG-based emotion, mental workload and stress visual monitoring

Real-time EEG (Electroencephalogram)-based user's emotion, mental workload and stress monitoring is a new direction in research and development of human-machine interfaces. It has attracted recently more attention from the research community and industry as wireless portable EEG devices became easily available on the market. EEG-based technology has been applied in anesthesiology, psychology, serious games or even in marketing. In this work, we describe available real-time algorithms of emotion recognition, mental workload, and stress recognition from EEG and propose a novel interface CogniMeter for the user's mental state visual monitoring. The system can be used in real time to assess human current emotions, levels of mental workload and stress. Currently, it is applied to monitor the user's emotional state, mental workload and stress in simulation scenarios or used as a tool to assess the subject's mental state in human factor study experiments

Prediction of Human Cognitive Abilities based on EEG measurements

The difference in cognitive abilities of humans could be assessed by indexes extracted from EEG. In this paper, we propose and implement an experiment with 60 subjects to study how cognitive abilities can be identified through EEG. We analyzed parameters of the individual frequency band that can be used for prediction of cognitive abilities of subjects. In the experiment, the subjects performed cognitive tests with EEG recording done prior to the tests. Different patterns of alpha band activity are proven to be indicators of cognitive abilities and performances. Our hypothesis is that cognitive abilities can be predicted based on EEG measurements. The results of analysis of the experiment show significant correlation between subjects' cognitive abilities assessed by the tests and the EEG measurements

Neurofeedback games for the enhancement of cognitive abilities related to multitasking

Various approaches and algorithms are proposed and implemented in neurofeedback systems for training of human cognitive abilities. Neurofeedback training can be done in different forms such as games, simple visual feedback such as color changes, or audio feedback such as a beeping sound. In this work, first, we propose and implement neurofeedback games that can be used with different neurofeedback algorithms to train cognitive abilities related to multitasking such as working memory and attention. Each game consists of a neurofeedback system with implemented neurofeedback algorithms and game flow part. The user has to choose a neurofeedback algorithm (for example, alpha, theta/beta or fractal dimension based training algorithm) through the interface. The user's individual alpha peak frequency and individual alpha band width can be calculated and entered as well to make the training more effective. Second, we propose a novel multitasking game that can be used both as the game for neurofeedback training and as the multitasking test with a score. The score includes the user's performance and reaction time. The player has to drive the car on the road and avoid the walls, react to the sounds in the auditory oddball task, and remember the letters and shoot the target in case the repeated letters appeared. Thus, the multitasking game includes 3 simultaneous tasks involving the player's coordination such as avoiding the walls during the car driving; player's comprehension of the sounds such as recognition of 2 different sounds, and memorization of the letters such that the player has to remember the letters shown on the screen. A neurofeedback training algorithm is integrated in the game as follows: if the targeted EEG parameter is achieved - the driving environment stays without changes; if it is not - the driving environment is shaken, and the color is changed. The multitasking game is more challenging for the player than traditional neurofeedback games because not only the neurofeedback algorithm is used in the game but the simultaneous tasks are integrated as well. Thus, the multitasking game needs to be tested and compared with the traditional neurofeedback games to assess its effectiveness

Synthesis of Silver Nanoparticles-Modified Graphitic Carbon Nitride Nanosheets for Highly Efficient Photocatalytic Hydrogen Peroxide Evolution

As a promising metal-free photocatalyst, graphitic carbon nitride (g-C3N4) is still limited by insufficient visible light absorption and rapid recombination of photogenerated carriers, resulting in low photocatalytic activity. Here, we adjusted the microstructure of the pristine bulk-g-C3N4 (PCN) and further loaded silver (Ag) nanoparticles. Abundant Ag nanoparticles were grown on the thin-layer g-C3N4 nanosheets (CNNS), and the Ag nanoparticles decorated g-C3N4 nanosheets (Ag@CNNS) were successfully synthesized. The thin-layer nanosheet-like structure was not only beneficial for the loading of Ag nanoparticles but also for the adsorption and activation of reactants via exposing more active sites. Moreover, the surface plasmon resonance (SPR) effect induced by Ag nanoparticles enhanced the absorption of visible light by narrowing the band gap of the substrate. Meanwhile, the composite band structure effectively promoted the separation and transfer of carriers. Benefiting from these merits, the Ag@CNNS reached a superior hydrogen peroxide (H2O2) yield of 120.53 μmol/g/h under visible light irradiation in pure water (about 8.0 times higher than that of PCN), significantly surpassing most previous reports. The design method of manipulating the microstructure of the catalyst combined with the modification of metal nanoparticles provides a new idea for the rational development and application of efficient photocatalysts

EEG-Based Human Factors Evaluation of Conflict Resolution Aid and Tactile User Interface in Future Air Traffic Control Systems

Currently, Air Traffic Control (ATC) systems are reliable with automation supports, however, the increased traffic density and complex air traffic situations bring new challenges to ATC systems and air-traffic controllers (ATCOs). We conduct an experiment to evaluate the current ATC system and test conflict resolution automation and tactile user interface to be the inputs of the future ATC system. We propose an Electroencephalogram (EEG)-based system to monitor and analyze human factors measurements of ATCOs in ATC systems to apply it in our experiment. The EEG-based tools are used to monitor and record the brain states of ATCOs during the experiment. Real-time EEG-based human factors evaluation of an ATC system allows researchers to analyze the changes of ATCOs' brain states during the performance of various ATC tasks. Based on the analyses of the objective real time data together with the subjective feedback from ATCOs, we are able to reliably evaluate current ATC systems and refine new concepts of future ATC system

Neuroscience based design: Fundamentals and applications

Neuroscience-based or neuroscience-informed design is a new application area of Brain-Computer Interaction (BCI). It takes its roots in study of human well-being in architecture, human factors study in engineering and manufacturing including neuroergonomics. In traditional human factors studies and/or well-being study, mental workload, stress, and emotion are obtained through questionnaires that are administered upon completion of some task and/or the whole experiment. Recent advances in BCI research allow for using Electroencephalogram (EEG) based brain state recognition algorithms to assess the interaction between brain and human performance. We propose and develop an EEG-based system CogniMeter to monitor and analyze human factors measurements of newly designed software/hardware systems and/or working places. Machine learning techniques are applied to the EEG data to recognize levels of mental workload, stress and emotions during each task. The EEG is used as a tool to monitor and record the brain states of subjects during human factors study experiments. We describe two applications of CogniMeter system: human performance assessment in maritime simulator and EEG-based human factors evaluation in Air Traffic Control (ATC) workplace. By utilizing the proposed EEG-based system, true understanding of subjects working patterns can be obtained. Based on the analyses of the objective real time EEG-based data together with the subjective feedback from the subjects, we are able to reliably evaluate current systems/hardware and/or working place design and refine new concepts and design of future systems