Robust Hand Motion Capture and Physics-Based Control for Grasping in Real Time

Hand motion capture technologies are being explored due to high demands in the fields such as video game, virtual reality, sign language recognition, human-computer interaction, and robotics. However, existing systems suffer a few limitations, e.g. they are high-cost (expensive capture devices), intrusive (additional wear-on sensors or complex configurations), and restrictive (limited motion varieties and restricted capture space). This dissertation mainly focus on exploring algorithms and applications for the hand motion capture system that is low-cost, non-intrusive, low-restriction, high-accuracy, and robust. More specifically, we develop a realtime and fully-automatic hand tracking system using a low-cost depth camera. We first introduce an efficient shape-indexed cascaded pose regressor that directly estimates 3D hand poses from depth images. A unique property of our hand pose regressor is to utilize a low-dimensional parametric hand geometric model to learn 3D shape-indexed features robust to variations in hand shapes, viewpoints and hand poses. We further introduce a hybrid tracking scheme that effectively complements our hand pose regressor with model-based hand tracking. In addition, we develop a rapid 3D hand shape modeling method that uses a small number of depth images to accurately construct a subject-specific skinned mesh model for hand tracking. This step not only automates the whole tracking system but also improves the robustness and accuracy of model-based tracking and hand pose regression. Additionally, we also propose a physically realistic human grasping synthesis method that is capable to grasp a wide variety of objects. Given an object to be grasped, our method is capable to compute required controls (e.g. forces and torques) that advance the simulation to achieve realistic grasping. Our method combines the power of data-driven synthesis and physics-based grasping control. We first introduce a data-driven method to synthesize a realistic grasping motion from large sets of prerecorded grasping motion data. And then we transform the synthesized kinematic motion to a physically realistic one by utilizing our online physics-based motion control method. In addition, we also provide a performance interface which allows the user to act out before a depth camera to control a virtual object

Robust Hand Motion Capture and Physics-Based Control for Grasping in Real Time

Hand motion capture technologies are being explored due to high demands in the fields such as video game, virtual reality, sign language recognition, human-computer interaction, and robotics. However, existing systems suffer a few limitations, e.g. they are high-cost (expensive capture devices), intrusive (additional wear-on sensors or complex configurations), and restrictive (limited motion varieties and restricted capture space). This dissertation mainly focus on exploring algorithms and applications for the hand motion capture system that is low-cost, non-intrusive, low-restriction, high-accuracy, and robust. More specifically, we develop a realtime and fully-automatic hand tracking system using a low-cost depth camera. We first introduce an efficient shape-indexed cascaded pose regressor that directly estimates 3D hand poses from depth images. A unique property of our hand pose regressor is to utilize a low-dimensional parametric hand geometric model to learn 3D shape-indexed features robust to variations in hand shapes, viewpoints and hand poses. We further introduce a hybrid tracking scheme that effectively complements our hand pose regressor with model-based hand tracking. In addition, we develop a rapid 3D hand shape modeling method that uses a small number of depth images to accurately construct a subject-specific skinned mesh model for hand tracking. This step not only automates the whole tracking system but also improves the robustness and accuracy of model-based tracking and hand pose regression. Additionally, we also propose a physically realistic human grasping synthesis method that is capable to grasp a wide variety of objects. Given an object to be grasped, our method is capable to compute required controls (e.g. forces and torques) that advance the simulation to achieve realistic grasping. Our method combines the power of data-driven synthesis and physics-based grasping control. We first introduce a data-driven method to synthesize a realistic grasping motion from large sets of prerecorded grasping motion data. And then we transform the synthesized kinematic motion to a physically realistic one by utilizing our online physics-based motion control method. In addition, we also provide a performance interface which allows the user to act out before a depth camera to control a virtual object

A hybrid adaptive control strategy for a smart prosthetic hand

This paper presents a hybrid of a soft computing technique of adaptive neuro-fuzzy inference system (ANFIS) and a hard computing technique of adaptive control for a two- dimensional movement of a prosthetic hand with a thumb and index finger. In articular, ANFIS is used for inverse kinematics, and the adaptive control is used for linearized dynamics to minimize tracking error. The simulations of this hybrid controller, when compared with the proportional-integral-derivative (PID) controller showed enhanced performance. Work is in progress to extend this methodology to a five-fingered, three-dimensional prosthetic hand.Peer ReviewedPostprint (published version

A hybrid optimal control strategy for a smart prosthetic hand

ABSTRACT This paper presents a hybrid of a soft computing or control technique of adaptive neuro-fuzzy inference system (AN-FIS) and a hard computing or control technique of the hybrid finite-time linear quadratic optimal control for a two-fingered (thumb and index) prosthetic hand. In particular, the ANFIS is used for inverse kinematics, and the optimal control is used to minimize tracking error utilizing feedback linearized dynamics. The simulations of this hybrid controller, when compared with the proportional-integral-derivative (PID) controller showed enhanced performance. Work is underway to extend this methodology to a five-fingered, three-dimensional prosthetic hand

Monopulse tracking system Patent

Monopulse tracking system with antenna array of three radiators for deriving azimuth and elevation indication

Combining reinforcement learning and conventional control to improve automatic guided vehicles tracking of complex trajectories

With the rapid growth of logistics transportation in the framework of Industry 4.0, automated guided vehicle (AGV) technologies have developed speedily. These systems present two coupled control problems: the control of the longitudinal velocity, essential to ensure the application requirements such as throughput and tag time, and the trajectory tracking control, necessary to ensure the proper accuracy in loading and unloading manoeuvres. When the paths are very short or have abrupt changes, the kinematic constraints play a restrictive role, and the tracking control becomes more challenging. In this case, advanced control strategies such as those based on intelligent techniques, including machine learning (ML) can be useful. Hence, in this work, we present an intelligent hybrid control scheme that combines reinforcement learning-based control (RLC) with conventional PI regulators to face both control problems simultaneously. On the one hand, PIs are used to control the speed of each wheel. On the other hand, the input reference of these regulators is calculated by the RLC in order to reduce the guiding error of the path tracking and to maintain the longitudinal speed. The latter is compared with a PID path following controller. The PID regulators have been tuned by genetic algorithms. The RLC allows the vehicle to learn how to improve the trajectory tracking in an adaptive way and thus, the AGV can face disturbances or unknown physical system parameters that may change due to friction and degradation of AGV mechanical components. Extensive simulation experiments of the proposed intelligent control strategy on a hybrid tricycle and differential AGV model, that considers the kinematics and the dynamics of the vehicle, prove the efficiency of the approach when following different demanding trajectories. The performance of the RL tracking controller in comparison with the optimized PID gives errors around 70% smaller, and the average maximum error is also 48% lower.Open access funding enabled and organized by Projekt DEAL

The Hybrid BCI

Nowadays, everybody knows what a hybrid car is. A hybrid car normally has two engines to enhance energy efficiency and reduce CO2 output. Similarly, a hybrid brain-computer interface (BCI) is composed of two BCIs, or at least one BCI and another system. A hybrid BCI, like any BCI, must fulfill the following four criteria: (i) the device must rely on signals recorded directly from the brain; (ii) there must be at least one recordable brain signal that the user can intentionally modulate to effect goal-directed behaviour; (iii) real time processing; and (iv) the user must obtain feedback. This paper introduces hybrid BCIs that have already been published or are in development. We also introduce concepts for future work. We describe BCIs that classify two EEG patterns: one is the event-related (de)synchronisation (ERD, ERS) of sensorimotor rhythms, and the other is the steady-state visual evoked potential (SSVEP). Hybrid BCIs can either process their inputs simultaneously, or operate two systems sequentially, where the first system can act as a “brain switch”. For example, we describe a hybrid BCI that simultaneously combines ERD and SSVEP BCIs. We also describe a sequential hybrid BCI, in which subjects could use a brain switch to control an SSVEP-based hand orthosis. Subjects who used this hybrid BCI exhibited about half the false positives encountered while using the SSVEP BCI alone. A brain switch can also rely on hemodynamic changes measured through near-infrared spectroscopy (NIRS). Hybrid BCIs can also use one brain signal and a different type of input. This additional input can be an electrophysiological signal such as the heart rate, or a signal from an external device such as an eye tracking system

Integration of a Multi-Camera Vision System and Strapdown Inertial Navigation System (SDINS) with a Modified Kalman Filter

This paper describes the development of a modified Kalman filter to integrate a multi-camera vision system and strapdown inertial navigation system (SDINS) for tracking a hand-held moving device for slow or nearly static applications over extended periods of time. In this algorithm, the magnitude of the changes in position and velocity are estimated and then added to the previous estimation of the position and velocity, respectively. The experimental results of the hybrid vision/SDINS design show that the position error of the tool tip in all directions is about one millimeter RMS. The proposed Kalman filter removes the effect of the gravitational force in the state-space model. As a result, the resulting error is eliminated and the resulting position is smoother and ripple-free

Hydrogen at the rooftop: Compact CPV-hydrogen system to convert sunlight to hydrogen

Despite being highest potential energy source, solar intermittency and low power density make it difficult for solar energy to compete with the conventional power plants. Highly efficient concentrated photovoltaic (CPV) system provides best technology to be paired with the electrolytic hydrogen production, as a sustainable energy source with long term energy storage. However, the conventional gigantic design of CPV system limits its market and application to the open desert fields without any rooftop installation scope, unlike conventional PV. This makes CPV less popular among solar energy customers. This paper discusses the development of compact CPV-Hydrogen system for the rooftop application in the urban region. The in-house built compact CPV system works with hybrid solar tracking of 0.1° accuracy, ensured through proposed double lens collimator based solar tracking sensor. With PEM based electrolyser, the compact CPV-hydrogen system showed 28% CPV efficiency and 18% sunlight to hydrogen (STH) efficiency, for rooftop operation in tropical region of Singapore. For plant designers, the solar to hydrogen production rating of 217 kWhe/kgH2 has been presented with 15% STH daily average efficiency, recorded from the long term field operation of the syste