A COLOR FEATURES-BASED METHOD FOR OBJECT TRACKING EMPLOYING A PARTICLE FILTER ALGORITHM

We proposed a method for object tracking employing a particle filter based on color feature method. A histogram‐based framework is used to describe the features. Histograms are useful because they have property that they allow changes in the object appearance while the histograms remain the same. Particle filtering is used because it is very robust for non‐linear and non‐Gaussian dynamic state estimation problems and performs well when clutter and occlusions are present on the image. Bhattacharyya distance is used to weight the samples in the particle filter by comparing each sample’s histogram with a specified target model and it makes the measurement matching and sample’s weight updating more reasonable. The method is capable to track successfully the moving object in different outdoor environment with and without initial positions information, and also, capable to track the moving object in the presence of occlusion using an appearance condition. In this paper, we propose a color features‐based method for object tracking based on the particle filters. The experimental results and data show the feasibility and the effectiveness of our method.International Conference on Power Control and Optimization, 1-3, June 2009, Bali, Indonesi

Performance Analysis of Tracking on Mobile Devices using Local Binary Descriptors

With the growing ubiquity of mobile devices, users are turning to their smartphones and tablets to perform more complex tasks than ever before. Performing computer vision tasks on mobile devices must be done despite the constraints on CPU performance, memory, and power consumption. One such task for mobile devices involves object tracking, an important area of computer vision. The computational complexity of tracking algorithms makes them ideal candidates for optimization on mobile platforms. This thesis presents a mobile implementation for real time object tracking. Currently few tracking approaches take into consideration the resource constraints on mobile devices. Optimizing performance for mobile devices can result in better and more efficient tracking approaches for mobile applications such as augmented reality. These performance benefits aim to increase the frame rate at which an object is tracked and reduce power consumption during tracking. For this thesis, we utilize binary descriptors, such as Binary Robust Independent Elementary Features (BRIEF), Oriented FAST and Rotated BRIEF (ORB), Binary Robust Invariant Scalable Keypoints (BRISK), and Fast Retina Keypoint (FREAK). The tracking performance of these descriptors is benchmarked on mobile devices. We consider an object tracking approach based on a dictionary of templates that involves generating keypoints of a detected object and candidate regions in subsequent frames. Descriptor matching, between candidate regions in a new frame and a dictionary of templates, identifies the location of the tracked object. These comparisons are often computationally intensive and require a great deal of memory and processing time. Google\u27s Android operating system is used to implement the tracking application on a Samsung Galaxy series phone and tablet. Control of the Android camera is largely done through OpenCV\u27s Android SDK. Power consumption is measured using the PowerTutor Android application. Other performance characteristics, such as processing time, are gathered using the Dalvik Debug Monitor Server (DDMS) tool included in the Android SDK. These metrics are used to evaluate the tracker\u27s performance on mobile devices

Cooperative Object Manipulation with Force Tracking on the da Vinci Research Kit

The da Vinci Surgical System is one of the most established robot-assisted surgery device commended for its dexterity and ergonomics in minimally invasive surgery. Conversely, it inherits disadvantages which are lack of autonomy and haptic feedback. In order to address these issues, this work proposes an industry-inspired solution to the field of force control in medical robotics. This approach contributes to shared autonomy by developing a controller for cooperative object manipulation with force tracking utilizing available manipulators and force feedback. To achieve simultaneous position and force tracking of the object, master and slave manipulators were assigned then controlled with Cartesian position control and impedance control respectively. Because impedance control requires a model-based feedforward compensation, we identified the lumped base parameters of mass, inertias, and frictions of a three degree-of-freedom double four-bar linkage mechanism with least squares and weighted least squares regression methods. Additionally, semidefinite programming was used to constrain the parameters to a feasible physical solution in standard parameter space. Robust stick-slip static friction compensation was applied where linear Viscous and Coulomb friction was inadequate in modeling the prismatic third joint. The Robot Operating System based controller was tested in RViz to check the cooperative kinematics of up to three manipulators. Additionally, simulation with the dynamic engine Gazebo verified the cooperative controller applying a constant tension force on a massless spring-damper virtual object. With adequate model feedback linearization, the cooperative impedance controller tested on the da Vinci Research Kit yielded stable tension force tracking while simultaneously moving in Cartesian space. The maximum force tracking error was +/- 0.5 N for both a compliant and stiff manipulated object

3D pose estimation based on planar object tracking for UAVs control

This article presents a real time Unmanned Aerial Vehicles UAVs 3D pose estimation method using planar object tracking, in order to be used on the control system of a UAV. The method explodes the rich information obtained by a projective transformation of planar objects on a calibrated camera. The algorithm obtains the metric and projective components of a reference object (landmark or helipad) with respect to the UAV camera coordinate system, using a robust real time object tracking based on homographies. The algorithm is validated on real flights that compare the estimated data against that obtained by the inertial measurement unit IMU, showing that the proposed method robustly estimates the helicopter's 3D position with respect to a reference landmark, with a high quality on the position and orientation estimation when the aircraft is flying at low altitudes, a situation in which the GPS information is often inaccurate. The obtained results indicate that the proposed algorithm is suitable for complex control tasks, such as autonomous landing, accurate low altitude positioning and dropping of payloads

Robust Hand Motion Capture and Physics-Based Control for Grasping in Real Time

Hand motion capture technologies are being explored due to high demands in the fields such as video game, virtual reality, sign language recognition, human-computer interaction, and robotics. However, existing systems suffer a few limitations, e.g. they are high-cost (expensive capture devices), intrusive (additional wear-on sensors or complex configurations), and restrictive (limited motion varieties and restricted capture space). This dissertation mainly focus on exploring algorithms and applications for the hand motion capture system that is low-cost, non-intrusive, low-restriction, high-accuracy, and robust. More specifically, we develop a realtime and fully-automatic hand tracking system using a low-cost depth camera. We first introduce an efficient shape-indexed cascaded pose regressor that directly estimates 3D hand poses from depth images. A unique property of our hand pose regressor is to utilize a low-dimensional parametric hand geometric model to learn 3D shape-indexed features robust to variations in hand shapes, viewpoints and hand poses. We further introduce a hybrid tracking scheme that effectively complements our hand pose regressor with model-based hand tracking. In addition, we develop a rapid 3D hand shape modeling method that uses a small number of depth images to accurately construct a subject-specific skinned mesh model for hand tracking. This step not only automates the whole tracking system but also improves the robustness and accuracy of model-based tracking and hand pose regression. Additionally, we also propose a physically realistic human grasping synthesis method that is capable to grasp a wide variety of objects. Given an object to be grasped, our method is capable to compute required controls (e.g. forces and torques) that advance the simulation to achieve realistic grasping. Our method combines the power of data-driven synthesis and physics-based grasping control. We first introduce a data-driven method to synthesize a realistic grasping motion from large sets of prerecorded grasping motion data. And then we transform the synthesized kinematic motion to a physically realistic one by utilizing our online physics-based motion control method. In addition, we also provide a performance interface which allows the user to act out before a depth camera to control a virtual object

Robust Hand Motion Capture and Physics-Based Control for Grasping in Real Time

Hand motion capture technologies are being explored due to high demands in the fields such as video game, virtual reality, sign language recognition, human-computer interaction, and robotics. However, existing systems suffer a few limitations, e.g. they are high-cost (expensive capture devices), intrusive (additional wear-on sensors or complex configurations), and restrictive (limited motion varieties and restricted capture space). This dissertation mainly focus on exploring algorithms and applications for the hand motion capture system that is low-cost, non-intrusive, low-restriction, high-accuracy, and robust. More specifically, we develop a realtime and fully-automatic hand tracking system using a low-cost depth camera. We first introduce an efficient shape-indexed cascaded pose regressor that directly estimates 3D hand poses from depth images. A unique property of our hand pose regressor is to utilize a low-dimensional parametric hand geometric model to learn 3D shape-indexed features robust to variations in hand shapes, viewpoints and hand poses. We further introduce a hybrid tracking scheme that effectively complements our hand pose regressor with model-based hand tracking. In addition, we develop a rapid 3D hand shape modeling method that uses a small number of depth images to accurately construct a subject-specific skinned mesh model for hand tracking. This step not only automates the whole tracking system but also improves the robustness and accuracy of model-based tracking and hand pose regression. Additionally, we also propose a physically realistic human grasping synthesis method that is capable to grasp a wide variety of objects. Given an object to be grasped, our method is capable to compute required controls (e.g. forces and torques) that advance the simulation to achieve realistic grasping. Our method combines the power of data-driven synthesis and physics-based grasping control. We first introduce a data-driven method to synthesize a realistic grasping motion from large sets of prerecorded grasping motion data. And then we transform the synthesized kinematic motion to a physically realistic one by utilizing our online physics-based motion control method. In addition, we also provide a performance interface which allows the user to act out before a depth camera to control a virtual object

Global Feed-Forward Adaptive Fuzzy Control of Uncertain MIMO Nonlinear Systems

This study proposes a novel adaptive control approach using a feedforward Takagi-Sugeno (TS) fuzzy approximator for a class of highly unknown multi-input multi-output (MIMO) nonlinear plants. First of all, the design concept, namely, feedforward fuzzy approximator (FFA) based control, is introduced to compensate the unknown feedforward terms required during steady state via a forward TS fuzzy system which takes the desired commands as the input variables. Different from the traditional fuzzy approximation approaches, this scheme allows easier implementation and drops the boundedness assumption on fuzzy universal approximation errors. Furthermore, the controller is synthesized to assure either the disturbance attenuation or the attenuation of both disturbances and estimated fuzzy parameter errors or globally asymptotic stable tracking. In addition, all the stability is guaranteed from a feasible gain solution of the derived linear matrix inequality (LMI). Meanwhile, the highly uncertain holonomic constrained systems are taken as applications with either guaranteed robust tracking performances or asymptotic stability in a global sense. It is demonstrated that the proposed adaptive control is easily and straightforwardly extended to the robust TS FFA-based motion/force tracking controller. Finally, two planar robots transporting a common object is taken as an application example to show the expected performance. The comparison between the proposed and traditional adaptive fuzzy control schemes is also performed in numerical simulations. Keywords: Adaptive control; Takagi-Sugeno (TS) fuzzy system; holonomic systems; motion/force control

EARL: Eye-on-Hand Reinforcement Learner for Dynamic Grasping with Active Pose Estimation

In this paper, we explore the dynamic grasping of moving objects through active pose tracking and reinforcement learning for hand-eye coordination systems. Most existing vision-based robotic grasping methods implicitly assume target objects are stationary or moving predictably. Performing grasping of unpredictably moving objects presents a unique set of challenges. For example, a pre-computed robust grasp can become unreachable or unstable as the target object moves, and motion planning must also be adaptive. In this work, we present a new approach, Eye-on-hAnd Reinforcement Learner (EARL), for enabling coupled Eye-on-Hand (EoH) robotic manipulation systems to perform real-time active pose tracking and dynamic grasping of novel objects without explicit motion prediction. EARL readily addresses many thorny issues in automated hand-eye coordination, including fast-tracking of 6D object pose from vision, learning control policy for a robotic arm to track a moving object while keeping the object in the camera's field of view, and performing dynamic grasping. We demonstrate the effectiveness of our approach in extensive experiments validated on multiple commercial robotic arms in both simulations and complex real-world tasks.Comment: Presented on IROS 2023 Corresponding author Siddarth Jai

Robust online visual tracking

University of Technology Sydney. Faculty of Engineering and Information Technology.Visual tracking plays a key role in many computer vision systems. In this thesis, we study online visual object tracking and try to tackle challenges that present in practical tracking scenarios. Motivated by different challenges, several robust online visual trackers have been developed by taking advantage of advanced techniques from machine learning and computer vision. In particular, we propose a robust distracter-resistant tracking approach by learning a discriminative metric to handle distracter problem. The proposed metric is elaborately designed for the tracking problem by forming a margin objective function which systematically includes distance margin maximization, reconstruction error constraint, and similarity propagation techniques. The distance metric obtained helps to preserve the most discriminative information to separate the target from distracters while ensuring the stability of the optimal metric. To handle background clutter problem and achieve better tracking performance, we develop a tracker using an approximate Least Absolute Deviation (LAD)-based multi-task multi-view sparse learning method to enjoy robustness of LAD and take advantage of multiple types of visual features. The proposed method is integrated in a particle filter framework where learning the sparse representation for each view of a single particle is regarded as an individual task. The underlying relationship between tasks across different views and different particles is jointly exploited in a unified robust multi-task formulation based on LAD. In addition, to capture the frequently emerging outlier tasks, we decompose the representation matrix to two collaborative components which enable a more robust and accurate approximation. In addition, a hierarchical appearance representation model is proposed for non-rigid object tracking, based on a graphical model that exploits shared information across multiple quantization levels. The tracker aims to find the most possible position of the target by jointly classifying the pixels and superpixels and obtaining the best configuration across all levels. The motion of the bounding box is taken into consideration, while Online Random Forests are used to provide pixel- and superpixel-level quantizations and progressively updated on-the-fly. Finally, inspired by the well-known Atkinson-Shiffrin Memory Model, we propose MUlti-Store Tracker, a dual-component approach consisting of short- and long-term memory stores to process target appearance memories. A powerful and efficient Integrated Correlation Filter is employed in the short-term store for short-term tracking. The integrated long-term component, which is based on keypoint matching-tracking and RANSAC estimation, can interact with the long-term memory and provide additional information for output control

Flight Dynamics-based Recovery of a UAV Trajectory using Ground Cameras

We propose a new method to estimate the 6-dof trajectory of a flying object such as a quadrotor UAV within a 3D airspace monitored using multiple fixed ground cameras. It is based on a new structure from motion formulation for the 3D reconstruction of a single moving point with known motion dynamics. Our main contribution is a new bundle adjustment procedure which in addition to optimizing the camera poses, regularizes the point trajectory using a prior based on motion dynamics (or specifically flight dynamics). Furthermore, we can infer the underlying control input sent to the UAV's autopilot that determined its flight trajectory. Our method requires neither perfect single-view tracking nor appearance matching across views. For robustness, we allow the tracker to generate multiple detections per frame in each video. The true detections and the data association across videos is estimated using robust multi-view triangulation and subsequently refined during our bundle adjustment procedure. Quantitative evaluation on simulated data and experiments on real videos from indoor and outdoor scenes demonstrates the effectiveness of our method