Motion estimation with chessboard pattern prediction strategy

Due to high correlations among the adjacent blocks, several algorithms utilize movement information of spatially and temporally correlated neighboring blocks to adapt their search patterns to that information. In this paper, this information is used to define a dynamic search pattern. Each frame is divided into two sets, black and white blocks, like a chessboard pattern and a different search pattern, is defined for each set. The advantage of this definition is that the number of spatially neighboring blocks is increased for each current block and it leads to a better prediction for each block. Simulation results show that the proposed algorithm is closer to the Full-Search algorithm in terms of quality metrics such as PSNR than the other state-of-the-art algorithms while at the same time the average number of search points is less.info:eu-repo/semantics/publishedVersio

RCDN -- Robust X-Corner Detection Algorithm based on Advanced CNN Model

Accurate detection and localization of X-corner on both planar and non-planar patterns is a core step in robotics and machine vision. However, previous works could not make a good balance between accuracy and robustness, which are both crucial criteria to evaluate the detectors performance. To address this problem, in this paper we present a novel detection algorithm which can maintain high sub-pixel precision on inputs under multiple interference, such as lens distortion, extreme poses and noise. The whole algorithm, adopting a coarse-to-fine strategy, contains a X-corner detection network and three post-processing techniques to distinguish the correct corner candidates, as well as a mixed sub-pixel refinement technique and an improved region growth strategy to recover the checkerboard pattern partially visible or occluded automatically. Evaluations on real and synthetic images indicate that the presented algorithm has the higher detection rate, sub-pixel accuracy and robustness than other commonly used methods. Finally, experiments of camera calibration and pose estimation verify it can also get smaller re-projection error in quantitative comparisons to the state-of-the-art.Comment: 15 pages, 8 figures and 4 tables. Unpublished further research and experiments of Checkerboard corner detection network CCDN (arXiv:2302.05097) and application exploration for robust camera calibration (https://ieeexplore.ieee.org/abstract/document/9428389

Modeling, Simulation and Control of Very Flexible Unmanned Aerial Vehicle

This dissertation presents research on modeling, simulation and control of very flexible aircraft. This work includes theoretical and numerical developments, as well as experimental validations. On the theoretical front, new kinematic equations for modeling sensors are derived. This formulation uses geometrically nonlinear strain-based finite elements developed as part of University of Michigan Nonlinear Aeroelastic Simulation Toolbox (UM/NAST). Numerical linearizations of both the flexible vehicle and the sensor measurements are developed, allowing a linear time invariant model to be extracted for control analysis and design. Two different algorithms to perform sensor fusion from different sensor sources to extract elastic deformation are investigated. Nonlinear least square method uses geometry and nonlinear beam strain-displacement kinematics to reconstruct the wing shape. Detailed information such as material properties or loading conditions are not required. The second method is the Kalman filter, implemented in a multi-rate form. This method requires a dynamical system representation to be available. However, it is more robust to noise corruption in sensor measurements. In order to control maneuver loads, Model Predictive Control is applied to maneuver load alleviation of a representative very flexible aircraft (X-HALE). Numerical studies are performed in UM/NAST for pitch up and roll maneuvers. Both control and state constraints are successfully enforced, while reference commands are still being tracked. MPC execution is also timed and current implementation is capable of almost real-time operation. On the experimental front, two aeroelastic testbed vehicles (ATV-6B and RRV-6B) are instrumented with sensors. On ATV-6B, an extensive set of sensors measuring structural, flight dynamic, and aerodynamic information are integrated on-board. A novel stereo-vision measurement system mounted on the body center looking towards the wing tip measures wing deformation. High brightness LEDs are used as target markers for easy detection and to allow each view to be captured with fast camera shutter speed. Experimental benchmarks are conducted to verify the accuracy of this methodology. RRV-6B flight test results are presented. System identification is applied to the experimental data to generate a SISO description of the flexible aircraft. System identification results indicate that the UM/NAST X-HALE model requires some tuning to match observed dynamics. However, the general trends predicted by the numerical model are in agreement with flight test results. Finally, using this identified plant, a stability augmentation autopilot is designed and flight tested. This augmentation autopilot utilizes a cascaded two-loop proportional integral control design, with the inner loop regulating angular rates and outer loop regulating attitude. Each of the three axes is assumed to be decoupled and designed using SISO methodology. This stabilization system demonstrates significant improvements in the RRV-6B handling qualities. This dissertation ends with a summary of the results and conclusions, and its main contribution to the field. Suggestions for future work are also presented.PHDAerospace EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttps://deepblue.lib.umich.edu/bitstream/2027.42/144019/1/pziyang_1.pd

Automatic registration with continuous pose updates for marker-less surgical navigation in spine surgery

Established surgical navigation systems for pedicle screw placement have been proven to be accurate, but still reveal limitations in registration or surgical guidance. Registration of preoperative data to the intraoperative anatomy remains a time-consuming, error-prone task that includes exposure to harmful radiation. Surgical guidance through conventional displays has well-known drawbacks, as information cannot be presented in-situ and from the surgeon's perspective. Consequently, radiation-free and more automatic registration methods with subsequent surgeon-centric navigation feedback are desirable. In this work, we present a marker-less approach that automatically solves the registration problem for lumbar spinal fusion surgery in a radiation-free manner. A deep neural network was trained to segment the lumbar spine and simultaneously predict its orientation, yielding an initial pose for preoperative models, which then is refined for each vertebra individually and updated in real-time with GPU acceleration while handling surgeon occlusions. An intuitive surgical guidance is provided thanks to the integration into an augmented reality based navigation system. The registration method was verified on a public dataset with a median of 100% successful registrations, a median target registration error of 2.7 mm, a median screw trajectory error of 1.6°and a median screw entry point error of 2.3 mm. Additionally, the whole pipeline was validated in an ex-vivo surgery, yielding a 100% screw accuracy and a median target registration error of 1.0 mm. Our results meet clinical demands and emphasize the potential of RGB-D data for fully automatic registration approaches in combination with augmented reality guidance

Perception Intelligence Integrated Vehicle-to-Vehicle Optical Camera Communication.

Ubiquitous usage of cameras and LEDs in modern road and aerial vehicles open up endless opportunities for novel applications in intelligent machine navigation, communication, and networking. To this end, in this thesis work, we hypothesize the benefit of dual-mode usage of vehicular built-in cameras through novel machine perception capabilities combined with optical camera communication (OCC). Current key conception of understanding a line-of-sight (LOS) scenery is from the aspect of object, event, and road situation detection. However, the idea of blending the non-line-of-sight (NLOS) information with the LOS information to achieve a see-through vision virtually is new. This improves the assistive driving performance by enabling a machine to see beyond occlusion. Another aspect of OCC in the vehicular setup is to understand the nature of mobility and its impact on the optical communication channel quality. The research questions gathered from both the car-car mobility modelling, and evaluating a working setup of OCC communication channel can also be inherited to aerial vehicular situations like drone-drone OCC. The aim of this thesis is to answer the research questions along these new application domains, particularly, (i) how to enable a virtual see-through perception in the car assisting system that alerts the human driver about the visible and invisible critical driving events to help drive more safely, (ii) how transmitter-receiver cars behaves while in the mobility and the overall channel performance of OCC in motion modality, (iii) how to help rescue lost Unmanned Aerial Vehicles (UAVs) through coordinated localization with fusion of OCC and WiFi, (iv) how to model and simulate an in-field drone swarm operation experience to design and validate UAV coordinated localization for group of positioning distressed drones. In this regard, in this thesis, we present the end-to-end system design, proposed novel algorithms to solve the challenges in applying such a system, and evaluation results through experimentation and/or simulation

Real-time people tracking in a camera network

Visual tracking is a fundamental key to the recognition and analysis of human behaviour. In this thesis we present an approach to track several subjects using multiple cameras in real time. The tracking framework employs a numerical Bayesian estimator, also known as a particle lter, which has been developed for parallel implementation on a Graphics Processing Unit (GPU). In order to integrate multiple cameras into a single tracking unit we represent the human body by a parametric ellipsoid in a 3D world. The elliptical boundary can be projected rapidly, several hundred times per subject per frame, onto any image for comparison with the image data within a likelihood model. Adding variables to encode visibility and persistence into the state vector, we tackle the problems of distraction and short-period occlusion. However, subjects may also disappear for longer periods due to blind spots between cameras elds of view. To recognise a desired subject after such a long-period, we add coloured texture to the ellipsoid surface, which is learnt and retained during the tracking process. This texture signature improves the recall rate from 60% to 70-80% when compared to state only data association. Compared to a standard Central Processing Unit (CPU) implementation, there is a signi cant speed-up ratio

Imitrob: Imitation Learning Dataset for Training and Evaluating 6D Object Pose Estimators

This paper introduces a dataset for training and evaluating methods for 6D pose estimation of hand-held tools in task demonstrations captured by a standard RGB camera. Despite the significant progress of 6D pose estimation methods, their performance is usually limited for heavily occluded objects, which is a common case in imitation learning where the object is typically partially occluded by the manipulating hand. Currently, there is a lack of datasets that would enable the development of robust 6D pose estimation methods for these conditions. To overcome this problem, we collect a new dataset (Imitrob) aimed at 6D pose estimation in imitation learning and other applications where a human holds a tool and performs a task. The dataset contains image sequences of three different tools and six manipulation tasks with two camera viewpoints, four human subjects, and left/right hand. Each image is accompanied by an accurate ground truth measurement of the 6D object pose, obtained by the HTC Vive motion tracking device. The use of the dataset is demonstrated by training and evaluating a recent 6D object pose estimation method (DOPE) in various setups. The dataset and code are publicly available at http://imitrob.ciirc.cvut.cz/imitrobdataset.php