A versatile high-performance visual fiducial marker detection system with scalable identity encoding

Fiducial markers have a wide field of applications in robotics, ranging from external localisation of single robots or robotic swarms, over self-localisation in marker-augmented environments, to simplifying perception by tagging objects in a robot’s surrounding. We propose a new family of circular markers allowing for a computationally efficient detection, identification and full 3D position estimation. A key concept of our system is the separation of the detection and identification steps, where the first step is based on a computationally efficient circular marker detection, and the identification step is based on an open-ended ‘Necklace code’, which allows for a theoretically infinite number of individually identifiable markers. The experimental evaluation of the system on a real robot indicates that while the proposed algorithm achieves similar accuracy to other state-of-the-art methods, it is faster by two orders of magnitude and it can detect markers from longer distances

Contribuciones a la estimación de la pose de la cámara en aplicaciones industriales de realidad aumentada

Augmented Reality (AR) aims to complement the visual perception of the user environment superimposing virtual elements. The main challenge of this technology is to combine the virtual and real world in a precise and natural way. To carry out this goal, estimating the user position and orientation in both worlds at all times is a crucial task. Currently, there are numerous techniques and algorithms developed for camera pose estimation. However, the use of synthetic square markers has become the fastest, most robust and simplest solution in these cases. In this scope, a big number of marker detection systems have been developed. Nevertheless, most of them presents some limitations, (1) their unattractive and non-customizable visual appearance prevent their use in industrial products and (2) the detection rate is drastically reduced in presence of noise, blurring and occlusions. In this doctoral dissertation the above-mentioned limitations are addressed. In first place, a comparison has been made between the different marker detection systems currently available in the literature, emphasizing the limitations of each. Secondly, a novel approach to design, detect and track customized markers capable of easily adapting to the visual limitations of commercial products has been developed. In third place, a method that combines the detection of black and white square markers with keypoints and contours has been implemented to estimate the camera position in AR applications. The main motivation of this work is to offer a versatile alternative (based on contours and keypoints) in cases where, due to noise, blurring or occlusions, it is not possible to identify markers in the images. Finally, a method for reconstruction and semantic segmentation of 3D objects using square markers in photogrammetry processes has been presented.La Realidad Aumentada (AR) tiene como objetivo complementar la percepción visual del entorno circunstante al usuario mediante la superposición de elementos virtuales. El principal reto de dicha tecnología se basa en fusionar, de forma precisa y natural, el mundo virtual con el mundo real. Para llevar a cabo dicha tarea, es de vital importancia conocer en todo momento tanto la posición, así como la orientación del usuario en ambos mundos. Actualmente, existen un gran número de técnicas de estimación de pose. No obstante, el uso de marcadores sintéticos cuadrados se ha convertido en la solución más rápida, robusta y sencilla utilizada en estos casos. En este ámbito de estudio, existen un gran número de sistemas de detección de marcadores ampliamente extendidos. Sin embargo, su uso presenta ciertas limitaciones, (1) su aspecto visual, poco atractivo y nada customizable impiden su uso en ciertos productos industriales en donde la personalización comercial es un aspecto crucial y (2) la tasa de detección se ve duramente decrementada ante la presencia de ruido, desenfoques y oclusiones Esta tesis doctoral se ocupa de las limitaciones anteriormente mencionadas. En primer lugar, se ha realizado una comparativa entre los diferentes sistemas de detección de marcadores actualmente en uso, enfatizando las limitaciones de cada uno. En segundo lugar, se ha desarrollado un novedoso enfoque para diseñar, detectar y trackear marcadores personalizados capaces de adaptarse fácilmente a las limitaciones visuales de productos comerciales. En tercer lugar, se ha implementado un método que combina la detección de marcadores cuadrados blancos y negros con keypoints y contornos, para estimar de la posición de la cámara en aplicaciones AR. La principal motivación de este trabajo se basa en ofrecer una alternativa versátil (basada en contornos y keypoints) en aquellos casos donde, por motivos de ruido, desenfoques u oclusiones no sea posible identificar marcadores en las imágenes. Por último, se ha desarrollado un método de reconstrucción y segmentación semántica de objetos 3D utilizando marcadores cuadrados en procesos de fotogrametría

Passive Resonant Coil Based Fast Registration And Tracking System For Real-Time Mri-Guided Minimally Invasive Surgery

This thesis presents a single-slice based fast stereotactic registration and tracking technique along with a corresponding modular system for guiding robotic mechanism or interventional instrument to perform needle-based interventions under live MRI guidance. The system can provide tracking of full 6 degree-of-freedom (DOF) in stereotactic interventional surgery based upon a single, rapidly acquired cross-sectional image. The whole system is constructed with a modular data transmission software framework and mechanical structure so that it supports remote supervision and manipulation between a 3D Matlab tracking user interface (UI) and an existing MRI robot controller by using the OpenIGTLink network communication protocol. It provides better closed-loop control by implementing a feedback output interface to the MRI-guided robot. A new compact fiducial frame design is presented, and the fiducial is wrapped with a passive resonant coil. The coil resonates at the Larmor frequency for 3T MRI to enhance signal strength and enable for rapid imaging. The fiducial can be attached near the distal end of the robot and coaxially with a needle so as to visualize target tissue and track the surgical tool synchronously. The MRI-compatible design of fiducial frame, robust tracking algorithm and modular interface allow this tracking system to be conveniently used on different robots or devices and in different size of MRI bores. Several iterations of the tracking fiducial and passive resonant coils were constructed and evaluated in a Phillips Achieva 3T MRI. To assess accuracy and robustness of the tracking algorithm, 25 groups of images with different poses were successively scanned along specific sequence in and MRI experiment. The translational RMS error along depth is 0.271mm with standard deviation of 0.277mm for totally 100 samples. The overall angular RMS error is less than 0.426 degree with standard deviation of 0.526 degree for totally 150 samples. The passive resonant coils were shown to significantly increase signal intensity in the fiducial relative to the surroundings and provide for rapid imaging with low flip angles

Towards a Vision-Based Mobile Manipulator for Autonomous Chess Gameplay

With the rise of robotic arms in both industrial and research applications, a growing need is observed for autonomous robotic arm applications. This thesis aims to provide an example case of this need and also to showcase the possibility and limitations of vision-based solutions, specifically in automating chess. The focus is on developing a modular system that is able to autonomously recognize chessboard, detect and manipulate chess pieces. The modular design allows for further exploration into autonomous mobile manipulators. The key components include chessboard recognition using fiducial markers to facilitate accurate chessboard recognition and utilizing image processing techniques like segmentation, absolute difference matching, and perspective warping to analyze and extract meaningful information. By mounting a camera above the chessboard, it enables the detection algorithm to accurately capture and analyze the most important information about the environment to determine the current state of the game. Using this information, human move detection is enabled. Then, a custom protocol is utilized to communicate between the detection algorithm and the chess engine, encapsulating information about the game state changes within the system. The chess engine serves the purpose of game analysis and provides legal moves for the robot manipulator to execute. Manipulation happens through careful motion planning and execution, ensuring the safety of the robot and its environment. Extensive evaluation proves that the system demonstrates high accuracy and success rates for piece manipulation and move detection

ViSE: Vision-Based 3D Online Shape Estimation of Continuously Deformable Robots

The precise control of soft and continuum robots requires knowledge of their shape. The shape of these robots has, in contrast to classical rigid robots, infinite degrees of freedom. To partially reconstruct the shape, proprioceptive techniques use built-in sensors resulting in inaccurate results and increased fabrication complexity. Exteroceptive methods so far rely on placing reflective markers on all tracked components and triangulating their position using multiple motion-tracking cameras. Tracking systems are expensive and infeasible for deformable robots interacting with the environment due to marker occlusion and damage. Here, we present a regression approach for 3D shape estimation using a convolutional neural network. The proposed approach takes advantage of data-driven supervised learning and is capable of real-time marker-less shape estimation during inference. Two images of a robotic system are taken simultaneously at 25 Hz from two different perspectives, and are fed to the network, which returns for each pair the parameterized shape. The proposed approach outperforms marker-less state-of-the-art methods by a maximum of 4.4% in estimation accuracy while at the same time being more robust and requiring no prior knowledge of the shape. The approach can be easily implemented due to only requiring two color cameras without depth and not needing an explicit calibration of the extrinsic parameters. Evaluations on two types of soft robotic arms and a soft robotic fish demonstrate our method's accuracy and versatility on highly deformable systems in real-time. The robust performance of the approach against different scene modifications (camera alignment and brightness) suggests its generalizability to a wider range of experimental setups, which will benefit downstream tasks such as robotic grasping and manipulation

Study and development of a reliable fiducials-based localization system for multicopter UAVs flying indoor

openThe recent evolution of technology in automation, agriculture, IoT, and aerospace fields has created a growing demand for mobile robots capable of autonomous operation and movement to accomplish various tasks. Aerial platforms are expected to play a central role in the future due to their versatility and swift intervention capabilities. However, the effective utilization of these platforms faces a significant challenge due to localization, which is a vital aspect for their interaction with the surrounding environment. While GNSS localization systems have established themselves as reliable solutions for open-space scenarios, the same approach is not viable for indoor settings, where localization remains an open problem as it is witnessed by the lack of extensive literature on the topic. In this thesis, we address this challenge by proposing a dependable solution for small multi-rotor UAVs using a Visual Inertial Odometry localization system. Our KF-based localization system reconstructs the pose by fusing data from onboard sensors. The primary source of information stems from the recognition of AprilTags fiducial markers, strategically placed in known positions to form a “map”. Building upon prior research and thesis work conducted at our university, we extend and enhance this system. We begin with a concise introduction, followed by a justification of our chosen strategies based on the current state of the art. We provide an overview of the key theoretical, mathematical, and technical aspects that support our work. These concepts are fundamental to the design of innovative strategies that address challenges such as data fusion from different AprilTag recognition and the elimination of misleading measurements. To validate our algorithms and their implementation, we conduct experimental tests using two distinct platforms by using localization accuracy and computational complexity as performance indices to demonstrate the practical viability of our proposed system. By tackling the critical issue of indoor localization for aerial platforms, this thesis tries to give some contribution to the advancement of robotics technology, opening avenues for enhanced autonomy and efficiency across various domains.The recent evolution of technology in automation, agriculture, IoT, and aerospace fields has created a growing demand for mobile robots capable of autonomous operation and movement to accomplish various tasks. Aerial platforms are expected to play a central role in the future due to their versatility and swift intervention capabilities. However, the effective utilization of these platforms faces a significant challenge due to localization, which is a vital aspect for their interaction with the surrounding environment. While GNSS localization systems have established themselves as reliable solutions for open-space scenarios, the same approach is not viable for indoor settings, where localization remains an open problem as it is witnessed by the lack of extensive literature on the topic. In this thesis, we address this challenge by proposing a dependable solution for small multi-rotor UAVs using a Visual Inertial Odometry localization system. Our KF-based localization system reconstructs the pose by fusing data from onboard sensors. The primary source of information stems from the recognition of AprilTags fiducial markers, strategically placed in known positions to form a “map”. Building upon prior research and thesis work conducted at our university, we extend and enhance this system. We begin with a concise introduction, followed by a justification of our chosen strategies based on the current state of the art. We provide an overview of the key theoretical, mathematical, and technical aspects that support our work. These concepts are fundamental to the design of innovative strategies that address challenges such as data fusion from different AprilTag recognition and the elimination of misleading measurements. To validate our algorithms and their implementation, we conduct experimental tests using two distinct platforms by using localization accuracy and computational complexity as performance indices to demonstrate the practical viability of our proposed system. By tackling the critical issue of indoor localization for aerial platforms, this thesis tries to give some contribution to the advancement of robotics technology, opening avenues for enhanced autonomy and efficiency across various domains

A practical multirobot localization system

We present a fast and precise vision-based software intended for multiple robot localization. The core component of the software is a novel and efficient algorithm for black and white pattern detection. The method is robust to variable lighting conditions, achieves sub-pixel precision and its computational complexity is independent of the processed image size. With off-the-shelf computational equipment and low-cost cameras, the core algorithm is able to process hundreds of images per second while tracking hundreds of objects with a millimeter precision. In addition, we present the method's mathematical model, which allows to estimate the expected localization precision, area of coverage, and processing speed from the camera's intrinsic parameters and hardware's processing capacity. The correctness of the presented model and performance of the algorithm in real-world conditions is verified in several experiments. Apart from the method description, we also make its source code public at \emph{http://purl.org/robotics/whycon}; so, it can be used as an enabling technology for various mobile robotic problems