Comparing Fiducial Marker Systems Occlusion Resilience through a Robot Eye

© 2017 IEEE. A fiducial marker is a system of unique planar markers, that are placed in an environment and should be automatically detected with a camera through marker-specific detection procedures. Their application varies greatly, while the most popular are industrial systems, augmented reality, and robot navigation. All these applications imply that a marker system must be robust to such factors as view angles, types of occlusions, distance and light condition variations etc. Our paper compares existing ARTag, AprilTag, and CALTag systems utilizing a high fidelity camera, which is a main vision sensor of a full-size Russian humanoid robot AR-601M. Our experimental comparison verified the three marker systems reliability and detection rate in occlusions of various types and intensities and a preferable for AR-601M robot applications marker system was selected

Target Tracking Using Optical Markers for Remote Handling in ITER

The thesis focuses on the development of a vision system to be used in the remote handling systems of the International Thermonuclear Experimental Rector - ITER. It presents and discusses a realistic solution to estimate the pose of key operational targets, while taking into account the specific needs and restrictions of the application. The contributions to the state of the art are in two main fronts: 1) the development of optical markers that can withstand the extreme conditions in the environment; 2) the development of a robust marker detection and identification framework that can be effectively applied to different use cases. The markers’ locations and labels are used in computing the pose. In the first part of the work, a retro reflective marker made up ITER compliant materials, particularly, fused silica and stainless steel, is designed. A methodology is proposed to optimize the markers’ performance. Highly distinguishable markers are manufactured and tested. In the second part, a hybrid pipeline is proposed that detects uncoded markers in low resolution images using classical methods and identifies them using a machine learning approach. It is demonstrated that the proposed methodology effectively generalizes to different marker constellations and can successfully detect both retro reflective markers and laser engravings. Lastly, a methodology is developed to evaluate the end-to-end accuracy of the proposed solution using the feedback provided by an industrial robotic arm. Results are evaluated in a realistic test setup for two significantly different use cases. Results show that marker based tracking is a viable solution for the problem at hand and can provide superior performance to the earlier stereo matching based approaches. The developed solutions could be applied to other use cases and applications

Contribuciones a la estimación de la pose de la cámara en aplicaciones industriales de realidad aumentada

Augmented Reality (AR) aims to complement the visual perception of the user environment superimposing virtual elements. The main challenge of this technology is to combine the virtual and real world in a precise and natural way. To carry out this goal, estimating the user position and orientation in both worlds at all times is a crucial task. Currently, there are numerous techniques and algorithms developed for camera pose estimation. However, the use of synthetic square markers has become the fastest, most robust and simplest solution in these cases. In this scope, a big number of marker detection systems have been developed. Nevertheless, most of them presents some limitations, (1) their unattractive and non-customizable visual appearance prevent their use in industrial products and (2) the detection rate is drastically reduced in presence of noise, blurring and occlusions. In this doctoral dissertation the above-mentioned limitations are addressed. In first place, a comparison has been made between the different marker detection systems currently available in the literature, emphasizing the limitations of each. Secondly, a novel approach to design, detect and track customized markers capable of easily adapting to the visual limitations of commercial products has been developed. In third place, a method that combines the detection of black and white square markers with keypoints and contours has been implemented to estimate the camera position in AR applications. The main motivation of this work is to offer a versatile alternative (based on contours and keypoints) in cases where, due to noise, blurring or occlusions, it is not possible to identify markers in the images. Finally, a method for reconstruction and semantic segmentation of 3D objects using square markers in photogrammetry processes has been presented.La Realidad Aumentada (AR) tiene como objetivo complementar la percepción visual del entorno circunstante al usuario mediante la superposición de elementos virtuales. El principal reto de dicha tecnología se basa en fusionar, de forma precisa y natural, el mundo virtual con el mundo real. Para llevar a cabo dicha tarea, es de vital importancia conocer en todo momento tanto la posición, así como la orientación del usuario en ambos mundos. Actualmente, existen un gran número de técnicas de estimación de pose. No obstante, el uso de marcadores sintéticos cuadrados se ha convertido en la solución más rápida, robusta y sencilla utilizada en estos casos. En este ámbito de estudio, existen un gran número de sistemas de detección de marcadores ampliamente extendidos. Sin embargo, su uso presenta ciertas limitaciones, (1) su aspecto visual, poco atractivo y nada customizable impiden su uso en ciertos productos industriales en donde la personalización comercial es un aspecto crucial y (2) la tasa de detección se ve duramente decrementada ante la presencia de ruido, desenfoques y oclusiones Esta tesis doctoral se ocupa de las limitaciones anteriormente mencionadas. En primer lugar, se ha realizado una comparativa entre los diferentes sistemas de detección de marcadores actualmente en uso, enfatizando las limitaciones de cada uno. En segundo lugar, se ha desarrollado un novedoso enfoque para diseñar, detectar y trackear marcadores personalizados capaces de adaptarse fácilmente a las limitaciones visuales de productos comerciales. En tercer lugar, se ha implementado un método que combina la detección de marcadores cuadrados blancos y negros con keypoints y contornos, para estimar de la posición de la cámara en aplicaciones AR. La principal motivación de este trabajo se basa en ofrecer una alternativa versátil (basada en contornos y keypoints) en aquellos casos donde, por motivos de ruido, desenfoques u oclusiones no sea posible identificar marcadores en las imágenes. Por último, se ha desarrollado un método de reconstrucción y segmentación semántica de objetos 3D utilizando marcadores cuadrados en procesos de fotogrametría

Development of perception module for bobotic manipulation tasks

Robots performing manipulation tasks require the accurate location and orientation of an object in space. Previously, at the Robotics Laboratory of IOC-UPC this data has been generated artificially. In order to automate the process, a perception module has been developed for providing task and motion planners with the localization and pose estimation of objects used in robot manipulation tasks. The Robot Operating System provided a great framework for incorporating vision provided by Microsoft Kinect V2 sensors and the presentation of obtained data to be used in the generation of Planning Domain Definition Language files, which define a robots environment. Localization and pose estimation was done using fiducial markers along with studying possible enhancements using deep learning methods. Perfectly calibrating hardware and setting up a system play a big role in enhancing perception accuracy and while fiducial markers provide a simple and robust solution in laboratory conditions, real world applications with varying lighting, viewing angles and partial occlusions should rely on AI visio

Developing a control architecture for a vision based automatic pallet picking

The goal of this project is to enhance the controlling performance of an articulated-frame-steering autonomous machine. The current system employs the vision sensor for the detection of the target object and controlling the machine movement by the data obtained by that. The problem we face is that at a far distance when location of the object is detected for the first time, the unreliable data specially the orientation of target will be acquired. So relying on this type of data for heading the machine can distract the whole system behavior, and also there should be a logic for switching between different states of machine in order to control the different stages of performance. In order to solve the problem, a smooth switching logic will be defined to control the machine. This switching logic should be in a case that operational coordinates with the visual servoing be synchronized and it should be planned to control robot degrees of freedom in each step. The MATLAB/Simulink is employed to execute the idea since this software is capable in the logic-based planning in a graphical way in stateflow. The results of the implemented method would be increasing the accuracy of picking the object and enabling the machine to have more space to have any needed maneuver, moreover the environmental disturbances will have the minimum effect on the final result

Scheduling Optimization And Coordination With Target Tracking Under Heterogeneous Networks In Automated Guided Vehicles (AGVs)

Throughout the development of the multi-AGV systems, prevailing research directions contain improving the performance of individual AGV, optimizing the coordination of multiple AGVs, and enhancing the efficiency of communication among AGVs. Current researchers tend to pay attention to one research direction at a time. There is a lack of research on the overall AGV system design that tackles multiple critical design aspects of the system. This PhD research addresses four key factors of the AGV system which are AGV prototypes, target tracking algorithms, AGVs scheduling optimization and the communication of a multi-AGV system. Extensive field experiments and algorithm optimization are implemented. Comprehensive literature review is conducted to identify the research gap. The proposed solutions cover the following three aspects of the AGV system design including communication between AGVs, AGVs scheduling and computer vision in AGVs.        For AGV communication, a network selection optimization algorithm is presented. An improved method for preventing convolutional neural network (CNN) immune from backdoor attack to ensure a multi-AGV system's communication security is presented. Meanwhile, a transmission framework for a multi-AGV system is presented. Those methods are used to establish a safe and efficient multi-AGV system's communication environment. For AGV scheduling, a multi-robot planning algorithm with quadtree map division for obstacles of irregular shape is presented. In addition, a scheduling optimization platform is presented. These methods are used to make a multi-AGV system have a shorter time delay and decrease the possibility of collision in a multi-robot system.Meanwhile, a scheduling optimization method based on the combination of a handover rule and the A* algorithm is proposed. The system properties that may affect the scheduling performance are also discussed. Finally, the overall performance of the newly integrated scheduling system is compared with other scheduling systems to validate its superiority and shortcomings in different corresponding work scenarios. Computer vision in AGV is investigated in detail. To improve an individual AGV's performance, an improved Camshift Algorithm has been proposed and applied to AGV prototypes. Furthermore, three deep learning models are tested under specific environments. In addition, based on the designed algorithm, the AGV prototype is able to make a convergent prediction of the pixels in the target area after the first detection of the object. Relative coordinates of the target can be located more accurately in less time. As tested in the experiments, the system architecture and new algorithm lead to reduced hardware cost, shorter time delay, improved robustness, and higher accuracy in tracking.        With the three design aspects in mind, a novel method for real-time visual tracking and distance measurement is proposed. Tracking and collision avoidance functions are tested in the designed multi-AGV prototype system. Detailed design procedure, numerical analysis of the measurement data and recommendations for further improvement of the system design are presented

Computer Vision System for Tactode Programming

A programação tangível, quando direcionada à robótica, torna a atividade de programar mais compreensível e direta. Este tipo de programação ajuda no desenvolvimento precoce das capacidades de programação e do pensamento computacional das crianças de uma forma interativa. Desta ideia surgiu o Tactode: um sistema de programação tangível composto por peças tipo puzzle e uma aplicação web que visa a programação de robôs. Os utilizadores alvo deste sistema são as crianças que, recorrendo às peças, formam um código tangível, tiram uma fotografia ao mesmo e depois podem carregá-lo para a aplicação para, posteriormente, ser testado e executado no robô. O projeto Tactode encontra-se desenvolvido com base em marcadores ArUco, isto é, cada peça contém um marcador deste tipo que facilita a sua deteção e distinção no código tangível. Posto isto, esta dissertação vai dar continuidade a este projeto através do desenvolvimento de um sistema de visão computacional capaz de detetar e identificar cada peça em fotografias de códigos Tactode, sem recorrer aos marcadores ArUco.Tangible programming, when applied to robotics, makes programming more understandable and straightforward. This type of programming helps children developing their abilities of programming and computational thinking interactively and at earlier stages of their lives. From this idea came Tactode: a tangible programming system composed by puzzle-like pieces and a web application that aims robot programming. The target users of this system are children who, using the pieces, build a tangible code, take a picture of it and then can upload it to the application to be tested and executed on the robot later. The Tactode project is developed based on ArUco markers, meaning that each piece have a marker of this type that facilitates its detection and distinction in the tangible code. Therefore, this dissertation will continue this project through the development of a computer vision system capable of detecting and identifying each piece in photographed Tactode codes without depending on the ArUco markers

Advances in top-down and bottom-up approaches to video-based camera tracking

Video-based camera tracking consists in trailing the three dimensional pose followed by a mobile camera using video as sole input. In order to estimate the pose of a camera with respect to a real scene, one or more three dimensional references are needed. Examples of such references are landmarks with known geometric shape, or objects for which a model is generated beforehand. By comparing what is seen by a camera with what is geometrically known from reality, it is possible to recover the pose of the camera that is sensing these references. In this thesis, we investigate the problem of camera tracking at two levels. Firstly, we work at the low level of feature point recognition. Feature points are used as references for tracking and we propose a method to robustly recognise them. More specifically, we introduce a rotation-discriminative region descriptor and an efficient rotation-discriminative method to match feature point descriptors. The descriptor is based on orientation gradient histograms and template intensity information. Secondly, we have worked at the higher level of camera tracking and propose a fusion of top-down (TDA) and bottom-up approaches (BUA). We combine marker-based tracking using a BUA and feature points recognised from a TDA into a particle filter. Feature points are recognised with the method described before. We take advantage of the identification of the rotation of points for tracking purposes. The goal of the fusion is to take advantage of their compensated strengths. In particular, we are interested in covering the main capabilities that a camera tracker should provide. These capabilities are automatic initialisation, automatic recovery after loss of track, and tracking beyond references known a priori. Experiments have been performed at the two levels of investigation. Firstly, tests have been conducted to evaluate the performance of the recognition method proposed. The assessment consists in a set of patches extracted from eight textured images. The images are rotated and matching is done for each patch. The results show that the method is capable of matching accurately despite the rotations. A comparison with similar techniques in the state of the art depicts the equal or even higher precision of our method with much lower computational cost. Secondly, experimental assessment of the tracking system is also conducted. The evaluation consists in four sequences with specific problematic situations namely, occlusions of the marker, illumination changes, and erratic and/or fast motion. Results show that the fusion tracker solves characteristic failure modes of the two combined approaches. A comparison with similar trackers shows competitive accuracy. In addition, the three capabilities stated earlier are fulfilled in our tracker, whereas the state of the art reveals that no other published tracker covers these three capabilities simultaneously. The camera tracking system has a potential application in the robotics domain. It has been successfully used as a man-machine interface and applied in Augmented Reality environments. In particular, the system has been used by students of the University of art and design Lausanne (ECAL) with the purpose of conceiving new interaction concepts. Moreover, in collaboration with ECAL and fabric | ch (studio for architecture & research), we have jointly developed the Augmented interactive Reality Toolkit (AiRToolkit). The system has also proved to be reliable in public events and is the basis of a game-oriented demonstrator installed in the Swiss National Museum of Audiovisual and Multimedia (Audiorama) in Montreux

Towards markerless orthopaedic navigation with intuitive Optical See-through Head-mounted displays

The potential of image-guided orthopaedic navigation to improve surgical outcomes has been well-recognised during the last two decades. According to the tracked pose of target bone, the anatomical information and preoperative plans are updated and displayed to surgeons, so that they can follow the guidance to reach the goal with higher accuracy, efficiency and reproducibility. Despite their success, current orthopaedic navigation systems have two main limitations: for target tracking, artificial markers have to be drilled into the bone and calibrated manually to the bone, which introduces the risk of additional harm to patients and increases operating complexity; for guidance visualisation, surgeons have to shift their attention from the patient to an external 2D monitor, which is disruptive and can be mentally stressful. Motivated by these limitations, this thesis explores the development of an intuitive, compact and reliable navigation system for orthopaedic surgery. To this end, conventional marker-based tracking is replaced by a novel markerless tracking algorithm, and the 2D display is replaced by a 3D holographic Optical see-through (OST) Head-mounted display (HMD) precisely calibrated to a user's perspective. Our markerless tracking, facilitated by a commercial RGBD camera, is achieved through deep learning-based bone segmentation followed by real-time pose registration. For robust segmentation, a new network is designed and efficiently augmented by a synthetic dataset. Our segmentation network outperforms the state-of-the-art regarding occlusion-robustness, device-agnostic behaviour, and target generalisability. For reliable pose registration, a novel Bounded Iterative Closest Point (BICP) workflow is proposed. The improved markerless tracking can achieve a clinically acceptable error of 0.95 deg and 2.17 mm according to a phantom test. OST displays allow ubiquitous enrichment of perceived real world with contextually blended virtual aids through semi-transparent glasses. They have been recognised as a suitable visual tool for surgical assistance, since they do not hinder the surgeon's natural eyesight and require no attention shift or perspective conversion. The OST calibration is crucial to ensure locational-coherent surgical guidance. Current calibration methods are either human error-prone or hardly applicable to commercial devices. To this end, we propose an offline camera-based calibration method that is highly accurate yet easy to implement in commercial products, and an online alignment-based refinement that is user-centric and robust against user error. The proposed methods are proven to be superior to other similar State-of- the-art (SOTA)s regarding calibration convenience and display accuracy. Motivated by the ambition to develop the world's first markerless OST navigation system, we integrated the developed markerless tracking and calibration scheme into a complete navigation workflow designed for femur drilling tasks during knee replacement surgery. We verify the usability of our designed OST system with an experienced orthopaedic surgeon by a cadaver study. Our test validates the potential of the proposed markerless navigation system for surgical assistance, although further improvement is required for clinical acceptance.Open Acces