RCDN -- Robust X-Corner Detection Algorithm based on Advanced CNN Model

Accurate detection and localization of X-corner on both planar and non-planar patterns is a core step in robotics and machine vision. However, previous works could not make a good balance between accuracy and robustness, which are both crucial criteria to evaluate the detectors performance. To address this problem, in this paper we present a novel detection algorithm which can maintain high sub-pixel precision on inputs under multiple interference, such as lens distortion, extreme poses and noise. The whole algorithm, adopting a coarse-to-fine strategy, contains a X-corner detection network and three post-processing techniques to distinguish the correct corner candidates, as well as a mixed sub-pixel refinement technique and an improved region growth strategy to recover the checkerboard pattern partially visible or occluded automatically. Evaluations on real and synthetic images indicate that the presented algorithm has the higher detection rate, sub-pixel accuracy and robustness than other commonly used methods. Finally, experiments of camera calibration and pose estimation verify it can also get smaller re-projection error in quantitative comparisons to the state-of-the-art.Comment: 15 pages, 8 figures and 4 tables. Unpublished further research and experiments of Checkerboard corner detection network CCDN (arXiv:2302.05097) and application exploration for robust camera calibration (https://ieeexplore.ieee.org/abstract/document/9428389

Neural Lens Modeling

Recent methods for 3D reconstruction and rendering increasingly benefit from end-to-end optimization of the entire image formation process. However, this approach is currently limited: effects of the optical hardware stack and in particular lenses are hard to model in a unified way. This limits the quality that can be achieved for camera calibration and the fidelity of the results of 3D reconstruction. In this paper, we propose NeuroLens, a neural lens model for distortion and vignetting that can be used for point projection and ray casting and can be optimized through both operations. This means that it can (optionally) be used to perform pre-capture calibration using classical calibration targets, and can later be used to perform calibration or refinement during 3D reconstruction, e.g., while optimizing a radiance field. To evaluate the performance of our proposed model, we create a comprehensive dataset assembled from the Lensfun database with a multitude of lenses. Using this and other real-world datasets, we show that the quality of our proposed lens model outperforms standard packages as well as recent approaches while being much easier to use and extend. The model generalizes across many lens types and is trivial to integrate into existing 3D reconstruction and rendering systems.Comment: To be presented at CVPR 2023, Project webpage: https://neural-lens.github.i

Saliency-based cooperative landing of a multirotor aerial vehicle on an autonomous surface vehicle

This paper presents a method for vision-based landing of a multirotor unmanned aerial vehicle (UAV) on an autonomous surface vehicle (ASV) equipped with a helipad. The method includes a mechanism for helipad behavioural search when outside the UAV’s field of view, a learning saliency-based mechanism for visual tracking the helipad, and a cooperative strategy for the final vision-based landing phase. Learning how to track the helipad from above occurs during takeoff and cooperation results from having the ASV tracking the UAV for assisting its landing. A set of experimental results with both simulated and physical robots show the feasibility of the presented method.info:eu-repo/semantics/acceptedVersio

REDI 4.0 - Robot for Demonstrations with Behaviour Defined on Paper

This project proposes to create a robot whose behaviour is defined on paper, focusing on making its programming as independent and self-sufficient as possible so it can be brought to schools for the younger generations to learn how to program in a new and fun way. Its first rendition was developed in 2008 as a standalone robot programmed by connecting pegs with electrical wires representing different commands. As that solution was both expensive and limiting in its use as each student had to wait their turn, this new iteration has the intention of replacing those connections with drawn-on paper instructions. This way, the associated costs are reduced and more users can define their programming strategy without having to wait for the robot to become available

Prior knowledge contribution to declarative learning. A study in amnesia, aging and Alzheimer's disease

L'étude expérimentale de la mémoire humaine a connu deux moments historiques dans les soixante dernières années. 1957 marque la découverte du rôle du lobe temporal interne bilatéral dans l'apprentissage conscient, déclaratif. 1997 marque la découverte de deux systèmes de mémoire déclarative, épisodique et sémantique. Ces découvertes résultent d'études de cas en neuropsychologie. Cette thèse s'inscrit dans la tradition neuropsychologique: sa genèse doit tout à un patient souffrant d'une forme atypique d'amnésie développementale, le patient KA. Son point de départ est une étude de cas approfondie, avec deux résultats surprenants. Malgré une amnésie sévère, KA dispose de connaissances sémantiques exceptionnelles. Par ailleurs, il montre des capacités préservées d'apprentissage explicite, mais uniquement pour des stimuli concrets, pas abstraits. En conséquence, cette thèse a exploré deux pistes de recherche. Premièrement, nous avons caractérisé les processus préservés d'apprentissage déclaratif et l'anatomie cérébrale chez ce patient. Deuxièmement, nous avons étudié le rôle des connaissances préalables dans l'apprentissage: comment ce que l'on sait influence ce dont nous nous souvenons ? Une première série d'expériences montre chez ce patient une atteinte sévère et sélective de l'ensemble du système hippocampique, alors que les structures sous- hippocampiques (cortex entorhinal, périrhinal et parahippocampique) sont préservées. Malgré une amnésie épisodique sévère, nous montrons des connaissances sémantiques supranormales et des aptitudes d'apprentissage explicite rapide. Ces aptitudes sont toutefois restreintes aux stimuli associés à des connaissances préalables. Une seconde série d'expériences explore l'hypothèse selon laquelle les connaissances préalables facilitent l'apprentissage en mémoire déclarative, même dans les situations où le lobe temporal interne est fragilisé, comme dans le vieillissement, ou lésé, comme chez le patient KA ou dans la maladie d'Alzheimer. Nos résultats suggèrent l'existence de processus d'apprentissage rapide en mémoire déclarative, indépendants du système hippocampique et sensibles à la présence de représentations préexistantes. Ces processus semblent affectés par la maladie d'Alzheimer, et ce en lien avec un défaut d'activité des régions sous-hippocampiques antérieures. A l'inverse, les sujets âgés sains peuvent utiliser les connaissances préalables et pourraient ainsi compenser le déclin de la mémoire associative. Ce travail s'accorde avec les modèles postulant une dissociation fonctionnelle au sein du lobe temporal interne pour l'apprentissage déclaratif. Il soutient les propositions neurocognitives et computationnelles récentes, suggérant une voie d'apprentissage néocortical rapide mobilisable dans certaines circonstances. Il met en exergue la dynamique des apprentissages en mémoire déclarative et notamment l'intrication fondamentale entre "savoir" et "se souvenir". Ce que je sais a un impact profond sur ce dont je vais me souvenir. Cette thèse permet d'envisager de nouveaux outils cognitifs pour le diagnostic de la maladie d'Alzheimer. De plus, il semble que des lésions temporales internes auront un impact distinct sur l'apprentissage selon le statut des informations à mémoriser en mémoire à long terme, offrant un regard nouveau sur les effets stimulus-dépendants dans l'amnésie. Une considération approfondie des connaissances préalables associées au contenu de nos expériences, et leur caractérisation détaillée, est requise pour affiner les modèles de la mémoire déclarative. Ces résultats apportent de nouvelles pistes de recherche quant aux circonstances épargnant l'apprentissage, notamment associatif, lors du vieillissement. Plus généralement, ils contribuent à la compréhension des déterminants d'un apprentissage réussi, en mettant l'accent sur les recouvrements entre processus de récupération et d'acquisition. Des applications potentielles en découlent dans le domaine éducatif.The experimental study of human memory has had two historic moments in the last sixty years. 1957 marks the discovery of the role of the medial temporal lobes in conscious learning. 1997 marks the discovery of two systems of declarative memory, namely episodic and semantic memories. These major breakthroughs are owed to clinical case studies in neuropsychology. This thesis follows on from the neuropsychological tradition: its genesis owes everything to a patient suffering from an atypical form of developmental amnesia, the patient KA. The starting point of this work was a thorough neuropsychological study of this patient. Two striking findings shortly arose. First, despite lifelong amnesia, KA had acquired exceptional levels of knowledge about the world. Second, remaining explicit learning abilities were restricted to meaningful, not meaningless, memoranda. As a consequence, we have investigated two research pathways in that thesis. First, we aimed at better characterizing preserved learning abilities and brain structure of the patient KA. Second, our goal was to explore how prior knowledge affects new declarative learning or, put simply, how do we learn what we know? In a first series of behavioural and neuroimaging experiments, we have shown in this patient a severe and selective damage of the whole extended hippocampal system, but preserved subhippocampal structures (entorhinal, perirhinal and parahippocampal cortex). The patient suffers from severe episodic amnesia, but we bring striking evidence for supranormal semantic knowledge as well as normal explicit learning skills. These skills were, however, restricted to familiar stimuli, that is, stimuli carrying pre-experimental knowledge. In a second series of behavioural and neuroimaging experiments, we explored the hypothesis that prior knowledge can facilitate new learning in declarative memory, even in aging or in situations where structures of the medial temporal lobe are or injured, as in amnesia or Alzheimer's disease. Our results suggest the existence of processes allowing fast learning in declarative memory, independently of the hippocampal system, and that are sensitive to the presence of pre-existing representations in long-term memory. Such learning processes appear to be selectively affected by Alzheimer's disease at the pre-dementia stage, in relation to a lack of activation of subhippocampal regions. In contrast, healthy elderly were able to rely on these learning processes to compensate for the decline in associative memory associated with aging. This work lends support to the models postulating a functional dissociation with respect to learning in declarative memory. It indeed strengthens recent neurocognitive and computational accounts that suggest a rapid neocortical learning path under certain circumstances. It highlights the dynamics of learning in declarative memory and in particular the fundamental entanglement between "knowing" and "remembering". What I know profoundly impacts what I will remember. The present thesis points towards new cognitive tools for the diagnosis of Alzheimer's disease. It further brings evidence that medial temporal lesions differentially impact learning depending on the status of the memoranda in long-term memory, which sheds a new light on material-specific effects in amnesia. Our work speaks for a thorough consideration of whether the contents of events have prior representations within long-term memory, and to further better characterize their nature if we are to better understand learning mechanisms. It also brings additional clues for a deeper understanding of how learning and memory can be preserved in aging. More generally, it contributes to a better understanding of the factors determining successful learning, with a focus on how retrieval and acquisition processes overlap during learning. Such findings have potential applications in the educational field

Advanced Calibration of Automotive Augmented Reality Head-Up Displays = Erweiterte Kalibrierung von Automotiven Augmented Reality-Head-Up-Displays

In dieser Arbeit werden fortschrittliche Kalibrierungsmethoden für Augmented-Reality-Head-up-Displays (AR-HUDs) in Kraftfahrzeugen vorgestellt, die auf parametrischen perspektivischen Projektionen und nichtparametrischen Verzerrungsmodellen basieren. Die AR-HUD-Kalibrierung ist wichtig, um virtuelle Objekte in relevanten Anwendungen wie z.B. Navigationssystemen oder Parkvorgängen korrekt zu platzieren. Obwohl es im Stand der Technik einige nützliche Ansätze für dieses Problem gibt, verfolgt diese Dissertation das Ziel, fortschrittlichere und dennoch weniger komplizierte Ansätze zu entwickeln. Als Voraussetzung für die Kalibrierung haben wir mehrere relevante Koordinatensysteme definiert, darunter die dreidimensionale (3D) Welt, den Ansichtspunkt-Raum, den HUD-Sichtfeld-Raum (HUD-FOV) und den zweidimensionalen (2D) virtuellen Bildraum. Wir beschreiben die Projektion der Bilder von einem AR-HUD-Projektor in Richtung der Augen des Fahrers als ein ansichtsabhängiges Lochkameramodell, das aus intrinsischen und extrinsischen Matrizen besteht. Unter dieser Annahme schätzen wir zunächst die intrinsische Matrix unter Verwendung der Grenzen des HUD-Sichtbereichs. Als nächstes kalibrieren wir die extrinsischen Matrizen an verschiedenen Blickpunkten innerhalb einer ausgewählten "Eyebox" unter Berücksichtigung der sich ändernden Augenpositionen des Fahrers. Die 3D-Positionen dieser Blickpunkte werden von einer Fahrerkamera verfolgt. Für jeden einzelnen Blickpunkt erhalten wir eine Gruppe von 2D-3D-Korrespondenzen zwischen einer Menge Punkten im virtuellen Bildraum und ihren übereinstimmenden Kontrollpunkten vor der Windschutzscheibe. Sobald diese Korrespondenzen verfügbar sind, berechnen wir die extrinsische Matrix am entsprechenden Betrachtungspunkt. Durch Vergleichen der neu projizierten und realen Pixelpositionen dieser virtuellen Punkte erhalten wir eine 2D-Verteilung von Bias-Vektoren, mit denen wir Warping-Karten rekonstruieren, welche die Informationen über die Bildverzerrung enthalten. Für die Vollständigkeit wiederholen wir die obigen extrinsischen Kalibrierungsverfahren an allen ausgewählten Betrachtungspunkten. Mit den kalibrierten extrinsischen Parametern stellen wir die Betrachtungspunkte wieder her im Weltkoordinatensystem. Da wir diese Punkte gleichzeitig im Raum der Fahrerkamera verfolgen, kalibrieren wir weiter die Transformation von der Fahrerkamera in den Weltraum unter Verwendung dieser 3D-3D-Korrespondenzen. Um mit nicht teilnehmenden Betrachtungspunkten innerhalb der Eyebox umzugehen, erhalten wir ihre extrinsischen Parameter und Warping-Karten durch nichtparametrische Interpolationen. Unsere Kombination aus parametrischen und nichtparametrischen Modellen übertrifft den Stand der Technik hinsichtlich der Zielkomplexität sowie Zeiteffizienz, während wir eine vergleichbare Kalibrierungsgenauigkeit beibehalten. Bei allen unseren Kalibrierungsschemen liegen die Projektionsfehler in der Auswertungsphase bei einer Entfernung von 7,5 Metern innerhalb weniger Millimeter, was einer Winkelgenauigkeit von ca. 2 Bogenminuten entspricht, was nahe am Auflösungvermögen des Auges liegt

On the detection of defects on specular car body surfaces

[EN] The automatic detection of small defects (of up to 0.2 mm in diameter) on car body surfaces following the painting process is currently one of the greatest issues facing quality control in the automotive industry. Although several systems have been developed during the last decade to provide a solution to this problem, these, to the best of our knowledge, have been focused solely on flat surfaces and have been unable to inspect other parts of the surfaces, namely style lines, edges and corners as well as deep concavities. This paper introduces a novel approach using deflectometry- and vision-based technologies in order to overcome this problem and ensure that the whole area is inspected. Moreover, since our approach, together with the system used, computes defects in less than 15 s, it satisfies cycle time production requirements (usually of around 30 s per car). Hence, a two-step algorithm is presented here: in the first step, a new pre-processing step (image fusion algorithm) is introduced to enhance the contrast between pixels with a low level of intensity (indicating the presence of defects) and those with a high level of intensity (indicating the absence of defects); for the second step, we present a novel post-processing step with an image background extraction approach based on a local directional blurring method and a modified image contrast enhancement, which enables detection of defects in the entire illuminated area. In addition, the post-processing step is processed several times using a multi-level structure, with computed image backgrounds of different resolution. In doing so, it is possible to detect larger defects, given that each level identifies defects of different sizes. Experimental results presented in this paper are obtained from the industrial automatic quality control system QEyeTunnel employed in the production line at the Mercedes-Benz factory in Vitoria, Spain. A complete analysis of the algorithm performance will be shown here, together with several tests proving the robustness and reliability of our proposal.This work is supported by VALi+d (APOSTD/2016/044) and PROMETEO (PROMETEOII/2014/044) Programs, both from Conselleria d'Educacio, Generalitat Valenciana.Molina, J.; Solanes Galbis, JE.; Arnal-Benedicto, L.; Tornero Montserrat, J. (2017). On the detection of defects on specular car body surfaces. Robotics and Computer-Integrated Manufacturing. 48:263-278. https://doi.org/10.1016/j.rcim.2017.04.009S2632784