Multi-step Multi-camera View Planning for Real-Time Visual Object Tracking

Abstract. We present a new method for planning the optimal next view for a probabilistic visual object tracking task. Our method uses a variable number of cameras, can plan an action sequence several time steps into the future, and allows for real-time usage due to a computation time which is linear both in the number of cameras and the number of time steps. The algorithm can also handle object loss in one, more or all cameras, interdependencies in the camera’s information contribution, and variable action costs. We evaluate our method by comparing it to previous approaches with a prere-corded sequence of real world images. From K. Franke et al., Pattern Recognition, 28th DAGM Symposium, Springer, 2006, (pp. 536–545).

Active Exploration for Robust Object Detection

Today, mobile robots are increasingly expected to operate in ever more complex and dynamic environments. In order to carry out many of the higher-level tasks envisioned a semantic understanding of a workspace is pivotal. Here our field has benefited significantly from successes in machine learning and vision: applications in robotics of off-the-shelf object detectors are plentiful. This paper outlines an online, any-time planning framework enabling the active exploration of such detections. Our approach exploits the ability to move to different vantage points and implicitly weighs the benefits of gaining more certainty about the existence of an object against the physical cost of the exploration required. The result is a robot which plans trajectories specifically to decrease the entropy of putative detections. Our system is demonstrated to significantly improve detection performance and trajectory length in simulated and real robot experiments.National Science Foundation (U.S.) (IIS grant 0546467)United States. Office of Naval Research (MURI N1141207-236214

POMDP solving: what rewards do you really expect at execution?

Partially Observable Markov Decision Processes have gained an increasing interest in many research communities, due to sensible improvements of their optimization algorithms and of computers capabilities. Yet, most research focus on optimizing either average accumulated rewards (AI planning) or direct entropy (active perception), whereas none of them matches the rewards actually gathered at execution. Indeed, the first optimization criterion linearly averages over all belief states, so that it does not gain best information from different observations, while the second one totally discards rewards. Thus, motivated by simple demonstrative examples, we study an additive combination of these two criteria to get the best of reward gathering and information acquisition at execution. We then compare our criterion with classical ones, and highlight the need to consider new hybrid non-linear criteria, on a realistic multi-target recognition and tracking mission

Intelligent viewpoint selection for efficient CT to video registration in laparoscopic liver surgery

PURPOSE: Minimally invasive surgery offers advantages over open surgery due to a shorter recovery time, less pain and trauma for the patient. However, inherent challenges such as lack of tactile feedback and difficulty in controlling bleeding lower the percentage of suitable cases. Augmented reality can show a better visualisation of sub-surface structures and tumour locations by fusing pre-operative CT data with real-time laparoscopic video. Such augmented reality visualisation requires a fast and robust video to CT registration that minimises interruption to the surgical procedure. METHODS: We propose to use view planning for efficient rigid registration. Given the trocar position, a set of camera positions are sampled and scored based on the corresponding liver surface properties. We implement a simulation framework to validate the proof of concept using a segmented CT model from a human patient. Furthermore, we apply the proposed method on clinical data acquired during a human liver resection. RESULTS: The first experiment motivates the viewpoint scoring strategy and investigates reliable liver regions for accurate registrations in an intuitive visualisation. The second experiment shows wider basins of convergence for higher scoring viewpoints. The third experiment shows that a comparable registration performance can be achieved by at least two merged high scoring views and four low scoring views. Hence, the focus could change from the acquisition of a large liver surface to a small number of distinctive patches, thereby giving a more explicit protocol for surface reconstruction. We discuss the application of the proposed method on clinical data and show initial results. CONCLUSION: The proposed simulation framework shows promising results to motivate more research into a comprehensive view planning method for efficient registration in laparoscopic liver surgery

Optimisation de POMDP : quelles récompenses sont réellement attendues à l'exécution de la politique ?

Les Processus Décisionnels Markoviens Partiellement Observables sont actuellement un sujet d'intérêt dans la communauté scientifique grâce aux progrès observés dans des algorithmes de résolution et dans les capacités numériques de calcul. La plupart de ces algorithmes sont focalisés sur la résolution d'un critère de performance, qui a pour ambition de caractériser les politiques qui permettront de générer les séquences de récompenses le plus importantes possibles. Dans la planification en Intelligence Artificielle, l'attention est tournée vers un critère qui optimise une somme pondérée des récompenses, et, pour des applications en perception active d'autre part, le critère est souvent défini en termes de gain d'information (entropie de Shannon). Aucun de ces critères ne prend en compte les récompenses réellement acquises lors de l'exécution de la politique. En effet, le premier critère est une moyenne linéaire sur l'espace d'états de croyance, de sorte que l'agent ne tend pas à obtenir une meilleure information des différentes observations, alors que le second critère ne prend pas en compte les récompenses. Ainsi, motivés par des exemples démonstratifs, nous étudions deux combinaisons, additive et multiplicative, de ces critères afin d'obtenir une meilleur séquence de récompenses et de gain d'information lors de l'exécution de la politique. Nous comparons nos critères avec le critère classique optimisé (y-pondéré) dans le cadre POMDP et nous soulignons l'intérêt de considérer un nouveau critère hybride non-linéaire pour des applications réalistes de reconnaissance et pistage multi-cibles

Recognizing shipbuilding parts using artificial neural networks and Fourier descriptors

Abstract: A pattern recognition system is described for recognizing shipbuilding parts using artificial neural networks and Fourier descriptors. The system uses shape contour information that is invariant of size, translation, and rotation. Fourier descriptors provide information, and the neural networks make decisions about the shapes. A brief review of the current state of the art is included, and results from testing show that the system distinguished between various shapes and proved to be a valid approach

Automatic Inspection of Aeronautical Mechanical Assemblies by Matching the 3D CAD Model and Real 2D Images

International audienceIn the aviation industry, automated inspection is essential for ensuring quality of production. It allows acceleration of procedures for quality control of parts or mechanical assemblies. As a result, the demand of intelligent visual inspection systems aimed at ensuring high quality in production lines is increasing. In this work, we address a very common problem in quality control. The problem is verification of presence of the correct part and verification of its position. We address the problem in two parts: first, automatic selection of informative viewpoints before the inspection process is started (offline preparation of the inspection) and, second, automatic treatment of the acquired images from said viewpoints by matching them with information in 3D CAD models is launched. We apply this inspection system for detecting defects on aeronautical mechanical assemblies with the aim of checking whether all the subparts are present and correctly mounted. The system can be used during manufacturing or maintenance operations. The accuracy of the system is evaluated on two kinds of platform. One is an autonomous navigation robot, and the other one is a handheld tablet. The experimental results show that our proposed approach is accurate and promising for industrial applications with possibility for real-time inspection

Décision séquentielle pour la perception active : p-POMDP versus POMDP

Cet article propose une étude du compromis entre la prise d’information et la décision dans un cadre applicatif qui se rapporte à une mission d’exploration, où l’agent interagit avec son environnement pour identifier l’état caché du système. Dans ce problème de décision séquentielle pour la perception, il est possible de faire reposer la fonction de récompense sur une mesure de l’incertitude sur l’état de croyance de l’agent (Araya-López et al., 2010; Candido & Hutchinson, 2011; Eidenberger & Scharinger, 2010). Sa forme est donc différente de celle utilisée dans le cadre classique des POMDP qui est, pour sa part, basée sur la paire état-action. Nous comparons donc deux approches d’optimisation des politiques pour ce type de problème. D’une part nous proposons un critère mixte qui couple une mesure de l’incertitude sur l’état de croyance avec les récompenses définies par les paires état-action et nous développons un schéma algorithmique de résolution pour ce critère. D’autre part, nous proposons d’ajouter au modèle des états but fictifs au moyen des actions de classification afin de revenir à une modélisation sous-forme de POMDP classique (critère non mixte). Une étude comparative de ces approches est ici présentée afin de vérifier leur équivalence en termes de prise d’informations. Les résultats nous mènent à conclure que ces approches sont non seulement comparables et équivalentes en termes de réduction d’incertitude, mais aussi, qu’elles peuvent être utilisées en parfaite complémentarité de façon à permettre : de caractériser une politique correspondant aux taux acceptables des bonnes et mauvaises classifications et de déterminer les bonnes valeurs des coûts et des récompenses du modèle POMDP classique

Active Object Classification from 3D Range Data with Mobile Robots

This thesis addresses the problem of how to improve the acquisition of 3D range data with a mobile robot for the task of object classification. Establishing the identities of objects in unknown environments is fundamental for robotic systems and helps enable many abilities such as grasping, manipulation, or semantic mapping. Objects are recognised by data obtained from sensor observations, however, data is highly dependent on viewpoint; the variation in position and orientation of the sensor relative to an object can result in large variation in the perception quality. Additionally, cluttered environments present a further challenge because key data may be missing. These issues are not always solved by traditional passive systems where data are collected from a fixed navigation process then fed into a perception pipeline. This thesis considers an active approach to data collection by deciding where is most appropriate to make observations for the perception task. The core contributions of this thesis are a non-myopic planning strategy to collect data efficiently under resource constraints, and supporting viewpoint prediction and evaluation methods for object classification. Our approach to planning uses Monte Carlo methods coupled with a classifier based on non-parametric Bayesian regression. We present a novel anytime and non-myopic planning algorithm, Monte Carlo active perception, that extends Monte Carlo tree search to partially observable environments and the active perception problem. This is combined with a particle-based estimation process and a learned observation likelihood model that uses Gaussian process regression. To support planning, we present 3D point cloud prediction algorithms and utility functions that measure the quality of viewpoints by their discriminatory ability and effectiveness under occlusion. The utility of viewpoints is quantified by information-theoretic metrics, such as mutual information, and an alternative utility function that exploits learned data is developed for special cases. The algorithms in this thesis are demonstrated in a variety of scenarios. We extensively test our online planning and classification methods in simulation as well as with indoor and outdoor datasets. Furthermore, we perform hardware experiments with different mobile platforms equipped with different types of sensors. Most significantly, our hardware experiments with an outdoor robot are to our knowledge the first demonstrations of online active perception in a real outdoor environment. Active perception has broad significance in many applications. This thesis emphasises the advantages of an active approach to object classification and presents its assimilation with a wide range of robotic systems, sensors, and perception algorithms. By demonstration of performance enhancements and diversity, our hope is that the concept of considering perception and planning in an integrated manner will be of benefit in improving current systems that rely on passive data collection