Challenges of Pose Estimation for Future OOS Missions

With an increase of launching more new satellites into space in the last decade, there is also a need for a more careful monitoring and maintenance of space environment. Space debris present a very critical problem since more often the evasive maneuvers are needed by operational satellites and other spacecrafts in order to avoid the collision. At German Aerospace Center we do research in the field of On-Orbit Servicing (OOS) and Active Debris Removal (ADR). In these terms, different activities can be planned for these missions, e.g. repairing, modernization or lifetime extension of a satellite. In case of no more functioning satellite or remains of rocket body, complete deorbiting of that objects is considered. Any of these activities needs a safe approach of the servicer spacecraft towards the space object. In the close-range approaches, the autonomous systems rely on visual navigation with cameras and/or lidars for estimation of position and orientation of a non-cooperative target. Developing of reliable flight-ready pose estimation techniques for such systems is a challenging task. Plenty of external factors and conditions affect the performance of the pose estimation algorithms. In this talk I am going to discuss difficulties and challenges appeared during development and testing pose estimation technique

Selecting Key Features of 3D Object Model for Relative Pose Estimation

This paper addresses the discussion of the complexity and level of detail of the 3D model of a target satellite that is required in order to get a stable and accurate relative pose estimation with monocular cameras during an On-Orbit Servicing (OOS) mission scenario. We assume to know the target 3D mesh with thousands of vertices before the mission takes place. Nevertheless, for on-board pose estimation we have to use compact 3D models of the targets with only some key points. In this paper we compare the pose estimation results using three models for tests. The first model has manually extracted key features, where the second one includes only features extracted with Harris3D technique. The third model includes a part of Harris3D key features and some manually selected points. The offline pose estimation tests were using the data from European Proximity Operations Simulator (EPOS) facility

Robust feature extraction pose estimation during fly-around and straight-line approach in close range.

This paper refers to the field of visual navigation in OnOrbit Servicing (OOS) and/or Active Debris Removal (ADR) missions. Mainly the robust feature extraction pose estimation technique is proposed here to estimate target while approaching it. This method is tested with two image datasets from different sensors in open loop. The stable tracking during the fly-around and straight line approach gives a positive feedback to consider this technique as a possible candidate for the future missions

Analysis and optimization of PMD-sensor data for rendezvous applications in space

In the last decade, people became more aware of the critical situation in space concerning the space debris. Nowadays, there are plenty of uncontrolled objects located on different orbits in the outer space. Some of them present the real hazards to the functioning satellites and to the International Space Station. The researches all over the world are working on optimal solutions for the space cleaning. There are projects which are directed to repair the satellites and extend their operational lifetime, or totally remove a no more useful space object from its orbit. Rendezvous is the one inevitable space operation all these tasks need. We are able to execute a space operation with a target satellite only if the servicer satellite approaches it. The goal of this thesis is to develop a visual navigation system with the Photonic Mixer Device (PMD) sensor for the close rendezvous phase with a non-cooperative target. The PMD sensor has never been used in space missions so far, but it has already been tested on the ground. In order to achieve the goal, this thesis provides two different pipelines for the pose acquisition and for the pose tracking using the images from the PMD sensor. In this work we show that the use of the PMD's amplitude and depth images together brings a great contribution to the visual navigation system. The pose acquisition is required in order to initialize a pose (position and orientation) of the non-cooperative space object before the tracking takes place. This task is very diffcult, because we have to deal only with the PMD images and the known 3D model of the target. We initialize the pose with the depth image and thereafter improve the obtained pose with the amplitude image. As soon as we have an initial estimation, the servicer starts to approach the target in a frame-to-frame mode. The pose for every frame is estimated by a fusion of the states, which are calculated with the developed techniques for the depth and the amplitude images correspondingly. This technique shows a stable tracking with low errors of the estimated pose even if there are some distortions in the depth image. This fact is very important for the close rendezvous phase, because any failures in Guidance, Navigation and Control (GNC) system can lead to an unpredictable behavior of the chaser, and in end effect, create more space debris. The techniques presented in this thesis are tested with real images of the PMD sensor. The rendezvous simulations are executed and evaluated with a high accuracy hardware-inthe- loop simulator. The tests of the visual navigation with a PMD sensor in a closed loop show promising results. The servicer satellite can smoothly approach the non-cooperative target by only using the PMD sensor for relative navigation

Initial Pose Estimation using PMD Sensor during the Rendezvous Phase in On-Orbit Servicing Missions

This paper describes a designed visual model-based algorithm using the PMD (Photonic Mixer Device) sensor for the initial pose estimation of the target in future On-Orbit Servicing missions. The initial relative pose (position and orientation) in a close range has to be estimated, starting less than 7 meters between target and a camera. The verification of the algorithm is conducted by comparing the estimated pose with a ground truth. The ground truth is derived from the high-accuracy hardware-in-the-loop European Proximity Operations Simulator offered for the simulations of On-Orbit Servicing scenarios on the ground. The results of the simulations have shown the feasibility of the algorithm to estimate the pose with sufficient accuracy as required for a pose initialization algorithm. Consequently, the designed algorithm is applicable for the initial pose estimation using PMD sensor with definite working parameters and conditions

Visual Navigation for Rendezvous and Docking using PMD Camera

There is one common problem of satellites which quite widely discussed: the lifespan. An idea to extend operational lifespan of a satellite in orbit, instead of replacing it by a new one, is concluded in On-Orbit Servicing (OOS) projects. The typical scenario of OOS is safe and reliable Rendezvous and Docking (RvD) of the approaching chaser to the target and taking over attitude control. In this paper relative navigation using new product Photonic Mixer Device (PMD), as new sort of the 3D Time-of-Flight (ToF) sensor, is suggested and of great interest, because the PMD sensor has never been used in space environment before. For optical navigation based on using a PMD camera, the following tasks and challenges have to be considered: initial pose estimation with a subsequent real-time object tracking, extension of the measurement range of the PMD sensor, camera calibration, performance measurements with a RvD simulation facility like EPOS 2.0 (European Proximity Operations Simulator). This paper mainly focuses on the first objective, namely pose estimation with a PMD sensor. Before calculating the pose estimate, the problem of actually finding target must be solved. The model based matching algorithm is proposed in this paper for 3D target acquisition and estimation of 6 degrees of freedom (3D position and 3D orientation) using data measurements from PMD camera

More Accurate Pose Initialization with Redundant Measurements

The problem described in this paper concerns the problem of initial pose estimation of a non-cooperative target for space applications. We propose to use a Photonic Mixer Device (PMD) sensor in a close range for the visual navigation in order to estimate position and attitude of the space object. The advantage of the ranging PMD sensor is that it provides two different sources of data: depth and amplitude information of the imaging scene. In this work we make use of it and propose a follow-up initial pose improvement technique with the amplitude images from PMD sensor. It means that we primary calculate the pose of the target with the depth image and then correct the pose to get more accurate result. The algorithm is tested for the set of images in the range 8 to 4.9 meters. The obtained results have shown the evident improvement of the initial pose after correction with the proposed technique

PMD Camera- and Hand-Eye-Calibration for On-Orbit Servicing Test Scenarios on the Ground

In this paper we present the ability to intrinsically and extrinsically calibrate a Time-of-Flight sensor, namely, a Photonic Mixer Device (PMD) camera, using the DLR CalDe and DLR CalLab camera calibration toolbox. This camera is intended as a visual sensor for pose estimation in the close rendezvous phase during future On-Orbit servicing. In order to test and verify the pose estimation algorithms on the ground, we conduct different rendezvous scenarios using the European Proximity Operation Simulator. It is necessary to accurately know intrinsic parameters like the focal length, the principal point, and the distortion parameters, as well as the extrinsic parameters, i.e., the position and orientation of the PMD camera relating to the mounting board, whenever it is fixed on the robot and involved in the process of target pose estimation. In this work we differentiate from state-of-the-art approaches for the calibration of PMD cameras in this context by making use of the motion of the mounting robotic manipulator alone, i.e., without the need for accurate positioning of the target calibration plate by a second robotic manipulator

Hardware-in-the-Loop Simulations with Umbra Conditions for Spacecraft Rendezvous with PMD Visual Sensors

This paper addresses the validation of a robust vision-based pose estimation technique using a Photonic Mixer Device (PMD) sensor as a single visual sensor in the close-range phase of spacecraft rendezvous. First, it was necessary to integrate the developed hybrid navigation technique for the PMD sensor into the hardware-in-the-loop (HIL) rendezvous system developed by the German Aerospace Center (DLR). Thereafter, HIL tests were conducted using the European Proximity Operation Simulator (EPOS) with sun simulation and in total darkness. For the future missions with an active sensor, e.g., a PMD camera, it could be useful to use only its own illumination during the rendezvous phase in penumbra or umbra, instead of additional flash light. In some tests, the rotational rate of the target object was also tuned. Unlike the rendezvous tests in other works, here we present for the first time closed-loop approaches with only depth and amplitude images of a PMD sensor. For the rendezvous tests in the EPOS laboratory, the Argos3D camera was used at the range of 8 to 5.5 meters; the performance showed promising results

Close Range Tracking of an Uncooperative Target in a Sequence of Photonic Mixer Device (PMD) Images

This paper presents a pose estimation routine for tracking attitude and position of an uncooperative tumbling spacecraft during close range rendezvous. The key innovation is the usage of a Photonic Mixer Device (PMD) sensor for the first time during space proximity for tracking the pose of the uncooperative target. This sensor requires lower power consumption and higher resolution if compared with existing flash Light Identification Detection and Ranging (LiDAR) sensors. In addition, the PMD sensor provides two different measurements at the same time: depth information (point cloud) and amplitude of the reflected signal, which generates a grayscale image. In this paper, a hybrid model-based navigation technique that employs both measurements is proposed. The principal pose estimation technique is the iterative closed point algorithm with reverse calibration, which relies on the depth image. The second technique is an image processing pipeline that generates a set of 2D-to-3D feature correspondences between amplitude image and spacecraft model followed by the Efficient Perspective-n-Points (EPnP) algorithm for pose estimation. In this way, we gain a redundant estimation of the target’s current state in real-time without hardware redundancy. The proposed navigation methodology is tested in the German Aerospace Center (DLR)’s European Proximity Operations Simulator. The hybrid navigation technique shows the capability to ensure robust pose estimation of an uncooperative tumbling target under severe illumination conditions. In fact, the EPnP-based technique allows to overcome the limitations of the primary technique when harsh illumination conditions arise