Adaptive and intelligent navigation of autonomous planetary rovers - A survey

The application of robotics and autonomous systems in space has increased dramatically. The ongoing Mars rover mission involving the Curiosity rover, along with the success of its predecessors, is a key milestone that showcases the existing capabilities of robotic technology. Nevertheless, there has still been a heavy reliance on human tele-operators to drive these systems. Reducing the reliance on human experts for navigational tasks on Mars remains a major challenge due to the harsh and complex nature of the Martian terrains. The development of a truly autonomous rover system with the capability to be effectively navigated in such environments requires intelligent and adaptive methods fitting for a system with limited resources. This paper surveys a representative selection of work applicable to autonomous planetary rover navigation, discussing some ongoing challenges and promising future research directions from the perspectives of the authors

Adaptive Multi-sensor Perception for Driving Automation in Outdoor Contexts

In this research, adaptive perception for driving automation is discussed so as to enable a vehicle to automatically detect driveable areas and obstacles in the scene. It is especially designed for outdoor contexts where conventional perception systems that rely on a priori knowledge of the terrain's geometric properties, appearance properties, or both, is prone to fail, due to the variability in the terrain properties and environmental conditions. In contrast, the proposed framework uses a self-learning approach to build a model of the ground class that is continuously adjusted online to reflect the latest ground appearance. The system also features high flexibility, as it can work using a single sensor modality or a multi-sensor combination. In the context of this research, different embodiments have been demonstrated using range data coming from either a radar or a stereo camera, and adopting self-supervised strategies where monocular vision is automatically trained by radar or stereo vision. A comprehensive set of experimental results, obtained with different ground vehicles operating in the field, are presented to validate and assess the performance of the system

Semantic terrain segmentation in the navigation vision of planetary rovers – a systematic literature review

Background: The planetary rover is an essential platform for planetary exploration. Visual semantic segmentation is significant in the localization, perception, and path planning of the rover autonomy. Recent advances in computer vision and artificial intelligence brought about new opportunities. A systematic literature review (SLR) can help analyze existing solutions, discover available data, and identify potential gaps. Methods: A rigorous SLR has been conducted, and papers are selected from three databases (IEEE Xplore, Web of Science, and Scopus) from the start of records to May 2022. The 320 candidate studies were found by searching with keywords and bool operators, and they address the semantic terrain segmentation in the navigation vision of planetary rovers. Finally, after four rounds of screening, 30 papers were included with robust inclusion and exclusion criteria as well as quality assessment. Results: 30 studies were included for the review, and sub-research areas include navigation (16 studies), geological analysis (7 studies), exploration efficiency (10 studies), and others (3 studies) (overlaps exist). Five distributions are extendedly depicted (time, study type, geographical location, publisher, and experimental setting), which analyzes the included study from the view of community interests, development status, and reimplementation ability. One key research question and six sub-research questions are discussed to evaluate the current achievements and future gaps. Conclusions: Many promising achievements in accuracy, available data, and real-time performance have been promoted by computer vision and artificial intelligence. However, a solution that satisfies pixel-level segmentation, real-time inference time, and onboard hardware does not exist, and an open, pixel-level annotated, and the real-world data-based dataset is not found. As planetary exploration projects progress worldwide, more promising studies will be proposed, and deep learning will bring more opportunities and contributions to future studies. Contributions: This SLR identifies future gaps and challenges by proposing a methodical, replicable, and transparent survey, which is the first review (also the first SLR) for semantic terrain segmentation in the navigation vision of planetary rovers

AI Applications on Planetary Rovers

The rise in the number of robotic missions to space is paving the way for the use of artificial intelligence and machine learning in the autonomy and augmentation of rover operations. For one, more rovers mean more images, and more images mean more data bandwidth required for downlinking as well as more mental bandwidth for analyzing the images. On the other hand, light-weight, low-powered microrover platforms are being developed to accommodate the drive for planetary exploration. As a result of the mass and power constraints, these microrover platforms will not carry typical navigational instruments like a stereocamera or a laser rangerfinder, relying instead on a single, monocular camera. The first project in this thesis explores the realm of novelty detection where the goal is to find `new\u27 and `interesting\u27 features such that instead of sending a whole set of images, the algorithm could simply flag any image that contains novel features to prioritize its downlink. This form of data triage allows the science team to redirect its attention to objects that could be of high science value. For this project, a combination of a Convolutional Neural Network (CNN) with a K-means algorithm as a tool for novelty detection is introduced. By leveraging the powerful feature extraction capabilities of a CNN, typical images could be tightly clustered into the number of expected entities within the rover\u27s environment. The distance between the extracted feature vector and the closest cluster centroid is then defined to be its novelty score. As such, a novel image will have a significantly higher distance to the cluster centroids compared to the typical images. This algorithm was trained on images obtained from the Canadian Space Agency\u27s Analogue Terrain Facility and was shown to be effective in capturing the majority of the novel images within the dataset. The second project in this thesis aims to augment microrover platforms that are lacking the instruments for distance measurements. Particularly, this project explores the application of monocular depth estimation where the goal is to estimate a depth map from a monocular image. This problem is inherently difficult to solve given that recovering depth from a 2D image is a mathematically ill-posed problem, compounded by the fact that the lunar environment is a dull, colourless landscape. To solve his problem, a dataset of images and their corresponding ground truth depth maps have been taken at Mission Control Space Service\u27s Indoor Analogue Terrain. An autoencoder was then trained to take in the image and output an estimated depth map. The results of this project show that the model is not reliable at gauging the distances of slopes and objects near the horizon. However, the generated depth maps are reliable in the short to mid range, where the distances are most relevant for remote rover operations

Rock segmentation in the navigation vision of the planetary rovers

Visual navigation is an essential part of planetary rover autonomy. Rock segmentation emerged as an important interdisciplinary topic among image processing, robotics, and mathematical modeling. Rock segmentation is a challenging topic for rover autonomy because of the high computational consumption, real-time requirement, and annotation difficulty. This research proposes a rock segmentation framework and a rock segmentation network (NI-U-Net++) to aid with the visual navigation of rovers. The framework consists of two stages: the pre-training process and the transfer-training process. The pre-training process applies the synthetic algorithm to generate the synthetic images; then, it uses the generated images to pre-train NI-U-Net++. The synthetic algorithm increases the size of the image dataset and provides pixel-level masks—both of which are challenges with machine learning tasks. The pre-training process accomplishes the state-of-the-art compared with the related studies, which achieved an accuracy, intersection over union (IoU), Dice score, and root mean squared error (RMSE) of 99.41%, 0.8991, 0.9459, and 0.0775, respectively. The transfer-training process fine-tunes the pre-trained NI-U-Net++ using the real-life images, which achieved an accuracy, IoU, Dice score, and RMSE of 99.58%, 0.7476, 0.8556, and 0.0557, respectively. Finally, the transfer-trained NI-U-Net++ is integrated into a planetary rover navigation vision and achieves a real-time performance of 32.57 frames per second (or the inference time is 0.0307 s per frame). The framework only manually annotates about 8% (183 images) of the 2250 images in the navigation vision, which is a labor-saving solution for rock segmentation tasks. The proposed rock segmentation framework and NI-U-Net++ improve the performance of the state-of-the-art models. The synthetic algorithm improves the process of creating valid data for the challenge of rock segmentation. All source codes, datasets, and trained models of this research are openly available in Cranfield Online Research Data (CORD)

On Semantic Segmentation and Path Planning for Autonomous Vehicles within Off-Road Environments

There are many challenges involved in creating a fully autonomous vehicle capable of safely navigating through off-road environments. In this work we focus on two of the most prominent such challenges, namely scene understanding and path planning. Scene understanding is a challenging computer vision task with recent advances in convolutional neural networks (CNN) achieving results that notably surpass prior traditional feature driven approaches. Here, we build on recent work in urban road-scene understanding, training a state of the art CNN architecture towards the task of classifying off-road scenes. We analyse the effects of transfer learning and training data set size on CNN performance, evaluating multiple configurations of the network at multiple points during the training cycle, investigating in depth how the training process is affected. We compare this CNN to a more traditional feature-driven approach with Support Vector Machine (SVM) classifier and demonstrate state-of-the-art results in this particularly challenging problem of off-road scene understanding. We then expand on this with the addition of multi-channel RGBD data, which we encode in multiple configurations for CNN input. We evaluate each of these configuration over our own off-road RGBD data set and compare performance to that of the network model trained using RGB data. Next, we investigate end-to-end navigation, whereby a machine learning algorithm optimises to predict the vehicle control inputs of a human driver. After evaluating such a technique in an off-road environment and identifying several limitations, we propose a new approach in which a CNN learns to predict vehicle path visually, combining a novel approach to automatic training data creation with state of the art CNN architecture to map a predicted route directly onto image pixels. We then evaluate this approach using our off-road data set, and demonstrate effectiveness surpassing existing end-to-end methods

On Martian Surface Exploration: Development of Automated 3D Reconstruction and Super-Resolution Restoration Techniques for Mars Orbital Images

Very high spatial resolution imaging and topographic (3D) data play an important role in modern Mars science research and engineering applications. This work describes a set of image processing and machine learning methods to produce the “best possible” high-resolution and high-quality 3D and imaging products from existing Mars orbital imaging datasets. The research work is described in nine chapters of which seven are based on separate published journal papers. These include a) a hybrid photogrammetric processing chain that combines the advantages of different stereo matching algorithms to compute stereo disparity with optimal completeness, fine-scale details, and minimised matching artefacts; b) image and 3D co-registration methods that correct a target image and/or 3D data to a reference image and/or 3D data to achieve robust cross-instrument multi-resolution 3D and image co-alignment; c) a deep learning network and processing chain to estimate pixel-scale surface topography from single-view imagery that outperforms traditional photogrammetric methods in terms of product quality and processing speed; d) a deep learning-based single-image super-resolution restoration (SRR) method to enhance the quality and effective resolution of Mars orbital imagery; e) a subpixel-scale 3D processing system using a combination of photogrammetric 3D reconstruction, SRR, and photoclinometric 3D refinement; and f) an optimised subpixel-scale 3D processing system using coupled deep learning based single-view SRR and deep learning based 3D estimation to derive the best possible (in terms of visual quality, effective resolution, and accuracy) 3D products out of present epoch Mars orbital images. The resultant 3D imaging products from the above listed new developments are qualitatively and quantitatively evaluated either in comparison with products from the official NASA planetary data system (PDS) and/or ESA planetary science archive (PSA) releases, and/or in comparison with products generated with different open-source systems. Examples of the scientific application of these novel 3D imaging products are discussed

Cost-effective robot for steep slope crops monitoring

This project aims to develop a low cost, simple and robust robot able to autonomously monitorcrops using simple sensors. It will be required do develop robotic sub-systems and integrate them with pre-selected mechanical components, electrical interfaces and robot systems (localization, navigation and perception) using ROS, for wine making regions and maize fields