Cross-spectral local descriptors via quadruplet network

This paper presents a novel CNN-based architecture, referred to as Q-Net, to learn local feature descriptors that are useful for matching image patches from two different spectral bands. Given correctly matched and non-matching cross-spectral image pairs, a quadruplet network is trained to map input image patches to a common Euclidean space, regardless of the input spectral band. Our approach is inspired by the recent success of triplet networks in the visible spectrum, but adapted for cross-spectral scenarios, where, for each matching pair, there are always two possible non-matching patches: one for each spectrum. Experimental evaluations on a public cross-spectral VIS-NIR dataset shows that the proposed approach improves the state-of-the-art. Moreover, the proposed technique can also be used in mono-spectral settings, obtaining a similar performance to triplet network descriptors, but requiring less training data

Robust multispectral image-based localisation solutions for autonomous systems

With the recent increase of interest in multispectral imaging, new image-based localisation solutions have emerged. However, its application to visual odometry remains overlooked. Most localisation techniques are still being developed with visible cameras only, because the portability they can offer and the wide variety of cameras available. Yet, other modalities have great potentials for navigation purposes. Infrared imaging for example, provides different information about the scene and is already used to enhance visible images. This is especially the case of far-infrared cameras which can produce images at night and see hot objects like other cars, animals or pedestrians. Therefore, the aim of this thesis is to tackle the lack of research in multispectral localisation and to explore new ways of performing visual odometry accurately with visible and thermal images. First, a new calibration pattern made of LED lights is presented in Chapter 3. Emitting both visible and thermal radiations, it can easily be seen by infrared and visible cameras. Due to its peculiar shape, the whole pattern can be moved around the cameras and automatically identified in the different images recorded. Monocular and stereo calibration are then performed to precisely estimate the camera parameters. Then, a multispectral monocular visual odometry algorithm is proposed in Chapter 4. This generic technique is able to operate in infrared and visible modalities, regardless of the nature of the images. Incoming images are processed at a high frame rate based on a 2D-to-2D unscaled motion estimation method. However, specific keyframes are carefully selected to avoid degenerate cases and a bundle adjustment optimisation is performed on a sliding window to refine the initial estimation. The advantage of visible-thermal odometry is shown on a scenario with extreme illumination conditions, where the limitation of each modality is reached. The simultaneous combination of visible and thermal images for visual odometry is also explored. In Chapter 5, two feature matching techniques are presented and tested in a multispectral stereo visual odometry framework. One method matches features between stereo pairs independently while the other estimates unscaled motion first, before matching the features altogether. Even though these techniques require more processing power to overcome the dissimilarities between V multimodal images, they have the benefit of estimating scaled transformations. Finally, the camera pose estimates obtained with multispectral stereo odometry are fused with inertial data to create a robustified localisation solution which is detailed in Chapter 6. The full state of the system is estimated, including position, velocity, orientation and IMU biases. It is shown that multispectral visual odometry can correct drifting IMU measurements effectively. Furthermore, it is demonstrated that such multi-sensors setups can be beneficial in challenging situations where features cannot be extracted or tracked. In that case, inertial data can be integrated to provide a state estimate while visual odometry cannot

Obstacle voidance for Unmanned Aerial Vehicles during teleoperation

Unmanned Aerial Vehicles (UAVs) use is on the rise, both for civilian and military applications. Autonomous UAV navigation is an active research topic, but human operators still provide a flexibility that currently matches or outperforms computers controlled aerial vehicles. For this reason, the remote control of a UAV by a human operator, or teleoperation, is an important subject of study. The challenge for UAV teleoperation comes from the loss of sensory information available for the operator who has to rely on onboard sensors to perceive the environment and the state of the UAV. Navigation in cluttered environment or small spaces is especially hard and demanding. A flight assistance framework could then bring significant benefits to the operator. In this thesis, an intelligent flight assistance framework for the teleoperation of rotary wings UAVs in small spaces is designed. A 3D haptic device serves as a remote control to improve ease of UAV manipulation for the operator. Moreover, the designed system provides benefits regarding three essential criteria: safety of the UAV, efficiency of the teleoperation and workload of the operator. In order to leverage the use of a 3D haptic controller, the initial obstacle avoidance algorithm proposed in this thesis is based on haptic feedback, where the feedback repels the UAV away from obstacles. This method is tested by human subjects, showing safety benefits but no manoeuvrability improvements. In order to improve on those criteria, the perception of the environment is studied using Light Detection And Ranging (LIDAR) and stereo cameras sensors data. The result of this led to the development of a mobile map of the obstacles surrounding the UAV using the LIDAR in addition to the stereo camera adopted to improve density. This map allows the creation of a flight assistance system that analyses and corrects the user’s inputs so that collisions are avoided while improving manoeuvrability. The proposed flight assistance system is validated through experiments involving untrained human subjects in a synthetically simulated environment. The results show that the proposed flight assistance system sharply reduces the number of collisions, the time required to complete the navigation task and the workload of the participant