Multimodal machine learning for intelligent mobility

Scientific problems are solved by finding the optimal solution for a specific task. Some problems can be solved analytically while other problems are solved using data driven methods. The use of digital technologies to improve the transportation of people and goods, which is referred to as intelligent mobility, is one of the principal beneficiaries of data driven solutions. Autonomous vehicles are at the heart of the developments that propel Intelligent Mobility. Due to the high dimensionality and complexities involved in real-world environments, it needs to become commonplace for intelligent mobility to use data-driven solutions. As it is near impossible to program decision making logic for every eventuality manually. While recent developments of data-driven solutions such as deep learning facilitate machines to learn effectively from large datasets, the application of techniques within safety-critical systems such as driverless cars remain scarce.Autonomous vehicles need to be able to make context-driven decisions autonomously in different environments in which they operate. The recent literature on driverless vehicle research is heavily focused only on road or highway environments but have discounted pedestrianized areas and indoor environments. These unstructured environments tend to have more clutter and change rapidly over time. Therefore, for intelligent mobility to make a significant impact on human life, it is vital to extend the application beyond the structured environments. To further advance intelligent mobility, researchers need to take cues from multiple sensor streams, and multiple machine learning algorithms so that decisions can be robust and reliable. Only then will machines indeed be able to operate in unstructured and dynamic environments safely. Towards addressing these limitations, this thesis investigates data driven solutions towards crucial building blocks in intelligent mobility. Specifically, the thesis investigates multimodal sensor data fusion, machine learning, multimodal deep representation learning and its application of intelligent mobility. This work demonstrates that mobile robots can use multimodal machine learning to derive driver policy and therefore make autonomous decisions.To facilitate autonomous decisions necessary to derive safe driving algorithms, we present an algorithm for free space detection and human activity recognition. Driving these decision-making algorithms are specific datasets collected throughout this study. They include the Loughborough London Autonomous Vehicle dataset, and the Loughborough London Human Activity Recognition dataset. The datasets were collected using an autonomous platform design and developed in house as part of this research activity. The proposed framework for Free-Space Detection is based on an active learning paradigm that leverages the relative uncertainty of multimodal sensor data streams (ultrasound and camera). It utilizes an online learning methodology to continuously update the learnt model whenever the vehicle experiences new environments. The proposed Free Space Detection algorithm enables an autonomous vehicle to self-learn, evolve and adapt to new environments never encountered before. The results illustrate that online learning mechanism is superior to one-off training of deep neural networks that require large datasets to generalize to unfamiliar surroundings. The thesis takes the view that human should be at the centre of any technological development related to artificial intelligence. It is imperative within the spectrum of intelligent mobility where an autonomous vehicle should be aware of what humans are doing in its vicinity. Towards improving the robustness of human activity recognition, this thesis proposes a novel algorithm that classifies point-cloud data originated from Light Detection and Ranging sensors. The proposed algorithm leverages multimodality by using the camera data to identify humans and segment the region of interest in point cloud data. The corresponding 3-dimensional data was converted to a Fisher Vector Representation before being classified by a deep Convolutional Neural Network. The proposed algorithm classifies the indoor activities performed by a human subject with an average precision of 90.3%. When compared to an alternative point cloud classifier, PointNet[1], [2], the proposed framework out preformed on all classes. The developed autonomous testbed for data collection and algorithm validation, as well as the multimodal data-driven solutions for driverless cars, is the major contributions of this thesis. It is anticipated that these results and the testbed will have significant implications on the future of intelligent mobility by amplifying the developments of intelligent driverless vehicles.</div

3D shape recognition system by ultrasonic sensor array and genetic algorithms

This paper describes 3D shape recognition system using ultrasound pressure data and a Genetic Algorithm. The ultrasonic 3D shape recognition system using has commonly used a Neural Network (NN). However, a NN perform poorly when lacking learned data. In order to overcome this problem when using a NN, we here attempt to replace the NN with a Genetic Algorithm (GA). Unlike a NN, the GA can recognize shapes without depending on learned data. Experimental results demonstrate that the recognition ratios of the proposed recognition system using the GA are higher than that of a conventional 3D shape recognition system using a NN. Therefore, it is shown that our ultrasonic 3D shape recognition system is effective for many industrial applications. </p

An electromagnetic imaging system for metallic object detection and classification

PhD ThesisElectromagnetic imaging currently plays a vital role in various disciplines, from engineering to medical applications and is based upon the characteristics of electromagnetic fields and their interaction with the properties of materials. The detection and characterisation of metallic objects which pose a threat to safety is of great interest in relation to public and homeland security worldwide. Inspections are conducted under the prerequisite that is divested of all metallic objects. These inspection conditions are problematic in terms of the disruption of the movement of people and produce a soft target for terrorist attack. Thus, there is a need for a new generation of detection systems and information technologies which can provide an enhanced characterisation and discrimination capabilities. This thesis proposes an automatic metallic object detection and classification system. Two related topics have been addressed: to design and implement a new metallic object detection system; and to develop an appropriate signal processing algorithm to classify the targeted signatures. The new detection system uses an array of sensors in conjunction with pulsed excitation. The contributions of this research can be summarised as follows: (1) investigating the possibility of using magneto-resistance sensors for metallic object detection; (2) evaluating the proposed system by generating a database consisting of 12 real handguns with more than 20 objects used in daily life; (3) extracted features from the system outcomes using four feature categories referring to the objects’ shape, material composition, time-frequency signal analysis and transient pulse response; and (4) applying two classification methods to classify the objects into threats and non-threats, giving a successful classification rate of more than 92% using the feature combination and classification framework of the new system. The study concludes that novel magnetic field imaging system and their signal outputs can be used to detect, identify and classify metallic objects. In comparison with conventional induction-based walk-through metal detectors, the magneto-resistance sensor array-based system shows great potential for object identification and discrimination. This novel system design and signal processing achievement may be able to produce significant improvements in automatic threat object detection and classification applications.Iraqi Cultural Attaché, Londo

Dynamic Hand Gesture Recognition Using Ultrasonic Sonar Sensors and Deep Learning

The space of hand gesture recognition using radar and sonar is dominated mostly by radar applications. In addition, the machine learning algorithms used by these systems are typically based on convolutional neural networks with some applications exploring the use of long short term memory networks. The goal of this study was to build and design a Sonar system that can classify hand gestures using a machine learning approach. Secondly, the study aims to compare convolutional neural networks to long short term memory networks as a means to classify hand gestures using sonar. A Doppler Sonar system was designed and built to be able to sense hand gestures. The Sonar system is a multi-static system containing one transmitter and three receivers. The sonar system can measure the Doppler frequency shifts caused by dynamic hand gestures. Since the system uses three receivers, three different Doppler frequency channels are measured. Three additional differential frequency channels are formed by computing the differences between the frequency of each of the receivers. These six channels are used as inputs to the deep learning models. Two different deep learning algorithms were used to classify the hand gestures; a Doppler biLSTM network [1] and a CNN [2]. Six basic hand gestures, two in each x- y- and z-axis, and two rotational hand gestures are recorded using both left and right hand at different distances. The gestures were also recorded using both left and right hands. Ten-Fold cross-validation is used to evaluate the networks' performance and classification accuracy. The LSTM was able to classify the six basic gestures with an accuracy of at least 96% but with the addition of the two rotational gestures, the accuracy drops to 47%. This result is acceptable since the basic gestures are more commonly used gestures than rotational gestures. The CNN was able to classify all the gestures with an accuracy of at least 98%. Additionally, The LSTM network is also able to classify separate left and right-hand gestures with an accuracy of 80% and The CNN with an accuracy of 83%. The study shows that CNN is the most widely used algorithm for hand gesture recognition as it can consistently classify gestures with various degrees of complexity. The study also shows that the LSTM network can also classify hand gestures with a high degree of accuracy. More experimentation, however, needs to be done in order to increase the complexity of recognisable gestures