Deep learning on signals : discretization invariance, lossless compression and nonuniform compression

Une grande variété d'information se prête bien à être interprétée comme signal; à peu près toute quantité fluctuant continuellement dans l'espace se trouve inclue. La vie quotidienne abonde d'exemples; les images peuvent être vues comme une variation de couleur à travers l'espace bidimensionnel; le son, la pression à travers le temps; les environnements physiques, la matière à travers l'espace tridimensionnel. Les calculs sur ce type d'information requièrent nécessairement une transformation de la forme continue vers la forme discrète, ce qui est accompli par le processus de discrétisation, où seules quelques valeurs du signal continu sous-jacent sont observées et compilées en un signal discret. Sous certaines conditions, à l'aide seulement d'un nombre fini de valeurs observées, le signal discret capture la totalité de l'information comprise dans le signal continu, et permet de le reconstruire parfaitement. Les divers systèmes de senseurs permettant d'acquérir des signaux effectuent tous ce processus jusqu'à un certain niveau de fidélité, qu'il s'agisse d'une caméra, d'un enregistreur audio, ou d'un système de capture tridimensionnelle. Le processus de discrétisation n'est pas unique par contre. Pour un seul signal continu, il existe une infinité de signaux discrets qui lui sont équivalents, et entre lesquels les différences sont contingentes. Ces différences correspondent étroitement aux différences entre systèmes de senseurs, qui ont chacun leur niveau de fidélité et leurs particularités techniques. Les réseaux de neurones profonds sont fréquemment spécialisés pour le type de données spécifiques sur lesquels ils opèrent. Cette spécialisation se traduit souvent par des biais inductifs qui supportent des symétries intrinsèques au type de donnée. Quand le comportement d'une architecture neuronale reste inchangé par une certaine opération, l'architecture est dite invariante sous cette opération. Quand le comportement est affecté d'une manière identique, l'architecture est dite équivariante sous cette opération. Nous explorons en détail l'idée que les architectures neuronales puissent être formulées de façon plus générale si nous abstrayions les spécificités contingentes des signaux discrets, qui dépendent généralement de particularités de systèmes de senseurs, et considérions plutôt l'unique signal continu représenté, qui est la réelle information d'importance. Cette idée correspond au biais inductif de l'invariance à la discrétisation, qui reconnaît que les signaux ont une forme de symétrie à la discrétisation. Nous formulons une architecture très générale qui respecte ce biais inductif. Du fait même, l'architecture gagne la capacité d'être évaluée sur des discrétisations de taille arbitraire avec une grande robustesse, à l'entraînement et à l'inférence. Cela permet d'accéder à de plus grands corpus de données pour l'entraînement, qui peuvent être formés à partir de discrétisations hétérogènes. Cela permet aussi de déployer l'architecture dans un plus grand nombre de contextes où des systèmes de senseurs produisent des discrétisations variées. Nous formulons aussi cette architecture de façon à se généraliser à n'importe quel nombre de dimensions, ce qui la rend idéale pour une grande variété de signaux. Nous notons aussi que son coût d'évaluation diminue avec la taille de la discrétisation, ce qui est peu commun d'architectures conçues pour les signaux, qui ont généralement une discrétisation fixe. Nous remarquons qu'il existe un lien entre l'invariance à la discrétisation, et la distinction séparant l'équivariance à la translation discrète et l'équivariance à la translation continue. Ces deux propriétés reflètent la même symétrie à la translation, mais l'une est plus diluée que l'autre. Nous notons que la plus grande part de la littérature entourant les architectures motivées par l'algèbre générale omettent cette distinction, ce qui affaiblit la force des biais inductifs implémentés. Nous incorporons aussi dans notre méthode la capacité d'implémenter d'autres invariances and equivariances plus générales à l'aide de couches formulées à partir de l'opérateur de dérivée partielle. La symétrie à la translation, la rotation, la réflexion, et la mise à l'échelle peuvent être adoptées, et l'expressivité et l'efficacité en paramètres de la couche résultante sont excellentes. Nous introduisons aussi un nouveau bloc résiduel Laplacien, qui permet de compresser l'architecture sans perte en fonction de la densité de la discrétisation. À mesure que le nombre d'échantillons de la discrétisation réduit, le nombre de couches requises pour l'évaluation diminue aussi. Le coût de calcul de l'architecture diminue ainsi à mesure que certaines de ses couches sont retirées, mais elle se comporte de façon virtuellement identique; c'est ainsi une forme de compression sans perte qui est appliquée. La validité de cette compression sans perte est prouvée théoriquement, et démontrée empiriquement. Cette capacité est absente de la littérature antérieure, au meilleur de notre savoir. Nous greffons à ce mécanisme une forme de décrochage Laplacien, qui applique effectivement une augmentation spectrale aux données pendant l'entraînement. Cela mène à une grande augmentation de la robustesse de l'architecture à des dégradations de qualité de la discrétisation, sans toutefois compromettre sa capacité à performer optimalement sur des discrétisations de haute qualité. Nous n'observons pas cette capacité dans les méthodes comparées. Nous introduisons aussi un algorithme d'initialisation des poids qui ne dépend pas de dérivations analytiques, ce qui permet un prototypage rapide de couches plus exotiques. Nous introduisons finalement une méthode qui généralise notre architecture de l'application à des signaux échantillonnés uniformément vers des signaux échantillonnés non uniformément. Les garanties théoriques que nous fournissons sur son efficacité d'échantillonnage sont positives, mais la complexité ajoutée par la méthode limite malheureusement sa viabilité.Signals are a useful representation for many types of information that consist of continuously changing quantities. Examples from everyday life are abundant: images are fluctuations of colour over two-dimensional space; sounds are fluctuations of air pressure over time; physical environments are fluctuations of material qualities over three-dimensional space. Computation over this information requires that we reduce its continuous form to some discrete form. This is done through the process of discretization, where only a few values of the underlying continuous signal are observed and compiled into a discrete signal. This process incurs no loss of information and is reversible under some conditions. Sensor systems, such as cameras, sound recorders, and laser scanners all effectively perform discretization when they capture signals, and they preserve them up to a certain degree. This process is not unique, however. Given a single continuous signal, there are countless discrete signals that correspond to it, and the specific choice of discrete signal is generally contingent. Sensor systems all have different technical characteristics that lead to different discretizations. Deep neural network architectures are often tailored to respect the fundamental properties of the specific data type they operate on. Their behaviour often implements inductive biases that respect some fundamental symmetry of the data. When behaviour is unchanged by some operation, the architecture is invariant under it. When behaviour transparently reproduces some operation, the architecture is equivariant under it. We explore in great detail the idea that neural network architectures can be formulated in a more general way if we abstract away the contingent details of the discrete signal, which generally depend on the implementation details of a sensor system, and only consider the underlying continuous signal, which is the true information of interest. This is the intuitive idea behind discretization invariance. We formulate a very general architecture that implements this inductive bias. This allows handling discretizations of various sizes with much greater robustness, both during training and inference. We find that training can leverage more data by allowing heterogeneous discretizations, and that inference can apply to discretizations produced by a broader range of sensor systems. The architecture is agnostic to dimensionality, which makes it widely applicable to different types of signals. The architecture also lowers its computational cost proportionally to the sample count, which is unusual and highly desirable. We find that discretization invariance is also key to the distinction between discrete shift equivariance and continuous shift equivariance. We underline the fact that the majority of previous work on architecture design motivated by abstract algebra fails to consider this distinction. This nuance impacts the robustness of convolutional neural network architectures to translations on signals, weakening their inductive biases if unaddressed. We also incorporate the ability to implement more general invariances and equivariances by formulating steerable layers based on the partial derivative operator, and a set of other compatible architectural blocks. The framework we propose supports shift, rotation, reflection, and scale. We find that this results in excellent expressivity and parameter efficiency. We further improve computational efficiency with a novel Laplacian residual structure that allows lossless compression of the whole network depending on the sample density of the discretization. As the number of samples reduces, the number of layers required for evaluation also reduces. Pruning these layers reduces computational cost and has virtually no effect on the behaviour of the architecture. This is proven theoretically and demonstrated empirically. This capability is absent from any prior work to our knowledge. We also incorporate a novel form of Laplacian dropout within this structure, which performs a spectral augmentation to the data during training. This leads to greatly improved robustness to changes in spectral volume, meaning the architecture has a much greater tolerance to low-quality discretizations without compromising its performance on high-quality discretization. We do not observe this phenomenon in competing methods. We also provide a simple data-driven weight initialization scheme that allows quickly prototyping exotic layer types without analytically deriving weight initialization. We finally provide a method that generalizes our architecture from uniformly sampled signals to nonuniformly sampled signals. While the best-case theoretical guarantees it provides for sample efficiency are excellent, we find it is not viable in practice because of the complications it brings to the discretization of the architecture

Signal processing with Fourier analysis, novel algorithms and applications

Fourier analysis is the study of the way general functions may be represented or approximated by sums of simpler trigonometric functions, also analogously known as sinusoidal modeling. The original idea of Fourier had a profound impact on mathematical analysis, physics and engineering because it diagonalizes time-invariant convolution operators. In the past signal processing was a topic that stayed almost exclusively in electrical engineering, where only the experts could cancel noise, compress and reconstruct signals. Nowadays it is almost ubiquitous, as everyone now deals with modern digital signals. Medical imaging, wireless communications and power systems of the future will experience more data processing conditions and wider range of applications requirements than the systems of today. Such systems will require more powerful, efficient and flexible signal processing algorithms that are well designed to handle such needs. No matter how advanced our hardware technology becomes we will still need intelligent and efficient algorithms to address the growing demands in signal processing. In this thesis, we investigate novel techniques to solve a suite of four fundamental problems in signal processing that have a wide range of applications. The relevant equations, literature of signal processing applications, analysis and final numerical algorithms/methods to solve them using Fourier analysis are discussed for different applications in the electrical engineering/computer science. The first four chapters cover the following topics of central importance in the field of signal processing: • Fast Phasor Estimation using Adaptive Signal Processing (Chapter 2) • Frequency Estimation from Nonuniform Samples (Chapter 3) • 2D Polar and 3D Spherical Polar Nonuniform Discrete Fourier Transform (Chapter 4) • Robust 3D registration using Spherical Polar Discrete Fourier Transform and Spherical Harmonics (Chapter 5) Even though each of these four methods discussed may seem completely disparate, the underlying motivation for more efficient processing by exploiting the Fourier domain signal structure remains the same. The main contribution of this thesis is the innovation in the analysis, synthesis, discretization of certain well known problems like phasor estimation, frequency estimation, computations of a particular non-uniform Fourier transform and signal registration on the transformed domain. We conduct propositions and evaluations of certain applications relevant algorithms such as, frequency estimation algorithm using non-uniform sampling, polar and spherical polar Fourier transform. The techniques proposed are also useful in the field of computer vision and medical imaging. From a practical perspective, the proposed algorithms are shown to improve the existing solutions in the respective fields where they are applied/evaluated. The formulation and final proposition is shown to have a variety of benefits. Future work with potentials in medical imaging, directional wavelets, volume rendering, video/3D object classifications, high dimensional registration are also discussed in the final chapter. Finally, in the spirit of reproducible research we release the implementation of these algorithms to the public using Github

Graph Spectral Image Processing

Recent advent of graph signal processing (GSP) has spurred intensive studies of signals that live naturally on irregular data kernels described by graphs (e.g., social networks, wireless sensor networks). Though a digital image contains pixels that reside on a regularly sampled 2D grid, if one can design an appropriate underlying graph connecting pixels with weights that reflect the image structure, then one can interpret the image (or image patch) as a signal on a graph, and apply GSP tools for processing and analysis of the signal in graph spectral domain. In this article, we overview recent graph spectral techniques in GSP specifically for image / video processing. The topics covered include image compression, image restoration, image filtering and image segmentation

Underwater Localization in a Confined Space Using Acoustic Positioning and Machine Learning

Localization is a critical step in any navigation system. Through localization, the vehicle can estimate its position in the surrounding environment and plan how to reach its goal without any collision. This thesis focuses on underwater source localization, using sound signals for position estimation. We propose a novel underwater localization method based on machine learning techniques in which source position is directly estimated from collected acoustic data. The position of the sound source is estimated by training Random Forest (RF), Support Vector Machine (SVM), Feedforward Neural Network (FNN), and Convolutional Neural Network (CNN). To train these data-driven methods, data are collected inside a confined test tank with dimensions of 6m x 4.5m x 1.7m. The transmission unit, which includes Xilinx LX45 FPGA and transducer, generates acoustic signal. The receiver unit collects and prepares propagated sound signals and transmit them to a computer. It consists of 4 hydrophones, Red Pitay analog front-end board, and NI 9234 data acquisition board. We used MATLAB 2018 to extract pitch, Mel-Frequency Cepstrum Coefficients (MFCC), and spectrogram from the sound signals. These features are used by MATLAB Toolboxes to train RF, SVM, FNN, and CNN. Experimental results show that CNN archives 4% of Mean Absolute Percentage Error (MAPE) in the test tank. The finding of this research can pave the way for Autonomous Underwater Vehicle (AUV) and Remotely Operated Vehicle (ROV) navigation in underwater open spaces

Plenoptic Signal Processing for Robust Vision in Field Robotics

This thesis proposes the use of plenoptic cameras for improving the robustness and simplicity of machine vision in field robotics applications. Dust, rain, fog, snow, murky water and insufficient light can cause even the most sophisticated vision systems to fail. Plenoptic cameras offer an appealing alternative to conventional imagery by gathering significantly more light over a wider depth of field, and capturing a rich 4D light field structure that encodes textural and geometric information. The key contributions of this work lie in exploring the properties of plenoptic signals and developing algorithms for exploiting them. It lays the groundwork for the deployment of plenoptic cameras in field robotics by establishing a decoding, calibration and rectification scheme appropriate to compact, lenslet-based devices. Next, the frequency-domain shape of plenoptic signals is elaborated and exploited by constructing a filter which focuses over a wide depth of field rather than at a single depth. This filter is shown to reject noise, improving contrast in low light and through attenuating media, while mitigating occluders such as snow, rain and underwater particulate matter. Next, a closed-form generalization of optical flow is presented which directly estimates camera motion from first-order derivatives. An elegant adaptation of this "plenoptic flow" to lenslet-based imagery is demonstrated, as well as a simple, additive method for rendering novel views. Finally, the isolation of dynamic elements from a static background is considered, a task complicated by the non-uniform apparent motion caused by a mobile camera. Two elegant closed-form solutions are presented dealing with monocular time-series and light field image pairs. This work emphasizes non-iterative, noise-tolerant, closed-form, linear methods with predictable and constant runtimes, making them suitable for real-time embedded implementation in field robotics applications

Shape reconstruction from shading using linear approximation

Shape from shading (SFS) deals with the recovery of 3D shape from a single monocular image. This problem was formally introduced by Horn in the early 1970s. Since then it has received considerable attention, and several efforts have been made to improve the shape recovery. In this thesis, we present a fast SFS algorithm, which is a purely local method and is highly parallelizable. In our approach, we first use the discrete approximations for surface gradients, p and q, using finite differences, then linearize the reflectance function in depth, Z ( x , y), instead of p and q. This method is simple and efficient, and yields better results for images with central illumination or low-angle illumination. Furthermore, our method is more general, and can be applied to either Lambertian surfaces or specular surfaces. The algorithm has been tested on several synthetic and real images of both Lambertian and specular surfaces, and good results have been obtained. However, our method assumes that the input image contains only single object with uniform albedo values, which is commonly assumed in most SFS methods. Our algorithm performs poorly on images with nonuniform albedo values and produces incorrect shape for images containing objects with scale ambiguity, because those images violate the basic assumptions made by our SFS method. Therefore, we extended our method for images with nonuniform albedo values. We first estimate the albedo values for each pixel, and segment the scene into regions with uniform albedo values. Then we adjust the intensity value for each pixel by dividing the corresponding albedo value before applying our linear shape from shading method. This way our modified method is able to deal with nonuniform albedo values. When multiple objects differing only in scale are present in a scene, there may be points with the same surface orientation but different depth values. No existing SFS methods can solve this kind of ambiguity directly. We also present a new approach to deal with images containing multiple objects with scale ambiguity. A depth estimate is derived from patches using a minimum downhill approach and re-aligned based on the background information to get the correct depth map. Experimental results are presented for several synthetic and real images. Finally, this thesis also investigates the problem of the discrete approximation under perspective projection. The straightforward finite difference approximation for surface gradients used under orthographic projection is no longer applicable here. because the image position components are in fact functions of the depth. In this thesis, we provide a direct solution for the discrete approximation under perspective projection. The surface gradient is derived mathematically by relating the depth value of the surface point with the depth value of the corresponding image point. We also demonstrate how we can apply the new discrete approximation to a more complicated and realistic reflectance model for SFS problem

Plenoptic Signal Processing for Robust Vision in Field Robotics

This thesis proposes the use of plenoptic cameras for improving the robustness and simplicity of machine vision in field robotics applications. Dust, rain, fog, snow, murky water and insufficient light can cause even the most sophisticated vision systems to fail. Plenoptic cameras offer an appealing alternative to conventional imagery by gathering significantly more light over a wider depth of field, and capturing a rich 4D light field structure that encodes textural and geometric information. The key contributions of this work lie in exploring the properties of plenoptic signals and developing algorithms for exploiting them. It lays the groundwork for the deployment of plenoptic cameras in field robotics by establishing a decoding, calibration and rectification scheme appropriate to compact, lenslet-based devices. Next, the frequency-domain shape of plenoptic signals is elaborated and exploited by constructing a filter which focuses over a wide depth of field rather than at a single depth. This filter is shown to reject noise, improving contrast in low light and through attenuating media, while mitigating occluders such as snow, rain and underwater particulate matter. Next, a closed-form generalization of optical flow is presented which directly estimates camera motion from first-order derivatives. An elegant adaptation of this "plenoptic flow" to lenslet-based imagery is demonstrated, as well as a simple, additive method for rendering novel views. Finally, the isolation of dynamic elements from a static background is considered, a task complicated by the non-uniform apparent motion caused by a mobile camera. Two elegant closed-form solutions are presented dealing with monocular time-series and light field image pairs. This work emphasizes non-iterative, noise-tolerant, closed-form, linear methods with predictable and constant runtimes, making them suitable for real-time embedded implementation in field robotics applications

Biologically inspired composite image sensor for deep field target tracking

The use of nonuniform image sensors in mobile based computer vision applications can be an effective solution when computational burden is problematic. Nonuniform image sensors are still in their infancy and as such have not been fully investigated for their unique qualities nor have they been extensively applied in practice. In this dissertation a system has been developed that can perform vision tasks in both the far field and the near field. In order to accomplish this, a new and novel image sensor system has been developed. Inspired by the biological aspects of the visual systems found in both falcons and primates, a composite multi-camera sensor was constructed. The sensor provides for expandable visual range, excellent depth of field, and produces a single compact output image based on the log-polar retinal-cortical mapping that occurs in primates. This mapping provides for scale and rotational tolerant processing which, in turn, supports the mitigation of perspective distortion found in strict Cartesian based sensor systems. Furthermore, the scale-tolerant representation of objects moving on trajectories parallel to the sensor\u27s optical axis allows for fast acquisition and tracking of objects moving at high rates of speed. In order to investigate how effective this combination would be for object detection and tracking at both near and far field, the system was tuned for the application of vehicle detection and tracking from a moving platform. Finally, it was shown that the capturing of license plate information in an autonomous fashion could easily be accomplished from the extraction of information contained in the mapped log-polar representation space. The novel composite log-polar deep-field image sensor opens new horizons for computer vision. This current work demonstrates features that can benefit applications beyond the high-speed vehicle tracking for drivers assistance and license plate capture. Some of the future applications envisioned include obstacle detection for high-speed trains, computer assisted aircraft landing, and computer assisted spacecraft docking