Improving Sonar Image Patch Matching via Deep Learning

Matching sonar images with high accuracy has been a problem for a long time, as sonar images are inherently hard to model due to reflections, noise and viewpoint dependence. Autonomous Underwater Vehicles require good sonar image matching capabilities for tasks such as tracking, simultaneous localization and mapping (SLAM) and some cases of object detection/recognition. We propose the use of Convolutional Neural Networks (CNN) to learn a matching function that can be trained from labeled sonar data, after pre-processing to generate matching and non-matching pairs. In a dataset of 39K training pairs, we obtain 0.91 Area under the ROC Curve (AUC) for a CNN that outputs a binary classification matching decision, and 0.89 AUC for another CNN that outputs a matching score. In comparison, classical keypoint matching methods like SIFT, SURF, ORB and AKAZE obtain AUC 0.61 to 0.68. Alternative learning methods obtain similar results, with a Random Forest Classifier obtaining AUC 0.79, and a Support Vector Machine resulting in AUC 0.66.Comment: Author versio

Detecting fish aggregations from reef habitats mapped with high resolution side scan sonar imagery

As part of a multibeam and side scan sonar (SSS) benthic survey of the Marine Conservation District (MCD) south of St. Thomas, USVI and the seasonal closed areas in St. Croix—Lang Bank (LB) for red hind (Epinephelus guttatus) and the Mutton Snapper (MS) (Lutjanus analis) area—we extracted signals from water column targets that represent individual and aggregated fish over various benthic habitats encountered in the SSS imagery. The survey covered a total of 18 km2 throughout the federal jurisdiction fishery management areas. The complementary set of 28 habitat classification digital maps covered a total of 5,462.3 ha; MCDW (West) accounted for 45% of that area, and MCDE (East) 26%, LB 17%, and MS the remaining 13%. With the exception of MS, corals and gorgonians on consolidated habitats were significantly more abundant than submerged aquatic vegetation (SAV) on unconsolidated sediments or unconsolidated sediments. Continuous coral habitat was the most abundant consolidated habitat for both MCDW and MCDE (41% and 43% respectively). Consolidated habitats in LB and MS predominantly consisted of gorgonian plain habitat with 95% and 83% respectively. Coral limestone habitat was more abundant than coral patch habitat; it was found near the shelf break in MS, MCDW, and MCDE. Coral limestone and coral patch habitats only covered LB minimally. The high spatial resolution (0.15 m) of the acquired imagery allowed the detection of differing fish aggregation (FA) types. The largest FA densities were located at MCDW and MCDE over coral communities that occupy up to 70% of the bottom cover. Counts of unidentified swimming objects (USOs), likely representing individual fish, were similar among locations and occurred primarily over sand and shelf edge areas. Fish aggregation school sizes were significantly smaller at MS than the other three locations (MCDW, MCDE, and LB). This study shows the advantages of utilizing SSS in determining fish distributions and density

Underwater Localization in Complex Environments

A capacidade de um veículo autónomo submarino (AUV) se localizar num ambiente complexo, bem como de extrair características relevantes do mesmo, é de grande importância para o sucesso da navegação. No entanto, esta tarefa é particularmente desafiante em ambientes subaquáticos devido à rápida atenuação sofrida pelos sinais de sistemas de posicionamento global ou outros sinais de radiofrequência, dispersão e reflexão, sendo assim necessário o uso de processos de filtragem. Ambiente complexo é definido aqui como um cenário com objetos destacados das paredes, por exemplo, o objeto pode ter uma certa variabilidade de orientação, portanto a sua posição nem sempre é conhecida. Exemplos de cenários podem ser um porto, um tanque ou mesmo uma barragem, onde existem paredes e dentro dessas paredes um AUV pode ter a necessidade de se localizar de acordo com os outros veículos na área e se posicionar em relação ao mesmo e analisá-lo. Os veículos autónomos empregam muitos tipos diferentes de sensores para localização e percepção dos seus ambientes e dependem dos computadores de bordo para realizar tarefas de direção autónoma. Para esta dissertação há um problema concreto a resolver, localizar um cabo suspenso numa coluna de água em uma região conhecida do mar e navegar de acordo com ela. Embora a posição do cabo no mundo seja bem conhecida, a dinâmica do cabo não permite saber exatamente onde ele está. Assim, para que o veículo se localize de acordo com este para que possa ser inspecionado, a localização deve ser baseada em sensores ópticos e acústicos. Este estudo explora o processamento e a análise de imagens óticas e acústicas, por meio dos dados adquiridos através de uma câmara e por um sonar de varrimento mecânico (MSIS),respetivamente, a fim de extrair características ambientais relevantes que possibilitem a estimação da localização do veículo. Os pontos de interesse extraídos de cada um dos sensores são utilizados para alimentar um estimador de posição, implementando um Filtro de Kalman Extendido (EKF), de modo a estimar a posição do cabo e através do feedback do filtro melhorar os processos de extração de pontos de interesse utilizados.The ability of an autonomous underwater vehicle (AUV) to locate itself in a complex environment as well as to detect relevant environmental features is of crucial importance for successful navigation. However, it's particularly challenging in underwater environments due to the rapid attenuation suffered by signals from global positioning systems or other radio frequency signals, dispersion and reflection thus needing a filtering process. Complex environment is defined here as a scenario with objects detached from the walls, for example the object can have a certain orientation variability therefore its position is not always known. Examples of scenarios can be a harbour, a tank or even a dam reservoir, where there are walls and within those walls an AUV may have the need to localize itself according to the other vehicles in the area and position itself relative to one to observe, analyse or scan it. Autonomous vehicles employ many different types of sensors for localization and perceiving their environments and they depend on the on-board computers to perform autonomous driving tasks. For this dissertation there is a concrete problem to solve, which is to locate a suspended cable in a water column in a known region in the sea and navigate according to it. Although the cable position in the world is well known, the cable dynamics does not allow knowing where it is exactly. So, in order to the vehicle localize itself according to it so it can be inspected, the localization has to be based on optical and acoustic sensors. This study explores the processing and analysis of optical and acoustic images, through the data acquired through a camera and by a mechanical scanning sonar (MSIS), respectively, in order to extract relevant environmental characteristics that allow the estimation of the location of the vehicle. The points of interest extracted from each of the sensors are used to feed a position estimator, by implementing an Extended Kalman Filter (EKF), in order to estimate the position of the cable and through the feedback of the filter improve the extraction processes of points of interest used

Underwater localization using imaging sonars in 3D environments

This work proposes a localization method using a mechanically scanned imaging sonar (MSIS), which stands out by its low cost and weight. The proposed method implements a Particle Filter, a Bayesian Estimator, and introduces a measurement model based on sonar simulation theory. To the best of author’s knowledge, there is no similar approach in the literature, as sonar simulation current methods target in syntethic data generation, mostly for object recognition . This stands as the major contribution of the thesis as allows the introduction of the computation of intensity values provided by imaging sonars, while maitaining compability with the already used methods, such as range extraction. Simulations shows the efficiency of the method as well its viability to the utilization of imaging sonar in underwater localization. The new approach make possible, under certain constraints, the extraction of 3D information from a sensor considered, in the literature, as 2D and also in situations where there is no reference at the same horizontal plane of the sensor transducer scanning axis. The localization in complex 3D environment is also an advantage provided by the proposed method.Este trabalho propõe um método de localização utilizando um sonar do tipo MSIS (Mechanically Scanned Imaging Sonar ), o qual se destaca por seu baixo custo e peso. O método implementa um filtro de partículas, um estimador Bayesiano, e introduz um modelo de medição baseado na teoria de simulação de sonares. No conhecimento do autor não há uma abordagem similar na literatura, uma vez que os métodos atuais de simulação de sonar visam a geração de dados sintéticos para o reconhecimento de objetos. Esta é a maior contribuição da tese pois permite a a computação dos valores de intensidade fornecidos pelos sonares do tipo imaging e ao mesmo tempo é compatível com os métodos já utilizados, como extração de distância. Simulações mostram o bom desempenho do método, assim como sua viabilidade para o uso de imaging sonars na localização submarina. A nova abordagem tornou possível, sob certas restrições, a extração de informações 3D de um sensor considerado, na literatura, como somente 2D e também em situações em que não há nehnuma referência no mesmo plano horizontal do eixo de escaneamento do transdutor. A localização em ambientes 3D complexos é também uma vantagem proporcionada pelo método proposto

A COLLISION AVOIDANCE SYSTEM FOR AUTONOMOUS UNDERWATER VEHICLES

The work in this thesis is concerned with the development of a novel and practical collision avoidance system for autonomous underwater vehicles (AUVs). Synergistically, advanced stochastic motion planning methods, dynamics quantisation approaches, multivariable tracking controller designs, sonar data processing and workspace representation, are combined to enhance significantly the survivability of modern AUVs. The recent proliferation of autonomous AUV deployments for various missions such as seafloor surveying, scientific data gathering and mine hunting has demanded a substantial increase in vehicle autonomy. One matching requirement of such missions is to allow all the AUV to navigate safely in a dynamic and unstructured environment. Therefore, it is vital that a robust and effective collision avoidance system should be forthcoming in order to preserve the structural integrity of the vehicle whilst simultaneously increasing its autonomy. This thesis not only provides a holistic framework but also an arsenal of computational techniques in the design of a collision avoidance system for AUVs. The design of an obstacle avoidance system is first addressed. The core paradigm is the application of the Rapidly-exploring Random Tree (RRT) algorithm and the newly developed version for use as a motion planning tool. Later, this technique is merged with the Manoeuvre Automaton (MA) representation to address the inherent disadvantages of the RRT. A novel multi-node version which can also address time varying final state is suggested. Clearly, the reference trajectory generated by the aforementioned embedded planner must be tracked. Hence, the feasibility of employing the linear quadratic regulator (LQG) and the nonlinear kinematic based state-dependent Ricatti equation (SDRE) controller as trajectory trackers are explored. The obstacle detection module, which comprises of sonar processing and workspace representation submodules, is developed and tested on actual sonar data acquired in a sea-trial via a prototype forward looking sonar (AT500). The sonar processing techniques applied are fundamentally derived from the image processing perspective. Likewise, a novel occupancy grid using nonlinear function is proposed for the workspace representation of the AUV. Results are presented that demonstrate the ability of an AUV to navigate a complex environment. To the author's knowledge, it is the first time the above newly developed methodologies have been applied to an A UV collision avoidance system, and, therefore, it is considered that the work constitutes a contribution of knowledge in this area of work.J&S MARINE LT

High-frequency side-scan sonar fish reconnaissance by autonomous underwater vehicles

Author Posting. © The Author(s), 2016. This is the author's version of the work. It is posted here by permission of NRC Research Press for personal use, not for redistribution. The definitive version was published in Canadian Journal of Fisheries and Aquatic Sciences 74 (2017): 240-255, doi:10.1139/cjfas-2015-0301.A dichotomy between depth penetration and resolution as a function of sonar frequency, draw resolution, and beam spread challenges fish target classification from sonar. Moving high-frequency sources to depth using autonomous underwater vehicles (AUVs) mitigates this and also co-locates transducers with other AUV-mounted short-range sensors to allow a holistic approach to ecological surveys. This widely available tool with a pedigree for bottom mapping is not commonly applied to fish reconnaissance and requires the development of an interpretation of pelagic reflective features, revisitation of count methods, image-processing rather than wave-form recognition for automation, and an understanding of bias. In a series of AUV mission test cases, side-scan sonar (600 and 900 kHz) returns often resolved individual school members, spacing, size, behavior, and (infrequently) species from anatomical features and could be intuitively classified by ecologists — but also produced artifacts. Fish often followed the AUV and thus were videographed, but in doing so removed themselves from the sonar aperture. AUV-supported high-frequency side-scan holds particular promise for survey of scarce, large species or for synergistic investigation of predators and their prey because the spatial scale of observations may be similar to those of predators.AUV missions were funded by an Office of Naval Research grant to the Woods Hole Oceanographic Institution and Rutgers University. The field work was supported by the Office of Naval Research under grant N00014-11-1-0160