ClusterNet: Detecting Small Objects in Large Scenes by Exploiting Spatio-Temporal Information

Object detection in wide area motion imagery (WAMI) has drawn the attention of the computer vision research community for a number of years. WAMI proposes a number of unique challenges including extremely small object sizes, both sparse and densely-packed objects, and extremely large search spaces (large video frames). Nearly all state-of-the-art methods in WAMI object detection report that appearance-based classifiers fail in this challenging data and instead rely almost entirely on motion information in the form of background subtraction or frame-differencing. In this work, we experimentally verify the failure of appearance-based classifiers in WAMI, such as Faster R-CNN and a heatmap-based fully convolutional neural network (CNN), and propose a novel two-stage spatio-temporal CNN which effectively and efficiently combines both appearance and motion information to significantly surpass the state-of-the-art in WAMI object detection. To reduce the large search space, the first stage (ClusterNet) takes in a set of extremely large video frames, combines the motion and appearance information within the convolutional architecture, and proposes regions of objects of interest (ROOBI). These ROOBI can contain from one to clusters of several hundred objects due to the large video frame size and varying object density in WAMI. The second stage (FoveaNet) then estimates the centroid location of all objects in that given ROOBI simultaneously via heatmap estimation. The proposed method exceeds state-of-the-art results on the WPAFB 2009 dataset by 5-16% for moving objects and nearly 50% for stopped objects, as well as being the first proposed method in wide area motion imagery to detect completely stationary objects.Comment: Main paper is 8 pages. Supplemental section contains a walk-through of our method (using a qualitative example) and qualitative results for WPAFB 2009 datase

Physics-Informed Computer Vision: A Review and Perspectives

Incorporation of physical information in machine learning frameworks are opening and transforming many application domains. Here the learning process is augmented through the induction of fundamental knowledge and governing physical laws. In this work we explore their utility for computer vision tasks in interpreting and understanding visual data. We present a systematic literature review of formulation and approaches to computer vision tasks guided by physical laws. We begin by decomposing the popular computer vision pipeline into a taxonomy of stages and investigate approaches to incorporate governing physical equations in each stage. Existing approaches in each task are analyzed with regard to what governing physical processes are modeled, formulated and how they are incorporated, i.e. modify data (observation bias), modify networks (inductive bias), and modify losses (learning bias). The taxonomy offers a unified view of the application of the physics-informed capability, highlighting where physics-informed learning has been conducted and where the gaps and opportunities are. Finally, we highlight open problems and challenges to inform future research. While still in its early days, the study of physics-informed computer vision has the promise to develop better computer vision models that can improve physical plausibility, accuracy, data efficiency and generalization in increasingly realistic applications

Time-consistent estimators of 2D/3D motion of atmospheric layers from pressure images

In this paper, we face the challenging problem of estimation of time-consistent layer motion fields at various atmospheric depths. Based on a vertical decomposition of the atmosphere, we propose three different dense motion estimator relying on multi-layer dynamical models. In the first method, we propose a mass conservation model which constitutes the physical background of a multi-layer dense estimator. In the perspective of adapting motion analysis to atmospheric motion, we propose in this method a two-stage decomposition estimation scheme. The second method proposed in this paper relying on a 3D physical model for a stack of interacting layers allows us to recover a vertical motion information. In the last method, we use the exact shallow-water formulation of the Navier-Stokes equations to control the motion evolution across the sequence. This is done through a variational approach derived from data assimilation principle which combines the dynamical model and the pressure difference observations obtained from satellite images. The three methods use sparse pressure difference image observations derived from top of cloud images and classification maps. The proposed approaches are validated on synthetic example and applied to real world meteorological satellite image sequences

A Comprehensive Survey of Deep Learning in Remote Sensing: Theories, Tools and Challenges for the Community

In recent years, deep learning (DL), a re-branding of neural networks (NNs), has risen to the top in numerous areas, namely computer vision (CV), speech recognition, natural language processing, etc. Whereas remote sensing (RS) possesses a number of unique challenges, primarily related to sensors and applications, inevitably RS draws from many of the same theories as CV; e.g., statistics, fusion, and machine learning, to name a few. This means that the RS community should be aware of, if not at the leading edge of, of advancements like DL. Herein, we provide the most comprehensive survey of state-of-the-art RS DL research. We also review recent new developments in the DL field that can be used in DL for RS. Namely, we focus on theories, tools and challenges for the RS community. Specifically, we focus on unsolved challenges and opportunities as it relates to (i) inadequate data sets, (ii) human-understandable solutions for modelling physical phenomena, (iii) Big Data, (iv) non-traditional heterogeneous data sources, (v) DL architectures and learning algorithms for spectral, spatial and temporal data, (vi) transfer learning, (vii) an improved theoretical understanding of DL systems, (viii) high barriers to entry, and (ix) training and optimizing the DL.Comment: 64 pages, 411 references. To appear in Journal of Applied Remote Sensin

The application of ocean front metrics for understanding habitat selection by marine predators

Marine predators such as seabirds, cetaceans, turtles, pinnipeds, sharks and large teleost fish are essential components of healthy, biologically diverse marine ecosystems. However, intense anthropogenic pressure on the global ocean is causing rapid and widespread change, and many predator populations are in decline. Conservation solutions are urgently required, yet only recently have we begun to comprehend how these animals interact with the vast and dynamic oceans that they inhabit. A better understanding of the mechanisms that underlie habitat selection at sea is critical to our knowledge of marine ecosystem functioning, and to ecologically-sensitive marine spatial planning. The collection of studies presented in this thesis aims to elucidate the influence of biophysical coupling at oceanographic fronts – physical interfaces at the transitions between water masses – on habitat selection by marine predators. High-resolution composite front mapping via Earth Observation remote sensing is used to provide oceanographic context to several biologging datasets describing the movements and behaviours of animals at sea. A series of species-habitat models reveal the influence of mesoscale (10s to 100s of kilometres) thermal and chlorophyll-a fronts on habitat selection by taxonomically diverse species inhabiting contrasting ocean regions; northern gannets (Morus bassanus; Celtic Sea), basking sharks (Cetorhinus maximus; north-east Atlantic), loggerhead turtles (Caretta caretta; Canary Current), and grey-headed albatrosses (Thalassarche chrysostoma; Southern Ocean). Original aspects of this work include an exploration of quantitative approaches to understanding habitat selection using remotely-sensed front metrics; and explicit investigation of how the biophysical properties of fronts and species-specific foraging ecology interact to influence associations. Main findings indicate that front metrics, particularly seasonal indices, are useful predictors of habitat preference across taxa. Moreover, frontal persistence and spatiotemporal predictability appear to mediate the use of front-associated foraging habitats, both in shelf seas and in the open oceans. These findings have implications for marine spatial planning and the design of protected area networks, and may prove useful in the development of tools supporting spatially dynamic ocean management