Exploring Motion Signatures for Vision-Based Tracking, Recognition and Navigation

As cameras become more and more popular in intelligent systems, algorithms and systems for understanding video data become more and more important. There is a broad range of applications, including object detection, tracking, scene understanding, and robot navigation. Besides the stationary information, video data contains rich motion information of the environment. Biological visual systems, like human and animal eyes, are very sensitive to the motion information. This inspires active research on vision-based motion analysis in recent years. The main focus of motion analysis has been on low level motion representations of pixels and image regions. However, the motion signatures can benefit a broader range of applications if further in-depth analysis techniques are developed. In this dissertation, we mainly discuss how to exploit motion signatures to solve problems in two applications: object recognition and robot navigation. First, we use bird species recognition as the application to explore motion signatures for object recognition. We begin with study of the periodic wingbeat motion of flying birds. To analyze the wing motion of a flying bird, we establish kinematics models for bird wings, and obtain wingbeat periodicity in image frames after the perspective projection. Time series of salient extremities on bird images are extracted, and the wingbeat frequency is acquired for species classification. Physical experiments show that the frequency based recognition method is robust to segmentation errors and measurement lost up to 30%. In addition to the wing motion, the body motion of the bird is also analyzed to extract the flying velocity in 3D space. An interacting multi-model approach is then designed to capture the combined object motion patterns and different environment conditions. The proposed systems and algorithms are tested in physical experiments, and the results show a false positive rate of around 20% with a low false negative rate close to zero. Second, we explore motion signatures for vision-based vehicle navigation. We discover that motion vectors (MVs) encoded in Moving Picture Experts Group (MPEG) videos provide rich information of the motion in the environment, which can be used to reconstruct the vehicle ego-motion and the structure of the scene. However, MVs suffer from high noise level. To handle the challenge, an error propagation model for MVs is first proposed. Several steps, including MV merging, plane-at-infinity elimination, and planar region extraction, are designed to further reduce noises. The extracted planes are used as landmarks in an extended Kalman filter (EKF) for simultaneous localization and mapping. Results show that the algorithm performs localization and plane mapping with a relative trajectory error below 5:1%. Exploiting the fact that MVs encodes both environment information and moving obstacles, we further propose to track moving objects at the same time of localization and mapping. This enables the two critical navigation functionalities, localization and obstacle avoidance, to be performed in a single framework. MVs are labeled as stationary or moving according to their consistency to geometric constraints. Therefore, the extracted planes are separated into moving objects and the stationary scene. Multiple EKFs are used to track the static scene and the moving objects simultaneously. In physical experiments, we show a detection rate of moving objects at 96:6% and a mean absolute localization error below 3:5 meters

Radar And Visual Observations Of Autumnal (Southward) Shorebird Migration On Guam

Several species of shorebirds migrate between eastern Asia and the southern Pacific islands, Australia, and New Zealand. Observations made from Guam (13°25′N, 144°45′E) during autumn 1983 indicate that a significant number of birds take a direct route over the western Pacific Ocean. Radar observations and ground counts of migrants on Guam showed two periods of autumnal migratory activity. The first, largely adult birds, was in August and September. The second, largely juveniles, was in late September and October. Radar indicated that large numbers of birds passed over the island to the south with no evidence of compensation for drift by the easterly winds. Comparison of radar and ground observations on Guam showed that only a small subset of migrants stop on the island, suggesting that some species may make nonstop flights between eastern Asia and the South Pacific

An approach toward an analysis of the pattern recognition involved in the stellar orientation of birds

A conditioning method was used to investigate the orientational responses of ducks as affected by manipulations of the stellar patterns in a planetarium. Under simulated natural skies it was possible to train a bird to a particular direction successively under all positions of the rotating sphere at a constant latitude. The responses were independent of the phase relationships between local time, season, and appearance of the sky provided the bird had been trained under the particular sector of the sphere some time before

Automatic detection, tracking and counting of birds in marine video content

Robust automatic detection of moving objects in a marine context is a multi-faceted problem due to the complexity of the observed scene. The dynamic nature of the sea caused by waves, boat wakes, and weather conditions poses huge challenges for the development of a stable background model. Moreover, camera motion, reflections, lightning and illumination changes may contribute to false detections. Dynamic background subtraction (DBGS) is widely considered as a solution to tackle this issue in the scope of vessel detection for maritime traffic analysis. In this paper, the DBGS techniques suggested for ships are investigated and optimized for the monitoring and tracking of birds in marine video content. In addition to background subtraction, foreground candidates are filtered by a classifier based on their feature descriptors in order to remove non-bird objects. Different types of classifiers have been evaluated and results on a ground truth labeled dataset of challenging video fragments show similar levels of precision and recall of about 95% for the best performing classifier. The remaining foreground items are counted and birds are tracked along the video sequence using spatio-temporal motion prediction. This allows marine scientists to study the presence and behavior of birds

Bird Migration Through A Mountain Pass Studied With High Resolution Radar, Ceilometers, And Census

Autumnal migration was studied with high-resolution radar, ceilometer, and daily census in the area of Franconia Notch, a major pass in the northern Appalachian Mountains. Under synoptic conditions favorable for migration, broadfront movements of migrants toward the south passed over the mountains, often above a temperature inversion. Birds at lower elevations appeared to be influenced by local topography. Birds moving southwest were concentrated along the face of the mountain range. Birds appeared to deviate their flights to follow local topography through the pass. Specific migratory behavior was not associated with species or species groups. Under synoptic conditions unfavorable for southward migration, multimodal movements probably associated with local flights were as dense as the southward migrations described above. Avian migrants reacting to local terrain may result in concentrations of migrants over ridge summits or other topographic features

Object-Oriented Dynamics Learning through Multi-Level Abstraction

Object-based approaches for learning action-conditioned dynamics has demonstrated promise for generalization and interpretability. However, existing approaches suffer from structural limitations and optimization difficulties for common environments with multiple dynamic objects. In this paper, we present a novel self-supervised learning framework, called Multi-level Abstraction Object-oriented Predictor (MAOP), which employs a three-level learning architecture that enables efficient object-based dynamics learning from raw visual observations. We also design a spatial-temporal relational reasoning mechanism for MAOP to support instance-level dynamics learning and handle partial observability. Our results show that MAOP significantly outperforms previous methods in terms of sample efficiency and generalization over novel environments for learning environment models. We also demonstrate that learned dynamics models enable efficient planning in unseen environments, comparable to true environment models. In addition, MAOP learns semantically and visually interpretable disentangled representations.Comment: Accepted to the Thirthy-Fourth AAAI Conference On Artificial Intelligence (AAAI), 202