Visual tracking of small animals in cluttered natural environments using a freely moving camera

Image-based tracking of animals in their natural habitats can provide rich behavioural data, but is very challenging due to complex and dynamic background and target appearances. We present an effective method to recover the positions of terrestrial animals in cluttered environments from video sequences filmed using a freely moving monocular camera. The method uses residual motion cues to detect the targets and is thus robust to different lighting conditions and requires no a-priori appearance model of the animal or environment. The detection is globally optimised based on an inference problem formulation using factor graphs. This handles ambiguities such as occlusions and intersections and provides automatic initialisation. Furthermore, this formulation allows a seamless integration of occasional user input for the most difficult situations, so that the effect of a few manual position estimates are smoothly distributed over long sequences. Testing our system against a benchmark dataset featuring small targets in natural scenes, we obtain 96% accuracy for fully automated tracking. We also demonstrate reliable tracking in a new data set that includes different targets (insects, vertebrates or artificial objects) in a variety of environments (desert, jungle, meadows, urban) using different imaging devices (day / night vision cameras, smart phones) and modalities (stationary, hand-held, drone operated)

Towards image-based animal tracking in natural environments using a freely moving camera

Background Image-based tracking of individual animals can provide rich data to underpin breakthroughs in biological and medical research, but few if any existing methods extend to tracking unconstrained natural behaviour in the field. New method We have developed a visual tracking system for animals filmed with a freely moving hand-held or drone-operated camera in their natural environment. This exploits a global inference method for detecting motion of an animal against a cluttered background. Trajectories are then generated by a novel video key-frame selection scheme in combination with a geometrically constrained image stitching algorithm, resulting in a two-dimensional panorama image of the environment on which the dense animal path is displayed. Results By introducing a minimal and plausible set of constraints regarding the camera orientation and movement, we demonstrate that both per-frame animal positions and overall trajectories can be extracted with reasonable accuracy, for a range of different animals, environments and imaging modalities. Comparison Our method requires only a single uncalibrated camera, does not require marking or training data to detect the animal, and makes no prior assumptions about appearance of the target or background. In particular it can detect targets occupying fewer than 20 pixels in the image, and deal with poor contrast, highly dynamic lighting and frequent occlusion. Conclusion Our algorithm produces highly informative qualitative trajectories embedded in a panorama of the environment. The results are still subject to rotational drift and additional scaling routines would be needed to obtain absolute real-world coordinates. It nevertheless provides a flexible and easy-to-use system to obtain rich data on natural animal behaviour in the field

SALSA: A Novel Dataset for Multimodal Group Behavior Analysis

Studying free-standing conversational groups (FCGs) in unstructured social settings (e.g., cocktail party ) is gratifying due to the wealth of information available at the group (mining social networks) and individual (recognizing native behavioral and personality traits) levels. However, analyzing social scenes involving FCGs is also highly challenging due to the difficulty in extracting behavioral cues such as target locations, their speaking activity and head/body pose due to crowdedness and presence of extreme occlusions. To this end, we propose SALSA, a novel dataset facilitating multimodal and Synergetic sociAL Scene Analysis, and make two main contributions to research on automated social interaction analysis: (1) SALSA records social interactions among 18 participants in a natural, indoor environment for over 60 minutes, under the poster presentation and cocktail party contexts presenting difficulties in the form of low-resolution images, lighting variations, numerous occlusions, reverberations and interfering sound sources; (2) To alleviate these problems we facilitate multimodal analysis by recording the social interplay using four static surveillance cameras and sociometric badges worn by each participant, comprising the microphone, accelerometer, bluetooth and infrared sensors. In addition to raw data, we also provide annotations concerning individuals' personality as well as their position, head, body orientation and F-formation information over the entire event duration. Through extensive experiments with state-of-the-art approaches, we show (a) the limitations of current methods and (b) how the recorded multiple cues synergetically aid automatic analysis of social interactions. SALSA is available at http://tev.fbk.eu/salsa.Comment: 14 pages, 11 figure

Flying Animal Inspired Behavior-Based Gap-Aiming Autonomous Flight with a Small Unmanned Rotorcraft in a Restricted Maneuverability Environment

This dissertation research shows a small unmanned rotorcraft system with onboard processing and a vision sensor can produce autonomous, collision-free flight in a restricted maneuverability environment with no a priori knowledge by using a gap-aiming behavior inspired by flying animals. Current approaches to autonomous flight with small unmanned aerial systems (SUAS) concentrate on detecting and explicitly avoiding obstacles. In contrast, biology indicates that birds, bats, and insects do the opposite; they react to open spaces, or gaps in the environment, with a gap_aiming behavior. Using flying animals as inspiration a behavior-based robotics approach is taken to implement and test their observed gap-aiming behavior in three dimensions. Because biological studies were unclear whether the flying animals were reacting to the largest gap perceived, the closest gap perceived, or all of the gaps three approaches for the perceptual schema were explored in simulation: detect_closest_gap, detect_largest_gap, and detect_all_gaps. The result of these simulations was used in a proof-of-concept implementation on a 3DRobotics Solo quadrotor platform in an environment designed to represent the navigational diffi- culties found inside a restricted maneuverability environment. The motor schema is implemented with an artificial potential field to produce the action of aiming to the center of the gap. Through two sets of field trials totaling fifteen flights conducted with a small unmanned quadrotor, the gap-aiming behavior observed in flying animals is shown to produce repeatable autonomous, collision-free flight in a restricted maneuverability environment. Additionally, using the distance from the starting location to perceived gaps, the horizontal and vertical distance traveled, and the distance from the center of the gap during traversal the implementation of the gap selection approach performs as intended, the three-dimensional movement produced by the motor schema and the accuracy of the motor schema are shown, respectively. This gap-aiming behavior provides the robotics community with the first known implementation of autonomous, collision-free flight on a small unmanned quadrotor without explicit obstacle detection and avoidance as seen with current implementations. Additionally, the testing environment described by quantitative metrics provides a benchmark for autonomous SUAS flight testing in confined environments. Finally, the success of the autonomous collision-free flight implementation on a small unmanned rotorcraft and field tested in a restricted maneuverability environment could have important societal impact in both the public and private sectors

Natural visual cues eliciting predator avoidance in fiddler crabs

To efficiently provide an animal with relevant information, the design of its visual system should reflect the distribution of natural signals and the animal’s tasks. In many behavioural contexts, however, we know comparatively little about the moment-to-moment information-processing challenges animals face in their daily lives. In predator avoidance, for instance, we lack an accurate description of the natural signal stream and its value for risk assessment throughout the prey’s defensive behaviour.We characterized the visual signals generated by real, potentially predatory events by video-recording bird approaches towards an Uca vomeris colony. Using four synchronized cameras allowed us to simultaneously monitor predator avoidance responses of crabs. We reconstructed the signals generated by dangerous and non-dangerous flying animals, identified the cues that triggered escape responses and compared them with those triggering responses to dummy predators. Fiddler crabs responded to a combination of multiple visual cues (including retinal speed, elevation and visual flicker) that reflect the visual signatures of distinct bird and insect behaviours. This allowed crabs to discriminate between dangerous and non-dangerous events. The results demonstrate the importance of measuring natural sensory signatures of biologically relevant events in order to understand biological information processing and its effects on behavioural organization

An Insect-Inspired Target Tracking Mechanism for Autonomous Vehicles

Target tracking is a complicated task from an engineering perspective, especially where targets are small and seen against complex natural environments. Due to the high demand for robust target tracking algorithms a great deal of research has focused on this area. However, most engineering solutions developed for this purpose are often unreliable in real world conditions or too computationally expensive to be used in real-time applications. While engineering methods try to solve the problem of target detection and tracking by using high resolution input images, fast processors, with typically computationally expensive methods, a quick glance at nature provides evidence that practical real world solutions for target tracking exist. Many animals track targets for predation, territorial or mating purposes and with millions of years of evolution behind them, it seems reasonable to assume that these solutions are highly efficient. For instance, despite their low resolution compound eyes and tiny brains, many flying insects have evolved superb abilities to track targets in visual clutter even in the presence of other distracting stimuli, such as swarms of prey and conspecifics. The accessibility of the dragonfly for stable electrophysiological recordings makes this insect an ideal and tractable model system for investigating the neuronal correlates for complex tasks such as target pursuit. Studies on dragonflies identified and characterized a set of neurons likely to mediate target detection and pursuit referred to as ‘small target motion detector’ (STMD) neurons. These neurons are selective for tiny targets, are velocity-tuned, contrast-sensitive and respond robustly to targets even against the motion of background. These neurons have shown several high-order properties which can contribute to the dragonfly’s ability to robustly pursue prey with over a 97% success rate. These include the recent electrophysiological observations of response ‘facilitation’ (a slow build-up of response to targets that move on long, continuous trajectories) and ‘selective attention’, a competitive mechanism that selects one target from alternatives. In this thesis, I adopted a bio-inspired approach to develop a solution for the problem of target tracking and pursuit. Directly inspired by recent physiological breakthroughs in understanding the insect brain, I developed a closed-loop target tracking system that uses an active saccadic gaze fixation strategy inspired by insect pursuit. First, I tested this model in virtual world simulations using MATLAB/Simulink. The results of these simulations show robust performance of this insect-inspired model, achieving high prey capture success even within complex background clutter, low contrast and high relative speed of pursued prey. Additionally, these results show that inclusion of facilitation not only substantially improves success for even short-duration pursuits, it also enhances the ability to ‘attend’ to one target in the presence of distracters. This inspect-inspired system has a relatively simple image processing strategy compared to state-of-the-art trackers developed recently for computer vision applications. Traditional machine vision approaches incorporate elaborations to handle challenges and non-idealities in the natural environments such as local flicker and illumination changes, and non-smooth and non-linear target trajectories. Therefore, the question arises as whether this insect inspired tracker can match their performance when given similar challenges? I investigated this question by testing both the efficacy and efficiency of this insect-inspired model in open-loop, using a widely-used set of videos recorded under natural conditions. I directly compared the performance of this model with several state-of-the-art engineering algorithms using the same hardware, software environment and stimuli. This insect-inspired model exhibits robust performance in tracking small moving targets even in very challenging natural scenarios, outperforming the best of the engineered approaches. Furthermore, it operates more efficiently compared to the other approaches, in some cases dramatically so. Computer vision literature traditionally test target tracking algorithms only in open-loop. However, one of the main purposes for developing these algorithms is implementation in real-time robotic applications. Therefore, it is still unclear how these algorithms might perform in closed-loop real-world applications where inclusion of sensors and actuators on a physical robot results in additional latency which can affect the stability of the feedback process. Additionally, studies show that animals interact with the target by changing eye or body movements, which then modulate the visual inputs underlying the detection and selection task (via closed-loop feedback). This active vision system may be a key to exploiting visual information by the simple insect brain for complex tasks such as target tracking. Therefore, I implemented this insect-inspired model along with insect active vision in a robotic platform. I tested this robotic implementation both in indoor and outdoor environments against different challenges which exist in real-world conditions such as vibration, illumination variation, and distracting stimuli. The experimental results show that the robotic implementation is capable of handling these challenges and robustly pursuing a target even in highly challenging scenarios.Thesis (Ph.D.) -- University of Adelaide, School of Mechanical Engineering, 201

Adaptations to changes in the acoustic scene of the echolocating bat

Our natural environment is noisy and in order to navigate it successfully, we must filter out the important components so that we may guide our next steps. In analyzing our acoustic scene, one of the most common challenges is to segregate speech communication sounds from background noise; this process is not unique to humans. Echolocating bats emit high frequency biosonar signals and listen to echoes returning off objects in their environment. The sound wave they receive is a merging of echoes reflecting off target prey and other scattered objects, conspecific calls and echoes, and any naturally-occurring environmental noises. The bat is faced with the challenge of segregating this complex sound wave into the components of interest to adapt its flight and echolocation behavior in response to fast and dynamic environmental changes. In this thesis, we employ two approaches to investigate the mechanisms that may aid the bat in analyzing its acoustic scene. First, we test the bat’s adaptations to changes of controlled echo-acoustic flow patterns, similar to those it may encounter when flying along forest edges and among clutter. Our findings show that big brown bats adapt their flight paths in response to the intervals between echoes, and suggest that there is a limit to how close objects can be spaced, before the bat does not represent them as distinct any longer. Further, we consider how bats that use different echolocation signals may navigate similar environments, and provide evidence of species-specific flight and echolocation adaptations. Second, we research how temporal patterning of echolocation calls is affected during competitive foraging of paired bats in open and cluttered environments. Our findings show that “silent behavior”, the ceasing of emitting echolocation calls, which had previously been proposed as a mechanism to avoid acoustic interference, or to “eavesdrop” on another bat, may not be as common as has been reported

A Neural Model of Visually Guided Steering, Obstacle Avoidance, and Route Selection

A neural model is developed to explain how humans can approach a goal object on foot while steering around obstacles to avoid collisions in a cluttered environment. The model uses optic flow from a 3D virtual reality environment to determine the position of objects based on motion discontinuities, and computes heading direction, or the direction of self-motion, from global optic flow. The cortical representation of heading interacts with the representations of a goal and obstacles such that the goal acts as an attractor of heading, while obstacles act as repellers. In addition the model maintains fixation on the goal object by generating smooth pursuit eye movements. Eye rotations can distort the optic flow field, complicating heading perception, and the model uses extraretinal signals to correct for this distortion and accurately represent heading. The model explains how motion processing mechanisms in cortical areas MT, MST, and posterior parietal cortex can be used to guide steering. The model quantitatively simulates human psychophysical data about visually-guided steering, obstacle avoidance, and route selection.Air Force Office of Scientific Research (F4960-01-1-0397); National Geospatial-Intelligence Agency (NMA201-01-1-2016); National Science Foundation (SBE-0354378); Office of Naval Research (N00014-01-1-0624