Automating image analysis by annotating landmarks with deep neural networks

Image and video analysis is often a crucial step in the study of animal behavior and kinematics. Often these analyses require that the position of one or more animal landmarks are annotated (marked) in numerous images. The process of annotating landmarks can require a significant amount of time and tedious labor, which motivates the need for algorithms that can automatically annotate landmarks. In the community of scientists that use image and video analysis to study the 3D flight of animals, there has been a trend of developing more automated approaches for annotating landmarks, yet they fall short of being generally applicable. Inspired by the success of Deep Neural Networks (DNNs) on many problems in the field of computer vision, we investigate how suitable DNNs are for accurate and automatic annotation of landmarks in video datasets representative of those collected by scientists studying animals. Our work shows, through extensive experimentation on videos of hawkmoths, that DNNs are suitable for automatic and accurate landmark localization. In particular, we show that one of our proposed DNNs is more accurate than the current best algorithm for automatic localization of landmarks on hawkmoth videos. Moreover, we demonstrate how these annotations can be used to quantitatively analyze the 3D flight of a hawkmoth. To facilitate the use of DNNs by scientists from many different fields, we provide a self contained explanation of what DNNs are, how they work, and how to apply them to other datasets using the freely available library Caffe and supplemental code that we provide.https://arxiv.org/abs/1702.00583Published versio

Discovering useful parts for pose estimation in sparsely annotated datasets

Our work introduces a novel way to increase pose estimation accuracy by discovering parts from unannotated regions of training images. Discovered parts are used to generate more accurate appearance likelihoods for traditional part-based models like Pictorial Structures and its derivatives. Our experiments on images of a hawkmoth in flight show that our proposed approach significantly improves over existing work for this application, while also being more generally applicable. Our proposed approach localizes landmarks at least twice as accurately as a baseline based on a Mixture of Pictorial Structures (MPS) model. Our unique High-Resolution Moth Flight (HRMF) dataset is made publicly available with annotations.https://arxiv.org/abs/1605.00707Accepted manuscrip

Damping in flapping flight and its implications for manoeuvring, scaling and evolution

Flying animals exhibit remarkable degrees of both stability and manoeuvrability. Our understanding of these capabilities has recently been improved by the identification of a source of passive damping specific to flapping flight. Examining how this damping effect scales among different species and how it affects active manoeuvres as well as recovery from perturbations provides general insights into the flight of insects, birds and bats. These new damping models offer a means to predict manoeuvrability and stability for a wide variety of flying animals using prior reports of the morphology and flapping motions of these species. Furthermore, the presence of passive damping is likely to have facilitated the evolution of powered flight in animals by providing a stability benefit associated with flapping

Direct lateral maneuvers in hawkmoths

ABSTRACTWe used videography to investigate direct lateral maneuvers, i.e. ‘sideslips’, of the hawkmoth Manduca sexta. M. sexta sideslip by rolling their entire body and wings to reorient their net force vector. During sideslip they increase net aerodynamic force by flapping with greater amplitude, (in both wing elevation and sweep), allowing them to continue to support body weight while rolled. To execute the roll maneuver we observed in sideslips, they use an asymmetric wing stroke; increasing the pitch of the roll-contralateral wing pair, while decreasing that of the roll-ipsilateral pair. They also increase the wing sweep amplitude of, and decrease the elevation amplitude of, the contralateral wing pair relative to the ipsilateral pair. The roll maneuver unfolds in a stairstep manner, with orientation changing more during downstroke than upstroke. This is due to smaller upstroke wing pitch angle asymmetries as well as increased upstroke flapping counter-torque from left-right differences in global reference frame wing velocity about the moth's roll axis. Rolls are also opposed by stabilizing aerodynamic moments from lateral motion, such that rightward roll velocity will be opposed by rightward motion. Computational modeling using blade-element approaches confirm the plausibility of a causal linkage between the previously mentioned wing kinematics and roll/sideslip. Model results also predict high degrees of axial and lateral damping. On the time scale of whole and half wing strokes, left-right wing pair asymmetries directly relate to the first, but not second, derivative of roll. Collectively, these results strongly support a roll-based sideslip with a high degree of roll damping in M. sexta.Summary: We show that hawkmoths fly sideways by rolling in the direction of movement, adding a left- or right-ward component to their net lift vector. The underlying roll maneuvers are produced from a suite of asymmetric wing kinematic changes and are heavily damped

Functional Morphology of Gliding Flight I. Modeling Reveals Distinct Performance Landscapes Based on Soaring Strategies

The physics of flight influences the morphology of bird wings through natural selection on flight performance. The connection between wing morphology and performance is unclear due to the complex relationships between various parameters of flight. In order to better understand this connection, we present a holistic analysis of gliding flight that preserves complex relationships between parameters. We use a computational model of gliding flight, along with analysis by uncertainty quantification, to 1) create performance landscapes of gliding based on output metrics (maximum lift-to-drag ratio, minimum gliding angle, minimum sinking speed, lift coefficient at minimum sinking speed); and 2) predict what parameters of flight (chordwise camber, wing aspect ratio, Reynolds number) would differ between gliding and non-gliding species of birds. We also examine performance based on soaring strategy for possible differences in morphology within gliding birds. Gliding birds likely have greater aspect ratios than non-gliding birds, due the high sensitivity of aspect ratio on most metrics of gliding performance. Furthermore, gliding birds can use two distinct soaring strategies based on performance landscapes. First, maximizing distance traveled (maximizing lift-to-drag ratio and minimizing gliding angle) should result in wings with high aspect ratios and middling-to-low wing chordwise camber. Second, maximizing lift extracted from updrafts should result in wings with middling aspect ratios and high wing chordwise camber. Following studies can test these hypotheses using morphological measurements

Functional Morphology of Gliding Flight II. Morphology Follows Predictions of Gliding Performance

The evolution of wing morphology among birds, and its functional consequences, remains an open question, despite much attention. This is in part because the connection between form and function is difficult to test directly. To address this deficit, in prior work we used computational modeling and sensitivity analysis to interrogate the impact of altering wing aspect ratio, camber, and Reynolds number on aerodynamic performance, revealing the performance landscapes that avian evolution has explored. In the present work, we used a dataset of three-dimensionally scanned bird wings coupled with the performance landscapes to test two hypotheses regarding the evolutionary diversification of wing morphology associated with gliding flight behavior: 1) gliding birds would exhibit higher wing aspect ratio and greater chordwise camber than their non-gliding counterparts; and 2) that two strategies for gliding flight exist, with divergent morphological conformations. In support of our first hypothesis, we found evidence of morphological divergence in both wing aspect ratio and camber between gliders and non-gliders, suggesting that wing morphology of birds that utilize gliding flight is under different selective pressures than the wings of non-gliding taxa. Furthermore, we found that these morphological differences also yielded differences in coefficient of lift measured both at the maximum lift to drag ratio and at minimum sinking speed, with gliding taxa exhibiting higher coefficient of lift in both cases. Minimum sinking speed was also lower in gliders than non-gliders. However, contrary to our hypothesis, we found that the maximum ratio of the coefficient of lift to the coefficient of drag differed between gliders and non-gliders. This may point to the need for gliders to maintain high lift capability for takeoff and landing independent of gliding performance, or could be due to the divergence in flight styles among gliders, as not all gliders are predicted to optimize either quantity. However, direct evidence for the existence of two morphologically defined gliding flight strategies was equivocal, with only slightly stronger support for an evolutionary model positing separate morphological optima for these strategies than an alternative model positing a single peak. The absence of a clear result may be an artifact of low statistical power owing to a relatively small sample size of gliding flyers expected to follow the “aerial search” strategy

Feeding rate in adult Manduca sexta is unaffected by proboscis submersion depth

Adult moths from framily Spingidae (i.e. hawkmoths or sphinx moths) commonly feed on flower nectar through an extended proboscis, often several centimeters in length and longer than the body of the moth. Feeding on a viscous liquid (nectar) through a long and narrow tube is a challenging fluid dynamic problem and the subject of long-running scientific investigation. Here we characterized the relationship between proboscis submergence depth and nectar drinking rate in Manduca sexta hawkmoths. Video recordings of moth feeding bouts were collected and neural networks were used to extract data by object localization, tracking the location of the nectar meniscus and moths’ proboscis tips. We found that although feeding rates vary among bouts, the variation was not associated with proboscis submergence depth. These results show that despite the theoretical possibility of fluid uptake through the walls of the proboscis, such effects do not have a substantial effect on nectar uptake rate, and suggest that nectar must traverse the full length of the proboscis

Flight Mechanics and Control of Escape Manoeuvres in Hummingbirds. I. Flight Kinematics

Hummingbirds are nature’s masters of aerobatic manoeuvres. Previous research shows that hummingbirds and insects converged evolutionarily upon similar aerodynamic mechanisms and kinematics in hovering. Herein, we use three-dimensional kinematic data to begin to test for similar convergence of kinematics used for escape flight and to explore the effects of body size upon manoeuvring. We studied four hummingbird species in North America including two large species (magnificent hummingbird, Eugenes fulgens, 7.8 g, and blue-throated hummingbird, Lampornis clemenciae, 8.0 g) and two smaller species (broad-billed hummingbird, Cynanthus latirostris, 3.4 g, and black-chinned hummingbirds Archilochus alexandri, 3.1 g). Starting from a steady hover, hummingbirds consistently manoeuvred away from perceived threats using a drastic escape response that featured body pitch and roll rotations coupled with a large linear acceleration. Hummingbirds changed their flapping frequency and wing trajectory in all three degrees of freedom on a stroke-by-stroke basis, likely causing rapid and significant alteration of the magnitude and direction of aerodynamic forces. Thus it appears that the flight control of hummingbirds does not obey the ‘helicopter model’ that is valid for similar escape manoeuvres in fruit flies. Except for broad-billed hummingbirds, the hummingbirds had faster reaction times than those reported for visual feedback control in insects. The two larger hummingbird species performed pitch rotations and global-yaw turns with considerably larger magnitude than the smaller species, but roll rates and cumulative roll angles were similar among the four species

The mechanics and control of pitching manoeuvres in a freely flying hawkmoth (Manduca sexta)

Insects produce a variety of exquisitely controlled manoeuvres during natural flight behaviour. Here we show how hawkmoths produce and control one such manoeuvre, an avoidance response consisting of rapid pitching up, rearward flight, pitching down (often past the original pitch angle), and then pitching up slowly to equilibrium. We triggered these manoeuvre