Activity-conditioned continuous human pose estimation for performance analysis of athletes using the example of swimming

In this paper we consider the problem of human pose estimation in real-world videos of swimmers. Swimming channels allow filming swimmers simultaneously above and below the water surface with a single stationary camera. These recordings can be used to quantitatively assess the athletes' performance. The quantitative evaluation, so far, requires manual annotations of body parts in each video frame. We therefore apply the concept of CNNs in order to automatically infer the required pose information. Starting with an off-the-shelf architecture, we develop extensions to leverage activity information - in our case the swimming style of an athlete - and the continuous nature of the video recordings. Our main contributions are threefold: (a) We apply and evaluate a fine-tuned Convolutional Pose Machine architecture as a baseline in our very challenging aquatic environment and discuss its error modes, (b) we propose an extension to input swimming style information into the fully convolutional architecture and (c) modify the architecture for continuous pose estimation in videos. With these additions we achieve reliable pose estimates with up to +16% more correct body joint detections compared to the baseline architecture.Comment: 10 pages, 9 figures, accepted at WACV 201

Automated classification of three-dimensional reconstructions of coral reefs using convolutional neural networks

© The Author(s), 2020. This article is distributed under the terms of the Creative Commons Attribution License. The definitive version was published in Hopkinson, B. M., King, A. C., Owen, D. P., Johnson-Roberson, M., Long, M. H., & Bhandarkar, S. M. Automated classification of three-dimensional reconstructions of coral reefs using convolutional neural networks. PLoS One, 15(3), (2020): e0230671, doi: 10.1371/journal.pone.0230671.Coral reefs are biologically diverse and structurally complex ecosystems, which have been severally affected by human actions. Consequently, there is a need for rapid ecological assessment of coral reefs, but current approaches require time consuming manual analysis, either during a dive survey or on images collected during a survey. Reef structural complexity is essential for ecological function but is challenging to measure and often relegated to simple metrics such as rugosity. Recent advances in computer vision and machine learning offer the potential to alleviate some of these limitations. We developed an approach to automatically classify 3D reconstructions of reef sections and assessed the accuracy of this approach. 3D reconstructions of reef sections were generated using commercial Structure-from-Motion software with images extracted from video surveys. To generate a 3D classified map, locations on the 3D reconstruction were mapped back into the original images to extract multiple views of the location. Several approaches were tested to merge information from multiple views of a point into a single classification, all of which used convolutional neural networks to classify or extract features from the images, but differ in the strategy employed for merging information. Approaches to merging information entailed voting, probability averaging, and a learned neural-network layer. All approaches performed similarly achieving overall classification accuracies of ~96% and >90% accuracy on most classes. With this high classification accuracy, these approaches are suitable for many ecological applications.This study was funded by grants from the Alfred P. Sloan Foundation (BMH, BR2014-049; https://sloan.org), and the National Science Foundation (MHL, OCE-1657727; https://www.nsf.gov). The funders had no role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript

Key-Pose Prediction in Cyclic Human Motion

In this paper we study the problem of estimating innercyclic time intervals within repetitive motion sequences of top-class swimmers in a swimming channel. Interval limits are given by temporal occurrences of key-poses, i.e. distinctive postures of the body. A key-pose is defined by means of only one or two specific features of the complete posture. It is often difficult to detect such subtle features directly. We therefore propose the following method: Given that we observe the swimmer from the side, we build a pictorial structure of poselets to robustly identify random support poses within the regular motion of a swimmer. We formulate a maximum likelihood model which predicts a key-pose given the occurrences of multiple support poses within one stroke. The maximum likelihood can be extended with prior knowledge about the temporal location of a key-pose in order to improve the prediction recall. We experimentally show that our models reliably and robustly detect key-poses with a high precision and that their performance can be improved by extending the framework with additional camera views.Comment: Accepted at WACV 2015, 8 pages, 3 figure

Now Hear This! Orientation and Behavioral Responses of Hatchling Loggerhead Sea Turtles, Caretta caretta, to Environmental Acoustic Cues

Although the visual and geologic orientation cues utilized by sea turtle hatchlings during seafinding, when they move from the nest to the sea after hatching, have been well studied, the potential for auditory stimuli to act as an orientation cue has not been well explored. Over the past several decades our knowledge of the auditory capacity of sea turtles has increased greatly, yet little is known about the biological significance of this sensory ability. To investigate whether hatchlings can use ocean sounds during seafinding, we measured the behavioral responses of hatchling loggerhead sea turtles (Caretta caretta) collected from nesting beaches in North Carolina to the presence of beach wave sound recorded on a nesting beach during the summer of 2015. The highest sound energy of beach waves occursHz, which overlaps with the most sensitive hearing range of loggerhead hatchlings (range of frequency detection: 50-1600 Hz, maximum sensitivity: 50-400 Hz). In our experiment, we placed turtles in a V-maze that isolated them from visual, vibratory, and chemical cues. One end of the V held a speaker producing beach wave sounds recorded from nesting beaches, while the other end held sound-reducing foam. We examined the phonotaxic behaviors of the hatchlings at two sound pressure levels (68 dB re: 20μPa and 64 dB re: 20μPa measured directly in front of the speaker). In the presence of the higher sound pressure level (68 dB re: 20μPa), hatchlings exhibited no phonotaxic response (p=1.0); yet, at the reduced sound pressure level (64 dB re: 20μPa), hatchlings exhibited a negative phonotaxic response (p=0.005). In control trials, hatchlings oriented to the two sides of the V-maze equally (p=0.701), suggesting the hatchlings in the lower volume treatment group were responding negatively to the sound. These results indicate the need for further auditory orientation experiments to better understand hatchling behavioral responses to environmental acoustic cues and to address possible impacts of anthropogenic beach sounds that have the potential to disorient hatchlings during seafinding

Photofocusing: Light and flow of phototactic microswimmer suspension

We explore in this paper the phenomenon of photofocusing: a coupling between flow vorticity and biased swimming of microalgae toward a light source that produces a focusing of the microswimmer suspension. We combine experiments that investigate the stationary state of this phenomenon as well as the transition regime with analytical and numerical modeling. We show that the experimentally observed scalings on the width of the focalized region and the establishment length as a function of the flow velocity are well described by a simple theoretical model