1 research outputs found
Investigating naturalistic hand movements by behavior mining in long-term video and neural recordings
Recent technological advances in brain recording and artificial intelligence
are propelling a new paradigm in neuroscience beyond the traditional controlled
experiment. Rather than focusing on cued, repeated trials, naturalistic
neuroscience studies neural processes underlying spontaneous behaviors
performed in unconstrained settings. However, analyzing such unstructured data
lacking a priori experimental design remains a significant challenge,
especially when the data is multi-modal and long-term. Here we describe an
automated approach for analyzing simultaneously recorded long-term,
naturalistic electrocorticography (ECoG) and naturalistic behavior video data.
We take a behavior-first approach to analyzing the long-term recordings. Using
a combination of computer vision, discrete latent-variable modeling, and string
pattern-matching on the behavioral video data, we find and annotate spontaneous
human upper-limb movement events. We show results from our approach applied to
data collected for 12 human subjects over 7--9 days for each subject. Our
pipeline discovers and annotates over 40,000 instances of naturalistic human
upper-limb movement events in the behavioral videos. Analysis of the
simultaneously recorded brain data reveals neural signatures of movement that
corroborate prior findings from traditional controlled experiments. We also
prototype a decoder for a movement initiation detection task to demonstrate the
efficacy of our pipeline as a source of training data for brain-computer
interfacing applications. Our work addresses the unique data analysis
challenges in studying naturalistic human behaviors, and contributes methods
that may generalize to other neural recording modalities beyond ECoG. We
publicly release our curated dataset, providing a resource to study
naturalistic neural and behavioral variability at a scale not previously
available