Periodic Motion Detection and Estimation via Space-Time Sampling

A novel technique to detect and localize periodic movements in video is presented. The distinctive feature of the technique is that it requires neither feature tracking nor object segmentation. Intensity patterns along linear sample paths in space-time are used in estimation of period of object motion in a given sequence of frames. Sample paths are obtained by connecting (in space-time) sample points from regions of high motion magnitude in the first and last frames. Oscillations in intensity values are induced at time instants when an object intersects the sample path. The locations of peaks in intensity are determined by parameters of both cyclic object motion and orientation of the sample path with respect to object motion. The information about peaks is used in a least squares framework to obtain an initial estimate of these parameters. The estimate is further refined using the full intensity profile. The best estimate for the period of cyclic object motion is obtained by looking for consensus among estimates from many sample paths. The proposed technique is evaluated with synthetic videos where ground-truth is known, and with American Sign Language videos where the goal is to detect periodic hand motions.National Science Foundation (CNS-0202067, IIS-0308213, IIS-0329009); Office of Naval Research (N00014-03-1-0108

The molecular architecture of engulfment during Bacillus subtilis sporulation.

The study of bacterial cell biology is limited by difficulties in visualizing cellular structures at high spatial resolution within their native milieu. Here, we visualize Bacillus subtilis sporulation using cryo-electron tomography coupled with cryo-focused ion beam milling, allowing the reconstruction of native-state cellular sections at molecular resolution. During sporulation, an asymmetrically-positioned septum generates a larger mother cell and a smaller forespore. Subsequently, the mother cell engulfs the forespore. We show that the septal peptidoglycan is not completely degraded at the onset of engulfment. Instead, the septum is uniformly and only slightly thinned as it curves towards the mother cell. Then, the mother cell membrane migrates around the forespore in tiny finger-like projections, whose formation requires the mother cell SpoIIDMP protein complex. We propose that a limited number of SpoIIDMP complexes tether to and degrade the peptidoglycan ahead of the engulfing membrane, generating an irregular membrane front

Visual Speech Enhancement

When video is shot in noisy environment, the voice of a speaker seen in the video can be enhanced using the visible mouth movements, reducing background noise. While most existing methods use audio-only inputs, improved performance is obtained with our visual speech enhancement, based on an audio-visual neural network. We include in the training data videos to which we added the voice of the target speaker as background noise. Since the audio input is not sufficient to separate the voice of a speaker from his own voice, the trained model better exploits the visual input and generalizes well to different noise types. The proposed model outperforms prior audio visual methods on two public lipreading datasets. It is also the first to be demonstrated on a dataset not designed for lipreading, such as the weekly addresses of Barack Obama.Comment: Accepted to Interspeech 2018. Supplementary video: https://www.youtube.com/watch?v=nyYarDGpcY

Real-time 3D analysis during electron tomography using tomviz

The demand for high-throughput electron tomography is rapidly increasing in biological and material sciences. However, this 3D imaging technique is computationally bottlenecked by alignment and reconstruction which runs from hours to days. We demonstrate real-time tomography with dynamic 3D tomographic visualization to enable rapid interpretation of specimen structure immediately as data is collected on an electron microscope. Using geometrically complex chiral nanoparticles, we show volumetric interpretation can begin in less than 10 minutes and a high quality tomogram is available within 30 minutes. Real time tomography is integrated into tomviz, an open source and cross platform 3D analysis tool that contains intuitive graphical user interfaces (GUI) to enable any scientist to characterize biological and material structure in 3D