VNect: Real-time 3D Human Pose Estimation with a Single RGB Camera

We present the first real-time method to capture the full global 3D skeletal pose of a human in a stable, temporally consistent manner using a single RGB camera. Our method combines a new convolutional neural network (CNN) based pose regressor with kinematic skeleton fitting. Our novel fully-convolutional pose formulation regresses 2D and 3D joint positions jointly in real time and does not require tightly cropped input frames. A real-time kinematic skeleton fitting method uses the CNN output to yield temporally stable 3D global pose reconstructions on the basis of a coherent kinematic skeleton. This makes our approach the first monocular RGB method usable in real-time applications such as 3D character control---thus far, the only monocular methods for such applications employed specialized RGB-D cameras. Our method's accuracy is quantitatively on par with the best offline 3D monocular RGB pose estimation methods. Our results are qualitatively comparable to, and sometimes better than, results from monocular RGB-D approaches, such as the Kinect. However, we show that our approach is more broadly applicable than RGB-D solutions, i.e. it works for outdoor scenes, community videos, and low quality commodity RGB cameras.Comment: Accepted to SIGGRAPH 201

Ultrasensitive Label-Free Nanosensing and High-Speed Tracking of Single Proteins

: Label-free detection, analysis, and rapid tracking of nanoparticles is crucial for future ultrasensitive sensing applications, ranging from understanding of biological interactions to the study of size-dependent classical-quantum transitions. Yet optical techniques to distinguish nanoparticles directly among their background remain challenging. Here we present amplified interferometric scattering microscopy (aiSCAT) as a new all-optical method capable of detecting individual nanoparticles as small as 15 kDa proteins that is equivalent to half a GFP. By balancing scattering and reflection amplitudes the interference contrast of the nanoparticle signal is amplified 1 to 2 orders of magnitude. Beyond high sensitivity, a-iSCAT allows high-speed image acquisition exceeding several hundreds of frames-per-second. We showcase the performance of our approach by detecting single Streptavidin binding events and by tracking single Ferritin proteins at 400 frames-per-second with 12 nm localization precision over seconds. Moreover, due to its extremely simple experimental realization, this advancement finally enables a cheap and routine implementation of label-free all-optical single nanoparticle detection platforms with sensitivity operating at the single protein level.Peer ReviewedPostprint (author's final draft

Single NanoParticle Photothermal Tracking (SNaPT) of 5 nm gold beads in live cells

Tracking individual nano-objets in live cells during arbitrary long times is an ubiquitous need in modern biology. We present here a method for tracking individual 5 nm gold nanoparticles on live cells. It relies on the photothermal effect and the detection of the Laser Induced Scattering around a NanoAbsorber (LISNA). The key point for recording trajectories at video rate is the use of a triangulation procedure. The effectiveness of the method is tested against Single fluorescent Molecule Tracking in live COS7 cells on subsecond time scales. We further demonstrate recordings for several minutes of AMPA receptors trajectories on the plasma membrane of live neurons. SNaPT has the unique potential to record arbitrary long trajectory of membrane proteins using non-fluorescent nanometer sized labels