Camera Focus Adjustment Using Depth Estimated via Ultra-wideband (UWB) Handshake

Many video conferencing applications support portrait mode where the participant is in focus and their surroundings are blurred. Portrait mode based on computer vision techniques can sometimes be unsatisfactory due to difficulties in separating the user and the background. This disclosure describes techniques that use existing ultra-wideband (UWB) hardware on commodity devices to perform crisp and more accurate segmentation. A UWB handshake protocol is utilized to estimate the distance and angle between the camera and another device that the conference participant is wearing. The estimate is used to automatically adjust camera focal length to focus on the conference participant while blurring other objects. The techniques can make high-resolution portrait mode video conferencing affordable for users with commodity devices

Recurrent Segmentation for Variable Computational Budgets

State-of-the-art systems for semantic image segmentation use feed-forward pipelines with fixed computational costs. Building an image segmentation system that works across a range of computational budgets is challenging and time-intensive as new architectures must be designed and trained for every computational setting. To address this problem we develop a recurrent neural network that successively improves prediction quality with each iteration. Importantly, the RNN may be deployed across a range of computational budgets by merely running the model for a variable number of iterations. We find that this architecture is uniquely suited for efficiently segmenting videos. By exploiting the segmentation of past frames, the RNN can perform video segmentation at similar quality but reduced computational cost compared to state-of-the-art image segmentation methods. When applied to static images in the PASCAL VOC 2012 and Cityscapes segmentation datasets, the RNN traces out a speed-accuracy curve that saturates near the performance of state-of-the-art segmentation methods

BGGAN: Bokeh-Glass Generative Adversarial Network for Rendering Realistic Bokeh

A photo captured with bokeh effect often means objects in focus are sharp while the out-of-focus areas are all blurred. DSLR can easily render this kind of effect naturally. However, due to the limitation of sensors, smartphones cannot capture images with depth-of-field effects directly. In this paper, we propose a novel generator called Glass-Net, which generates bokeh images not relying on complex hardware. Meanwhile, the GAN-based method and perceptual loss are combined for rendering a realistic bokeh effect in the stage of finetuning the model. Moreover, Instance Normalization(IN) is reimplemented in our network, which ensures our tflite model with IN can be accelerated on smartphone GPU. Experiments show that our method is able to render a high-quality bokeh effect and process one $1024 \times 1536$ pixel image in 1.9 seconds on all smartphone chipsets. This approach ranked First in AIM 2020 Rendering Realistic Bokeh Challenge Track 1 \& Track 2.Comment: accepted by ECCV workshop 202