Mixed Neural Voxels for Fast Multi-view Video Synthesis

Synthesizing high-fidelity videos from real-world multi-view input is challenging because of the complexities of real-world environments and highly dynamic motions. Previous works based on neural radiance fields have demonstrated high-quality reconstructions of dynamic scenes. However, training such models on real-world scenes is time-consuming, usually taking days or weeks. In this paper, we present a novel method named MixVoxels to better represent the dynamic scenes with fast training speed and competitive rendering qualities. The proposed MixVoxels represents the 4D dynamic scenes as a mixture of static and dynamic voxels and processes them with different networks. In this way, the computation of the required modalities for static voxels can be processed by a lightweight model, which essentially reduces the amount of computation, especially for many daily dynamic scenes dominated by the static background. To separate the two kinds of voxels, we propose a novel variation field to estimate the temporal variance of each voxel. For the dynamic voxels, we design an inner-product time query method to efficiently query multiple time steps, which is essential to recover the high-dynamic motions. As a result, with 15 minutes of training for dynamic scenes with inputs of 300-frame videos, MixVoxels achieves better PSNR than previous methods. Codes and trained models are available at https://github.com/fengres/mixvoxelsComment: ICCV 2023 (Oral

Deep cascade gradient RBF networks with output-relevant feature extraction and adaptation for nonlinear and nonstationary processes

The main challenge for industrial predictive models is how to effectively deal with big data from high-dimensional processes with nonstationary characteristics. Although deep networks, such as the stacked autoencoder (SAE), can learn useful features from massive data with multilevel architecture, it is difficult to adapt them online to track fast time-varying process dynamics. To integrate feature learning and online adaptation, this paper proposes a deep cascade gradient radial basis function (GRBF) network for online modeling and prediction of nonlinear and nonstationary processes. The proposed deep learning method consists of three modules. First, a preliminary prediction result is generated by a GRBF weak predictor, which is further combined with raw input data for feature extraction. By incorporating the prior weak prediction information, deep output-relevant features are extracted using a SAE. Online prediction is finally produced upon the extracted features with a GRBF predictor, whose weights and structure are updated online to capture fast time-varying process characteristics. Three real-world industrial case studies demonstrate that the proposed deep cascade GRBF network outperforms existing state-of-the-art online modeling approaches as well as deep networks, in terms of both online prediction accuracy and computational complexity