Generalization and Estimation Error Bounds for Model-based Neural Networks

Model-based neural networks provide unparalleled performance for various tasks, such as sparse coding and compressed sensing problems. Due to the strong connection with the sensing model, these networks are interpretable and inherit prior structure of the problem. In practice, model-based neural networks exhibit higher generalization capability compared to ReLU neural networks. However, this phenomenon was not addressed theoretically. Here, we leverage complexity measures including the global and local Rademacher complexities, in order to provide upper bounds on the generalization and estimation errors of model-based networks. We show that the generalization abilities of model-based networks for sparse recovery outperform those of regular ReLU networks, and derive practical design rules that allow to construct model-based networks with guaranteed high generalization. We demonstrate through a series of experiments that our theoretical insights shed light on a few behaviours experienced in practice, including the fact that ISTA and ADMM networks exhibit higher generalization abilities (especially for small number of training samples), compared to ReLU networks

Identification of Non-Linear RF Systems Using Backpropagation

In this work, we use deep unfolding to view cascaded non-linear RF systems as model-based neural networks. This view enables the direct use of a wide range of neural network tools and optimizers to efficiently identify such cascaded models. We demonstrate the effectiveness of this approach through the example of digital self-interference cancellation in full-duplex communications where an IQ imbalance model and a non-linear PA model are cascaded in series. For a self-interference cancellation performance of approximately 44.5 dB, the number of model parameters can be reduced by 74% and the number of operations per sample can be reduced by 79% compared to an expanded linear-in-parameters polynomial model.Comment: To be presented at the 2020 IEEE International Conference on Communications (Workshop on Full-Duplex Communications for Future Wireless Networks

Introducing Nonuniform Sparse Proximal Averaging Network for Seismic Reflectivity Inversion

We consider the problem of seismic reflectivity inversion, which pertains to the high-resolution recovery of interface locations and reflection coefficients from seismic measurements, which are vital for estimating the subsurface structure. We develop two model-based neural networks within the framework of deep-unfolding . First, we propose a nonuniform minimax concave penalty regularized formulation for reflectivity inversion and unfold the resulting iterative algorithm into a network. Second, we propose a nonuniform sparse model that relies on a combination of regularizers (composite regularization) and develop the nonuniform sparse proximal averaging network (NuSPAN). We demonstrate the efficacy of the proposed approaches over the benchmark techniques through numerical experiments on synthetic 1-D seismic traces and 2-D wedge models. We also report validations on the 2-D Marmousi2 simulated model and 3-D real field measurements from the Penobscot 3D survey off the coast of Nova Scotia, Canada. The accuracy of the proposed approaches is higher than the state-of-the-art techniques in terms of amplitude and support recovery. Further, for Marmousi2, the proposed deep-unfolding networks result in 600× faster inference than the fast iterative shrinkage-thresholding algorithm (FISTA). In terms of combined training and inference times, the learned iterative shrinkage-thresholding algorithm (LISTA) is the fastest. The inference speed-up is significant given that the volume of data is typically large in seismic signal processing

Non-Linear Self-Interference Cancellation via Tensor Completion

Non-linear self-interference (SI) cancellation constitutes a fundamental problem in full-duplex communications, which is typically tackled using either polynomial models or neural networks. In this work, we explore the applicability of a recently proposed method based on low-rank tensor completion, called canonical system identification (CSID), to non-linear SI cancellation. Our results show that CSID is very effective in modeling and cancelling the non-linear SI signal and can have lower computational complexity than existing methods, albeit at the cost of increased memory requirements.Comment: To be presented at the 2020 Asilomar Conference for Signals, Systems, and Computer

Comparing Semi-Parametric Model Learning Algorithms for Dynamic Model Estimation in Robotics

Physical modeling of robotic system behavior is the foundation for controlling many robotic mechanisms to a satisfactory degree. Mechanisms are also typically designed in a way that good model accuracy can be achieved with relatively simple models and model identification strategies. If the modeling accuracy using physically based models is not enough or too complex, model-free methods based on machine learning techniques can help. Of particular interest to us was therefore the question to what degree semi-parametric modeling techniques, meaning combinations of physical models with machine learning, increase the modeling accuracy of inverse dynamics models which are typically used in robot control. To this end, we evaluated semi-parametric Gaussian process regression and a novel model-based neural network architecture, and compared their modeling accuracy to a series of naive semi-parametric, parametric-only and non-parametric-only regression methods. The comparison has been carried out on three test scenarios, one involving a real test-bed and two involving simulated scenarios, with the most complex scenario targeting the modeling a simulated robot's inverse dynamics model. We found that in all but one case, semi-parametric Gaussian process regression yields the most accurate models, also with little tuning required for the training procedure