Learning Robust Deep Equilibrium Models

Deep equilibrium (DEQ) models have emerged as a promising class of implicit layer models in deep learning, which abandon traditional depth by solving for the fixed points of a single nonlinear layer. Despite their success, the stability of the fixed points for these models remains poorly understood. Recently, Lyapunov theory has been applied to Neural ODEs, another type of implicit layer model, to confer adversarial robustness. By considering DEQ models as nonlinear dynamic systems, we propose a robust DEQ model named LyaDEQ with guaranteed provable stability via Lyapunov theory. The crux of our method is ensuring the fixed points of the DEQ models are Lyapunov stable, which enables the LyaDEQ models to resist minor initial perturbations. To avoid poor adversarial defense due to Lyapunov-stable fixed points being located near each other, we add an orthogonal fully connected layer after the Lyapunov stability module to separate different fixed points. We evaluate LyaDEQ models on several widely used datasets under well-known adversarial attacks, and experimental results demonstrate significant improvement in robustness. Furthermore, we show that the LyaDEQ model can be combined with other defense methods, such as adversarial training, to achieve even better adversarial robustness

Deep Equilibrium Models Meet Federated Learning

In this study the problem of Federated Learning (FL) is explored under a new perspective by utilizing the Deep Equilibrium (DEQ) models instead of conventional deep learning networks. We claim that incorporating DEQ models into the federated learning framework naturally addresses several open problems in FL, such as the communication overhead due to the sharing large models and the ability to incorporate heterogeneous edge devices with significantly different computation capabilities. Additionally, a weighted average fusion rule is proposed at the server-side of the FL framework to account for the different qualities of models from heterogeneous edge devices. To the best of our knowledge, this study is the first to establish a connection between DEQ models and federated learning, contributing to the development of an efficient and effective FL framework. Finally, promising initial experimental results are presented, demonstrating the potential of this approach in addressing challenges of FL.Comment: The paper has been accepted for publication in European Signal Processing Conference, Eusipco 202

Non-uniqueness of deep parameters and shocks in estimated DSGE models: a health warning

Estimation of dynamic stochastic general equilibrium (DSGE)models using state space methods implies vector autoregressive moving average (VARMA)representations of the observables. Following Lippi and Reichlin’s (1994)analysis of nonfundamentalness, this note highlights the potential dangers of end of non-uniqueness, both of estimates of deep parameters and of structural innovations

TorchDEQ: A Library for Deep Equilibrium Models

Deep Equilibrium (DEQ) Models, an emerging class of implicit models that maps inputs to fixed points of neural networks, are of growing interest in the deep learning community. However, training and applying DEQ models is currently done in an ad-hoc fashion, with various techniques spread across the literature. In this work, we systematically revisit DEQs and present TorchDEQ, an out-of-the-box PyTorch-based library that allows users to define, train, and infer using DEQs over multiple domains with minimal code and best practices. Using TorchDEQ, we build a ``DEQ Zoo'' that supports six published implicit models across different domains. By developing a joint framework that incorporates the best practices across all models, we have substantially improved the performance, training stability, and efficiency of DEQs on ten datasets across all six projects in the DEQ Zoo. TorchDEQ and DEQ Zoo are released as \href{https://github.com/locuslab/torchdeq}{open source}

Deep Equilibrium Nets

We introduce deep equilibrium nets (DEQNs)—a deep learning‐based method to compute approximate functional rational expectations equilibria of economic models featuring a significant amount of heterogeneity, uncertainty, and occasionally binding constraints. DEQNs are neural networks trained in an unsupervised fashion to satisfy all equilibrium conditions along simulated paths of the economy. Since DEQNs approximate the equilibrium functions directly, simulating the economy is computationally cheap, and training data can be generated at virtually zero cost. We demonstrate that DEQNs can accurately solve economically relevant models by applying them to two challenging life‐cycle models and a Bewley‐style model with aggregate risk

Spatial distribution of local tunneling conductivity due to interference and Coulomb interaction effects for deep and shallow impurities on semiconductor surfaces

Spatial distribution of local tunneling conductivity was investigated for deep and shallow impurities on semiconductor surfaces. Non-equilibrium Coulomb interaction and interference effects were taken into account and analyzed theoretically with the help of Keldysh formalism. Two models were investigated: mean field self-consistent approach for shallow impurity state and Hubbard-{I} model for deep impurity state. We have found that not only above the impurity but also at the distances comparable to the lattice period both effects interference between direct and resonant tunneling channels and on-site Coulomb repulsion of localized electrons strongly modifies form of tunneling conductivity measured by the scanning tunneling microscopy/spectroscopy (STM/STS).Comment: 5 pages, 3 figure