248 research outputs found
MgNO: Efficient Parameterization of Linear Operators via Multigrid
In this work, we propose a concise neural operator architecture for operator
learning. Drawing an analogy with a conventional fully connected neural
network, we define the neural operator as follows: the output of the -th
neuron in a nonlinear operator layer is defined by . Here,
denotes the bounded linear operator connecting -th input
neuron to -th output neuron, and the bias takes the form
of a function rather than a scalar. Given its new universal approximation
property, the efficient parameterization of the bounded linear operators
between two neurons (Banach spaces) plays a critical role. As a result, we
introduce MgNO, utilizing multigrid structures to parameterize these linear
operators between neurons. This approach offers both mathematical rigor and
practical expressivity. Additionally, MgNO obviates the need for conventional
lifting and projecting operators typically required in previous neural
operators. Moreover, it seamlessly accommodates diverse boundary conditions.
Our empirical observations reveal that MgNO exhibits superior ease of training
compared to other CNN-based models, while also displaying a reduced
susceptibility to overfitting when contrasted with spectral-type neural
operators. We demonstrate the efficiency and accuracy of our method with
consistently state-of-the-art performance on different types of partial
differential equations (PDEs)
Multi-Zone Unit for Recurrent Neural Networks
Recurrent neural networks (RNNs) have been widely used to deal with sequence
learning problems. The input-dependent transition function, which folds new
observations into hidden states to sequentially construct fixed-length
representations of arbitrary-length sequences, plays a critical role in RNNs.
Based on single space composition, transition functions in existing RNNs often
have difficulty in capturing complicated long-range dependencies. In this
paper, we introduce a new Multi-zone Unit (MZU) for RNNs. The key idea is to
design a transition function that is capable of modeling multiple space
composition. The MZU consists of three components: zone generation, zone
composition, and zone aggregation. Experimental results on multiple datasets of
the character-level language modeling task and the aspect-based sentiment
analysis task demonstrate the superiority of the MZU.Comment: Accepted at AAAI 202
Compressed Air Energy Storage
ústav energetik
BPKD: Boundary Privileged Knowledge Distillation For Semantic Segmentation
Current knowledge distillation approaches in semantic segmentation tend to
adopt a holistic approach that treats all spatial locations equally. However,
for dense prediction, students' predictions on edge regions are highly
uncertain due to contextual information leakage, requiring higher spatial
sensitivity knowledge than the body regions. To address this challenge, this
paper proposes a novel approach called boundary-privileged knowledge
distillation (BPKD). BPKD distills the knowledge of the teacher model's body
and edges separately to the compact student model. Specifically, we employ two
distinct loss functions: (i) edge loss, which aims to distinguish between
ambiguous classes at the pixel level in edge regions; (ii) body loss, which
utilizes shape constraints and selectively attends to the inner-semantic
regions. Our experiments demonstrate that the proposed BPKD method provides
extensive refinements and aggregation for edge and body regions. Additionally,
the method achieves state-of-the-art distillation performance for semantic
segmentation on three popular benchmark datasets, highlighting its
effectiveness and generalization ability. BPKD shows consistent improvements
across a diverse array of lightweight segmentation structures, including both
CNNs and transformers, underscoring its architecture-agnostic adaptability. The
code is available at \url{https://github.com/AkideLiu/BPKD}.Comment: 17 pages, 9 figures, 9 table
- …