119 research outputs found
Learning to Generate Parameters of ConvNets for Unseen Image Data
Typical Convolutional Neural Networks (ConvNets) depend heavily on large
amounts of image data and resort to an iterative optimization algorithm (e.g.,
SGD or Adam) to learn network parameters, which makes training very time- and
resource-intensive. In this paper, we propose a new training paradigm and
formulate the parameter learning of ConvNets into a prediction task: given a
ConvNet architecture, we observe there exists correlations between image
datasets and their corresponding optimal network parameters, and explore if we
can learn a hyper-mapping between them to capture the relations, such that we
can directly predict the parameters of the network for an image dataset never
seen during the training phase. To do this, we put forward a new hypernetwork
based model, called PudNet, which intends to learn a mapping between datasets
and their corresponding network parameters, and then predicts parameters for
unseen data with only a single forward propagation. Moreover, our model
benefits from a series of adaptive hyper recurrent units sharing weights to
capture the dependencies of parameters among different network layers.
Extensive experiments demonstrate that our proposed method achieves good
efficacy for unseen image datasets on two kinds of settings: Intra-dataset
prediction and Inter-dataset prediction. Our PudNet can also well scale up to
large-scale datasets, e.g., ImageNet-1K. It takes 8967 GPU seconds to train
ResNet-18 on the ImageNet-1K using GC from scratch and obtain a top-5 accuracy
of 44.65 %. However, our PudNet costs only 3.89 GPU seconds to predict the
network parameters of ResNet-18 achieving comparable performance (44.92 %),
more than 2,300 times faster than the traditional training paradigm
FreeKD: Free-direction Knowledge Distillation for Graph Neural Networks
Knowledge distillation (KD) has demonstrated its effectiveness to boost the
performance of graph neural networks (GNNs), where its goal is to distill
knowledge from a deeper teacher GNN into a shallower student GNN. However, it
is actually difficult to train a satisfactory teacher GNN due to the well-known
over-parametrized and over-smoothing issues, leading to invalid knowledge
transfer in practical applications. In this paper, we propose the first
Free-direction Knowledge Distillation framework via Reinforcement learning for
GNNs, called FreeKD, which is no longer required to provide a deeper
well-optimized teacher GNN. The core idea of our work is to collaboratively
build two shallower GNNs in an effort to exchange knowledge between them via
reinforcement learning in a hierarchical way. As we observe that one typical
GNN model often has better and worse performances at different nodes during
training, we devise a dynamic and free-direction knowledge transfer strategy
that consists of two levels of actions: 1) node-level action determines the
directions of knowledge transfer between the corresponding nodes of two
networks; and then 2) structure-level action determines which of the local
structures generated by the node-level actions to be propagated. In essence,
our FreeKD is a general and principled framework which can be naturally
compatible with GNNs of different architectures. Extensive experiments on five
benchmark datasets demonstrate our FreeKD outperforms two base GNNs in a large
margin, and shows its efficacy to various GNNs. More surprisingly, our FreeKD
has comparable or even better performance than traditional KD algorithms that
distill knowledge from a deeper and stronger teacher GNN.Comment: Accepted to KDD 202
Robust Knowledge Adaptation for Dynamic Graph Neural Networks
Graph structured data often possess dynamic characters in nature, e.g., the
addition of links and nodes, in many real-world applications. Recent years have
witnessed the increasing attentions paid to dynamic graph neural networks for
modelling such graph data, where almost all the existing approaches assume that
when a new link is built, the embeddings of the neighbor nodes should be
updated by learning the temporal dynamics to propagate new information.
However, such approaches suffer from the limitation that if the node introduced
by a new connection contains noisy information, propagating its knowledge to
other nodes is not reliable and even leads to the collapse of the model. In
this paper, we propose AdaNet: a robust knowledge Adaptation framework via
reinforcement learning for dynamic graph neural Networks. In contrast to
previous approaches immediately updating the embeddings of the neighbor nodes
once adding a new link, AdaNet attempts to adaptively determine which nodes
should be updated because of the new link involved. Considering that the
decision whether to update the embedding of one neighbor node will have great
impact on other neighbor nodes, we thus formulate the selection of node update
as a sequence decision problem, and address this problem via reinforcement
learning. By this means, we can adaptively propagate knowledge to other nodes
for learning robust node embedding representations. To the best of our
knowledge, our approach constitutes the first attempt to explore robust
knowledge adaptation via reinforcement learning for dynamic graph neural
networks. Extensive experiments on three benchmark datasets demonstrate that
AdaNet achieves the state-of-the-art performance. In addition, we perform the
experiments by adding different degrees of noise into the dataset,
quantitatively and qualitatively illustrating the robustness of AdaNet.Comment: 14 pages, 6 figure
Parameter-Efficient Conformers via Sharing Sparsely-Gated Experts for End-to-End Speech Recognition
While transformers and their variant conformers show promising performance in
speech recognition, the parameterized property leads to much memory cost during
training and inference. Some works use cross-layer weight-sharing to reduce the
parameters of the model. However, the inevitable loss of capacity harms the
model performance. To address this issue, this paper proposes a
parameter-efficient conformer via sharing sparsely-gated experts. Specifically,
we use sparsely-gated mixture-of-experts (MoE) to extend the capacity of a
conformer block without increasing computation. Then, the parameters of the
grouped conformer blocks are shared so that the number of parameters is
reduced. Next, to ensure the shared blocks with the flexibility of adapting
representations at different levels, we design the MoE routers and
normalization individually. Moreover, we use knowledge distillation to further
improve the performance. Experimental results show that the proposed model
achieves competitive performance with 1/3 of the parameters of the encoder,
compared with the full-parameter model.Comment: accepted in INTERSPEECH 202
The stability of Einstein static universe in the DGP braneworld
The stability of an Einstein static universe in the DGP braneworld scenario
is studied in this paper. Two separate branches denoted by of
the DGP model are analyzed. Assuming the existence of a perfect fluid with a
constant equation of state, , in the universe, we find that, for the branch
with , there is no a stable Einstein static solution, while, for
the case with , the Einstein static universe exists and it is
stable when . Thus, the universe can stay at this stable state
past-eternally and may undergo a series of infinite, non-singular oscillations.
Therefore, the big bang singularity problem in the standard cosmological model
can be resolved.Comment: 10 pages, 2 figures, to appear in PL
Composition and processing activity of a semi-recombinant holo U7 snRNP
In animal cells, replication-dependent histone pre-mRNAs are cleaved at the 3' end by U7 snRNP consisting of two core components: a ∼60-nucleotide U7
snRNA and a ring of seven proteins, with Lsm10 and
Lsm11 replacing the spliceosomal SmD1 and SmD2.
Lsm11 interacts with FLASH and together they recruit the endonuclease CPSF73 and other polyadenylation factors, forming catalytically active holo U7
snRNP. Here, we assembled core U7 snRNP bound to
FLASH from recombinant components and analyzed
its appearance by electron microscopy and ability to
support histone pre-mRNA processing in the presence of polyadenylation factors from nuclear extracts. We demonstrate that semi-recombinant holo
U7 snRNP reconstituted in this manner has the same
composition and functional properties as endogenous U7 snRNP, and accurately cleaves histone pre-mRNAs in a reconstituted in vitro processing reaction. We also demonstrate that the U7-specific Sm
ring assembles efficiently in vitro on a spliceosomal
Sm site but the engineered U7 snRNP is functionally
impaired. This approach offers a unique opportunity
to study the importance of various regions in the Sm
proteins and U7 snRNA in 3' end processing of histone pre-mRNAs
The transcription factor Sp1 modulates RNA polymerase III gene transcription by controlling BRF1 and GTF3C2 expression in human cells
Specificity protein 1 (Sp1) is an important transcription factor implicated in numerous cellular processes. However, whether Sp1 is involved in the regulation of RNA polymerase III (Pol III)directed gene transcription in human cells remains unknown. Here, we first show that filamin A (FLNA) represses Sp1 expression as well as expression of TFIIB-related factor 1 (BRF1) and general transcription factor III C subunit 2 (GTF3C2) in HeLa, 293T, and SaOS2 cell lines stably expressing FLNA-silencing shRNAs. Both BRF1 promoter 4 (BRF1P4) and GTF3C2 promoter 2 (GTF3C2P2) contain putative Sp1-binding sites, suggesting that Sp1 affects Pol III gene transcription by regulating BRF1 and GTF3C2 expression. We demonstrate that Sp1 knockdown inhibits Pol III gene transcription, BRF1 and GTF3C2 expression, and the proliferation of 293T and HeLa cells, whereas Sp1 overexpression enhances these activities. We obtained a comparable result in a cell line in which both FLNA and Sp1 were depleted. These results indicate that Sp1 is involved in the regulation of Pol III gene transcription independently of FLNA expression. Reporter gene assays showed that alteration of Sp1 expression affects BRF1P4 and GTF3C2P2 activation, suggesting that Sp1 modulates Pol III-mediated gene transcription by controlling BRF1 and GTF3C2 gene expression. Further analysis revealed that Sp1 interacts with and thereby promotes the occupancies of TATA box- binding protein, TFIIAα, and p300 at both BRF1P4 and GTF3C2P2. These findings indicate that Sp1 controls Pol III- directed transcription and shed light on how Sp1 regulates cancer cell proliferation
Structure and ion-release mechanism of PIB-4-type ATPases.
Funder: The memorial foundation of manufacturer Vilhelm Pedersen and wife - and the Aarhus Wilson consortiumFunder: China Scholarship Council; FundRef: http://dx.doi.org/10.13039/501100004543Funder: Carl Tryggers Stiftelse för Vetenskaplig Forskning; FundRef: http://dx.doi.org/10.13039/501100002805; Grant(s): CTS 17:22Funder: Agnes og Poul Friis Fond; FundRef: http://dx.doi.org/10.13039/100009512Transition metals, such as zinc, are essential micronutrients in all organisms, but also highly toxic in excessive amounts. Heavy-metal transporting P-type (PIB) ATPases are crucial for homeostasis, conferring cellular detoxification and redistribution through transport of these ions across cellular membranes. No structural information is available for the PIB-4-ATPases, the subclass with the broadest cargo scope, and hence even their topology remains elusive. Here, we present structures and complementary functional analyses of an archetypal PIB-4-ATPase, sCoaT from Sulfitobacter sp. NAS14-1. The data disclose the architecture, devoid of classical so-called heavy-metal-binding domains (HMBDs), and provide fundamentally new insights into the mechanism and diversity of heavy-metal transporters. We reveal several novel P-type ATPase features, including a dual role in heavy-metal release and as an internal counter ion of an invariant histidine. We also establish that the turnover of PIB-ATPases is potassium independent, contrasting to many other P-type ATPases. Combined with new inhibitory compounds, our results open up for efforts in for example drug discovery, since PIB-4-ATPases function as virulence factors in many pathogens
- …