239 research outputs found
Decouple knowledge from paramters for plug-and-play language modeling
Pre-trained language models(PLM) have made impressive results in various NLP
tasks. It has been revealed that one of the key factors to their success is the
parameters of these models implicitly learn all kinds of knowledge during
pre-training. However, encoding knowledge implicitly in the model parameters
has two fundamental drawbacks. First, the knowledge is neither editable nor
scalable once the model is trained, which is especially problematic in that
knowledge is consistently evolving. Second, it lacks interpretability and
prevents humans from understanding which knowledge PLM requires for a certain
problem. In this paper, we introduce PlugLM, a pre-training model with
differentiable plug-in memory(DPM). The key intuition is to decouple the
knowledge storage from model parameters with an editable and scalable key-value
memory and leverage knowledge in an explainable manner by knowledge retrieval
in the DPM. To justify this design choice, we conduct evaluations in three
settings including: (1) domain adaptation. PlugLM obtains 3.95 F1 improvements
across four domains on average without any in-domain pre-training. (2)
knowledge update. PlugLM could absorb new knowledge in a training-free way
after pre-training is done. (3) in-task knowledge learning. PlugLM could be
further improved by incorporating training samples into DPM with knowledge
prompting.Comment: ACL2023 Finding
Valley vortex states and degeneracy lifting via photonic higher-band excitation
We demonstrate valley-dependent vortex generation in a photonic graphene.
Without breaking the inversion symmetry, excitation of two equivalent valleys
leads to formation of an optical vortex upon Bragg-reflection to the third
valley, with its chirality determined by the valley degree of freedom.
Vortex-antivortex pairs with valley-dependent topological charge flipping are
also observed and corroborated by numerical simulations. Furthermore, we
develop a three-band effective Hamiltonian model to describe the dynamics of
the coupled valleys, and find that the commonly used two-band model is not
sufficient to explain the observed vortex degeneracy lifting. Such
valley-polarized vortex states arise from high-band excitation without
inversion symmetry breaking or synthetic-field-induced gap opening. Our results
from a photonic setting may provide insight for the study of valley contrasting
and Berry-phase mediated topological phenomena in other systems
- …