426 research outputs found
Preserving the Hypernym Tree of WordNet in Dense Embeddings
In this paper, we provide a novel way to generate low-dimension (dense)
vector embeddings for the noun and verb synsets in WordNet, so that the
hypernym-hyponym tree structure is preserved in the embeddings. We call this
embedding the sense spectrum (and sense spectra for embeddings). In order to
create suitable labels for the training of sense spectra, we designed a new
similarity measurement for noun and verb synsets in WordNet. We call this
similarity measurement the hypernym intersection similarity (HIS), since it
compares the common and unique hypernyms between two synsets.
Our experiments show that on the noun and verb pairs of the SimLex-999
dataset, HIS outperforms the three similarity measurements in WordNet.
Moreover, to the best of our knowledge, the sense spectra is the first dense
embedding system that can explicitly and completely measure the
hypernym-hyponym relationship in WordNet
Understanding the Spectral Bias of Coordinate Based MLPs Via Training Dynamics
Spectral bias is an important observation of neural network training, stating
that the network will learn a low frequency representation of the target
function before converging to higher frequency components. This property is
interesting due to its link to good generalization in over-parameterized
networks. However, in applications to scene rendering, where multi-layer
perceptrons (MLPs) with ReLU activations utilize dense, low dimensional
coordinate based inputs, a severe spectral bias occurs that obstructs
convergence to high freqeuncy components entirely. In order to overcome this
limitation, one can encode the inputs using high frequency sinusoids. Previous
works attempted to explain both spectral bias and its severity in the
coordinate based regime using Neural Tangent Kernel (NTK) and Fourier analysis.
However, such methods come with various limitations, since NTK does not capture
real network dynamics, and Fourier analysis only offers a global perspective on
the frequency components of the network. In this paper, we provide a novel
approach towards understanding spectral bias by directly studying ReLU MLP
training dynamics, in order to gain further insight on the properties that
induce this behavior in the real network. Specifically, we focus on the
connection between the computations of ReLU networks (activation regions), and
the convergence of gradient descent. We study these dynamics in relation to the
spatial information of the signal to provide a clearer understanding as to how
they influence spectral bias, which has yet to be demonstrated. Additionally,
we use this formulation to further study the severity of spectral bias in the
coordinate based setting, and why positional encoding overcomes this.Comment: 8 pages, 10 figures, preprin
- …