In this paper we explore

the possibility to merge the world of

Compositional Distributional Semantic

Models (CDSM) with Tree Kernels

(TK). In particular, we will introduce a

specific tree kernel (smoothed tree kernel,

or STK) and then show that is

possibile to approximate such kernel

with the dot product of two vectors

obtained compositionally from the sentences,

creating in such a way a new

CDSM

Ferrone, L

Zanzotto, F

Cineca Institutional Research Information System

Distributed Smoothed Tree Kernel

In this paper we explore
the possibility to merge the world of
Compositional Distributional Semantic
Models (CDSM) with Tree Kernels
(TK). In particular, we will introduce a
specific tree kernel (smoothed tree kernel,
or STK) and then show that is
possibile to approximate such kernel
with the dot product of two vectors
obtained compositionally from the sentences,
creating in such a way a new
CDSM

ZANZOTTO, FABIO MASSIMO

English

In this paper we explore the possibility to merge the world of Compositional Distributional Semantic Models (CDSM) with Tree Kernels (TK). In particular, we will introduce a specific tree kernel (smoothed tree kernel, or STK) and then show that is possibile to approximate such kernel with the dot product of two vectors obtained compositionally from the sentences, creating in such a way a new CDSM

Archivio della Ricerca - Università di Roma Tor vergata

Distributed Smoothed Tree Kernel
Lorenzo Ferrone
University of Rome “Tor Vergata”
Via del Politecnico 1
00133 Roma, Italy
lorenzo.ferrone@gmail.com
Fabio Massimo Zanzotto
University of Rome “Tor Vergata”
Via del Politecnico 1
00133 Roma, Italy
fabio.massimo.zanzotto@uniroma2.it
Abstract
English. In this paper we explore
the possibility to merge the world of
Compositional Distributional Semantic
Models (CDSM) with Tree Kernels
(TK). In particular, we will introduce a
specific tree kernel (smoothed tree ker-
nel, or STK) and then show that is
possibile to approximate such kernel
with the dot product of two vectors
obtained compositionally from the sen-
tences, creating in such a way a new
CDSM.
Italiano. In questo paper vogliamo
esplorare la possibilita` di unire il mon-
do dei metodi di semantica distribuzio-
ne composizionale (CDSM) con quello
dei tree Kernel (TK). In particolare in-
trodurremo un particolare tree kernel e
poi mostreremo che possibile appros-
simare questo kernel tramite il prodot-
to scalare tra due vettori ottenuti com-
posizionalmente a partire dalle frasi di
partenza, creando cos`ı di fatto un nuo-
vo modello di semantica distribuzionale
composizionale.
1 Introduction
Compositional distributional semantics is a
flourishing research area that leverages dis-
tributional semantics (see Baroni and Lenci
(2010)) to produce meaning of simple phrases
and full sentences (hereafter called text frag-
ments). The aim is to scale up the success
of word-level relatedness detection to longer
fragments of text. Determining similarity or
relatedness among sentences is useful for many
applications, such as multi-document summar-
ization, recognizing textual entailment (Dagan
et al., 2013), and semantic textual similarity
detection (Agirre et al., 2013; Jurgens et al.,
2014). Compositional distributional semantics
models (CDSMs) are functions mapping text
fragments to vectors (or higher-order tensors).
Functions for simple phrases directly map dis-
tributional vectors of words to distributional
vectors for the phrases (Mitchell and Lapata,
2008; Baroni and Zamparelli, 2010; Zanzotto
et al., 2010). Functions for full sentences are
generally defined as recursive functions over
the ones for phrases (Socher et al., 2011). Dis-
tributional vectors for text fragments are then
used as inner layers in neural networks, or to
compute similarity among text fragments via
dot product.
CDSMs generally exploit structured repres-
entations tx of text fragments x to derive their
meaning f(tx), but the structural information,
although extremely important, is obfuscated
in the final vectors. Structure and meaning
can interact in unexpected ways when comput-
ing cosine similarity (or dot product) between
vectors of two text fragments, as shown for
full additive models in (Ferrone and Zanzotto,
2013).
Smoothed tree kernels (STK) (Croce et
al., 2011) instead realize a clearer interaction
between structural information and distribu-
tional meaning. STKs are specific realiza-
tions of convolution kernels (Haussler, 1999)
where the similarity function is recursively
(and, thus, compositionally) computed. Dis-
tributional vectors are used to represent word
meaning in computing the similarity among
nodes. STKs, however, are not considered part
of the CDSMs family. As usual in kernel ma-
chines (Cristianini and Shawe-Taylor, 2000),
STKs directly compute the similarity between
two text fragments x and y over their tree rep-
resentations tx and ty, that is, STK(tx, ty).
The function f that maps trees into vectors is
169
10.12871/CLICIT2014133
only implicitly used, and, thus, STK(tx, ty) is
not explicitly expressed as the dot product or
the cosine between f(tx) and f(ty).
Such a function f , which is the underlying
reproducing function of the kernel (Aronszajn,
1950), is a CDSM since it maps trees to vectors
by using distributional meaning. However, the
huge nality of Rn (since it has to represent the
set of all possible subtrees) prevents to actu-
ally compute the function f(t), which thus can
only remain implicit.
Distributed tree kernels (DTK) (Zanzotto
and Dell’Arciprete, 2012) partially solve the
last problem. DTKs approximate standard
tree kernels (such as (Collins and Duffy, 2002))
by defining an explicit function DT that maps
trees to vectors in Rm where m  n and Rn
is the explicit space for tree kernels. DTKs
approximate standard tree kernels (TK), that
is, 〈DT (tx), DT (ty)〉 ≈ TK(tx, ty), by approx-
imating the corresponding reproducing func-
tion. Thus, these distributed trees are small
vectors that encode structural information. In
DTKs tree nodes u and v are represented by
nearly orthonormal vectors, that is, vectors
→
u
and
→
v such that 〈→u,→v 〉 ≈ δ(→u,→v ) where δ is
the Kroneker’s delta. This is in contrast with
distributional semantics vectors where 〈→u,→v 〉
is allowed to be any value in [0, 1] according
to the similarity between the words v and u.
In this paper, leveraging on distributed trees,
we present a novel class of CDSMs that en-
code both structure and distributional mean-
ing: the distributed smoothed trees (DST).
DSTs carry structure and distributional mean-
ing on a rank-2 tensor (a matrix): one dimen-
sion encodes the structure and one dimension
encodes the meaning. By using DSTs to com-
pute the similarity among sentences with a
generalized dot product (or cosine), we impli-
citly define the distributed smoothed tree ker-
nels (DSTK) which approximate the corres-
ponding STKs. We present two DSTs along
with the two smoothed tree kernels (STKs)
that they approximate. We experiment with
our DSTs to show that their generalized dot
products approximate STKs by directly com-
paring the produced similarities and by com-
paring their performances on two tasks: re-
cognizing textual entailment (RTE) and se-
mantic similarity detection (STS). Both ex-
periments show that the dot product on DSTs
approximates STKs and, thus, DSTs encode
both structural and distributional semantics
of text fragments in tractable rank-2 tensors.
Experiments on STS and RTE show that dis-
tributional semantics encoded in DSTs in-
creases performance over structure-only ker-
nels. DSTs are the first positive way of taking
into account both structure and distributional
meaning in CDSMs. The rest of the paper is
organized as follows. Section 2.1 introduces
the basic notation used in the paper. Sec-
tion 2 describe our distributed smoothed trees
as compositional distributional semantic mod-
els that can represent both structural and se-
mantic information. Section 4 reports on the
experiments. Finally, Section 5 draws some
conclusions.
2 Distributed Smoothed Tree
Kernel
We here propose a model that can be con-
sidered a compositional distributional se-
mantic model as it transforms sentences into
matrices that can then used by the learner
as feature vectors. Our model is called Dis-
tributed Smoothed Tree Kernel (Ferrone and
Zanzotto, 2014) as it mixes the distributed
trees (Zanzotto and Dell’Arciprete, 2012) rep-
resenting syntactic information with distribu-
tional semantic vectors representing semantic
information.
S:booked::vXXXXX

NP:we::p
PRP:we::p
We
VP:booked::v
PPPP

V:booked::v
booked
NP:flight::n
aaa
!!!
DT:the::d
the
NN:flight::n
flight
Figure 1: A lexicalized tree
2.1 Notation
Before describing the distributed smoothed
trees (DST) we introduce a formal way to de-
note constituency-based lexicalized parse trees,
as DSTs exploit this kind of data structures.
Lexicalized trees are denoted with the letter t
and N(t) denotes the set of non terminal nodes
of tree t. Each non-terminal node n ∈ N(t)
170
S(t) = {
S:booked::v
ZZ
NP VP
,
VP:booked::v
ll,,
V NP
,
NP:we::p
PRP
,
S:booked::v
ll,,
NP
PRP
VP , . . . ,
VP:booked::v
HH
V
booked
NP
@ 
DT NN
, . . . }
Figure 2: Subtrees of the tree t in Figure 1 (a non-exhaustive list)
has a label ln composed of two parts ln =
(sn, wn): sn is the syntactic label, while wn
is the semantic headword of the tree headed
by n, along with its part-of-speech tag. Ter-
minal nodes of trees are treated differently,
these nodes represent only words wn without
any additional information, and their labels
thus only consist of the word itself (see Fig.
1). The structure of a DST is represented as
follows: Given a tree t, h(t) is its root node and
s(t) is the tree formed from t but considering
only the syntactic structure (that is, only the
sn part of the labels), ci(n) denotes i-th child
of a node n. As usual for constituency-based
parse trees, pre-terminal nodes are nodes that
have a single terminal node as child.
Finally, we use
→
wn ∈ Rk to denote the dis-
tributional vector for word wn.
2.2 The method at a glance
We describe here the approach in a few sen-
tences. In line with tree kernels over struc-
tures (Collins and Duffy, 2002), we introduce
the set S(t) of the subtrees ti of a given lexic-
alized tree t. A subtree ti is in the set S(t) if
s(ti) is a subtree of s(t) and, if n is a node in
ti, all the siblings of n in t are in ti. For each
node of ti we only consider its syntactic label
sn, except for the head h(ti) for which we also
consider its semantic component wn (see Fig.
2). The functions DSTs we define compute the
following:
DST (t) = T =
∑
ti∈S(t)
Ti
where Ti is the matrix associated to each sub-
tree ti. The similarity between two text frag-
ments a and b represented as lexicalized trees
ta and tb can be computed using the Frobenius
product between the two matrices Ta and Tb,
that is:
〈Ta,Tb〉F =
∑
tai ∈S(ta)
tbj∈S(tb)
〈Tai ,Tbj〉F (1)
We want to obtain that the product 〈Tai ,Tbj〉F
approximates the dot product between the
distributional vectors of the head words
(〈Tai ,Tbj〉F ≈ 〈
→
h(tai ),
→
h(tbj)〉) whenever the syn-
tactic structure of the subtrees is the same
(that is s(tai ) = s(t
b
j)), and 〈Tai ,Tbj〉F ≈ 0 oth-
erwise. This property is expressed as:
〈Tai ,Tbj〉F ≈ δ(s(tai ), s(tbj)) · 〈
→
h(tai ),
→
h(tbj)〉 (2)
To obtain the above property, we define
Ti =
→
s(ti)
→
wh(ti)
>
where
→
s(ti) are distributed tree fragment (Zan-
zotto and Dell’Arciprete, 2012) for the sub-
tree t and
→
wh(ti) is the distributional vec-
tor of the head of the subtree t. Distrib-
uted tree fragments have the property that
→
s(ti)
→
s(tj) ≈ δ(ti, tj). Thus, exploiting the fact
that: 〈→a→w>,
→
b
→
v
>〉F = 〈→a ,
→
b 〉·〈→w,→v 〉, we have
that Equation 2 is satisfied as:
〈Ti,Tj〉F = 〈
→
s(ti),
→
s(tj)〉 · 〈 →wh(ti),
→
wh(tj)〉
≈ δ(s(ti), s(tj)) · 〈 →wh(ti),
→
wh(tj)〉
It is possible to show that the overall composi-
tional distributional model DST (t) can be ob-
tained with a recursive algorithm that exploits
vectors of the nodes of the tree.
3 The Approximated Smoothed
Tree Kernels
The CDSM we proposed approximates a spe-
cific tree kernel belonging to the smoothed tree
kernels class. This recursively computes (but,
the recursive formulation is not given here) the
following general equation:
STK(ta, tb) =
∑
ti∈S(ta)
tj∈S(tb)
ω(ti, tj)
171
RTE1 RTE2 RTE3 RTE5 headl FNWN OnWN SMT
STK vs DSTK
1024 0.86 0.84 0.90 0.84 0.87 0.65 0.95 0.77
2048 0.87 0.84 0.91 0.84 0.90 0.65 0.96 0.77
Table 1: Spearman’s correlation between Distributed Smoothed Tree Kernels and Smoothed
Tree Kernels
where ω(ti, tj) is the similarity weight between
two subtrees ti and tj . DTSK approximates
STK, where the weights are defined as follows:
ω(ti, tj) = α · 〈 →wh(ti),
→
wh(tj)〉 · δ(s(ti), s(tj))
Where α =
√
λ|N(ti)|+|N(tj)| and λ is a para-
meter.
4 Experimental investigation
Generic settings We experimented with
two datasets: the Recognizing Textual Entail-
ment datasets (RTE) (Dagan et al., 2006) and
the the Semantic Textual Similarity 2013 data-
sets (STS) (Agirre et al., 2013). The STS task
consists of determining the degree of similar-
ity (ranging from 0 to 5) between two sen-
tences. The STS datasets contains 5 datasets:
headlines, OnWN, FNWN and SMT which
contains respectively 750, 561, 189 and 750
RTE is instead the task of deciding whether
a long text T entails a shorter text, typically
a single sentence, called hypothesis H. It has
been often seen as a classification task. We
used four datasets: RTE1, RTE2, RTE3, and
RTE5. We parsed the sentence with the Stan-
ford Parser (Klein and Manning, 2003) and
extracted the heads for use in the lexicalized
trees with Collins’ rules (Collins, 2003). Dis-
tributional vectors are derived with DISSECT
(Dinu et al., 2013) from a corpus obtained
by the concatenation of ukWaC, a mid-2009
dump of the English Wikipedia and the British
National Corpus for a total of about 2.8 bil-
lion words. The raw count vectors were trans-
formed into positive Pointwise Mutual Inform-
ation scores and reduced to 300 dimensions by
Singular Value Decomposition. This setup was
picked without tuning, as we found it effective
in previous, unrelated experiments. To build
our DTSKs we used the implementation of the
distributed tree kernels1. We used 1024 and
2048 as the dimension of the distributed vec-
tors, the weight λ is set to 0.4 as it is a value
1http://code.google.com/p/distributed-tree-
kernels/
generally considered optimal for many applic-
ations (see also (Zanzotto and Dell’Arciprete,
2012)). To test the quality of the approxima-
tion we computed the Spearman’s correlation
between values produced by our DSTK and
by the standard versions of the smoothed tree
kernel. We obtained text fragment pairs by
randomly sampling two text fragments in the
selected set. For each set, we produced ex-
actly the number of examples in the set, e.g.,
we produced 567 pairs for RTE1, etc.
Results Table 1 reports the results for the
correlation experiments. We report the Spear-
man’s correlations over the different sets (and
different dimensions of distributed vectors)
between our DSTK and the STK. The cor-
relation is above 0.80 in average for both RTE
and STS datasets. The approximation also de-
pends on the size of the distributed vectors.
Higher dimensions yield to better approxim-
ation: if we increase the distributed vectors
dimension from 1024 to 2048 the correlation
between DSTK and STK increases. This dir-
ect analysis of the correlation shows that our
CDSM are approximating the corresponding
kernel function and there is room of improve-
ment by increasing the size of distributed vec-
tors.
5 Conclusions and future work
Distributed Smoothed Trees (DST) are a
novel class of Compositional Distributional Se-
mantics Models (CDSM) that effectively en-
code structural information and distributional
semantics in tractable rank-2 tensors, as ex-
periments show. The paper shows that DSTs
contribute to close the gap between two appar-
ently different approaches: CDSMs and convo-
lution kernels. This contribute to start a dis-
cussion on a deeper understanding of the rep-
resentation power of structural information of
existing CDSMs.
172
References
[Agirre et al.2013] Eneko Agirre, Daniel Cer, Mona
Diab, Aitor Gonzalez-Agirre, and Weiwei Guo.
2013. *sem 2013 shared task: Semantic tex-
tual similarity. In Second Joint Conference on
Lexical and Computational Semantics (*SEM),
Volume 1: Proceedings of the Main Conference
and the Shared Task: Semantic Textual Similar-
ity, pages 32–43, Atlanta, Georgia, USA, June.
Association for Computational Linguistics.
[Aronszajn1950] N. Aronszajn. 1950. Theory of re-
producing kernels. Transactions of the Amer-
ican Mathematical Society, 68(3):337–404.
[Baroni and Lenci2010] Marco Baroni and Aless-
andro Lenci. 2010. Distributional memory: A
general framework for corpus-based semantics.
Comput. Linguist., 36(4):673–721, December.
[Baroni and Zamparelli2010] Marco Baroni and
Roberto Zamparelli. 2010. Nouns are vec-
tors, adjectives are matrices: Representing
adjective-noun constructions in semantic space.
In Proceedings of the 2010 Conference on Em-
pirical Methods in Natural Language Processing,
pages 1183–1193, Cambridge, MA, October.
Association for Computational Linguistics.
[Collins and Duffy2002] Michael Collins and Nigel
Duffy. 2002. New ranking algorithms for pars-
ing and tagging: Kernels over discrete struc-
tures, and the voted perceptron. In Proceedings
of ACL02.
[Collins2003] Michael Collins. 2003. Head-driven
statistical models for natural language parsing.
Comput. Linguist., 29(4):589–637.
[Cristianini and Shawe-Taylor2000] Nello Cris-
tianini and John Shawe-Taylor. 2000. An
Introduction to Support Vector Machines
and Other Kernel-based Learning Methods.
Cambridge University Press, March.
[Croce et al.2011] Danilo Croce, Alessandro
Moschitti, and Roberto Basili. 2011. Struc-
tured lexical similarity via convolution kernels
on dependency trees. In Proceedings of the
Conference on Empirical Methods in Natural
Language Processing, EMNLP ’11, pages 1034–
1046, Stroudsburg, PA, USA. Association for
Computational Linguistics.
[Dagan et al.2006] Ido Dagan, Oren Glickman, and
Bernardo Magnini. 2006. The pascal re-
cognising textual entailment challenge. In
Quin˜onero-Candela et al., editor, LNAI 3944:
MLCW 2005, pages 177–190. Springer-Verlag,
Milan, Italy.
[Dagan et al.2013] Ido Dagan, Dan Roth, Mark
Sammons, and Fabio Massimo Zanzotto. 2013.
Recognizing Textual Entailment: Models and
Applications. Synthesis Lectures on Human
Language Technologies. Morgan & Claypool
Publishers.
[Dinu et al.2013] Georgiana Dinu, Nghia The
Pham, and Marco Baroni. 2013. DISSECT:
DIStributional SEmantics Composition Toolkit.
In Proceedings of ACL (System Demonstra-
tions), pages 31–36, Sofia, Bulgaria.
[Ferrone and Zanzotto2013] Lorenzo Ferrone and
Fabio Massimo Zanzotto. 2013. Linear compos-
itional distributional semantics and structural
kernels. In Proceedings of the Joint Symposium
of Semantic Processing (JSSP).
[Ferrone and Zanzotto2014] Lorenzo Ferrone and
Fabio Massimo Zanzotto. 2014. Towards
syntax-aware compositional distributional se-
mantic models. In Proceedings of Coling 2014.
COLING, Dublin, Ireland, Aug 23–Aug 29.
[Haussler1999] David Haussler. 1999. Convolution
kernels on discrete structures. Technical report,
University of California at Santa Cruz.
[Jurgens et al.2014] David Jurgens, Mo-
hammad Taher Pilehvar, and Roberto Navigli.
2014. Semeval-2014 task 3: Cross-level se-
mantic similarity. In Proceedings of the 8th
International Workshop on Semantic Evalu-
ation (SemEval 2014), pages 17–26, Dublin,
Ireland, August. Association for Computational
Linguistics and Dublin City University.
[Klein and Manning2003] Dan Klein and Chris-
topher D. Manning. 2003. Accurate unlexical-
ized parsing. In Proceedings of the 41st Annual
Meeting on Association for Computational Lin-
guistics - Volume 1, ACL ’03, pages 423–430,
Stroudsburg, PA, USA. Association for Compu-
tational Linguistics.
[Mitchell and Lapata2008] Jeff Mitchell and
Mirella Lapata. 2008. Vector-based models
of semantic composition. In Proceedings of
ACL-08: HLT, pages 236–244, Columbus,
Ohio, June. Association for Computational
Linguistics.
[Socher et al.2011] Richard Socher, Eric H. Huang,
Jeffrey Pennington, Andrew Y. Ng, and Chris-
topher D. Manning. 2011. Dynamic pooling and
unfolding recursive autoencoders for paraphrase
detection. In Advances in Neural Information
Processing Systems 24.
[Zanzotto and Dell’Arciprete2012] F.M. Zanzotto
and L. Dell’Arciprete. 2012. Distributed tree
kernels. In Proceedings of International Confer-
ence on Machine Learning, pages 193–200.
[Zanzotto et al.2010] Fabio Massimo Zanzotto,
Ioannis Korkontzelos, Francesca Fallucchi,
and Suresh Manandhar. 2010. Estimating
linear models for compositional distributional
173
semantics. In Proceedings of the 23rd Interna-
tional Conference on Computational Linguistics
(COLING), August,.
174


Zanzotto, FM

https://art.torvergata.it/retrieve/handle/2108/106807/314812/CLICIT2014133.pdf

Distributed Smoothed Tree Kernel

Abstract

Similar works

Full text

Available Versions

Cineca Institutional Research Information System

ART

ART

Archivio della Ricerca - Università di Roma Tor vergata

Cineca Institutional Research Information System

Archivio della Ricerca - Università di Roma Tor vergata