In this paper, we propose a classifier for predicting sentiments of Italian Twitter messages. This work builds upon a deep learning approach where we leverage large amounts of weakly labelled data to train a 2-layer convolutional neural network. To train our network we apply a form of multi-task training. Our system participated in the EvalItalia-2016 competition and outperformed all other approaches on the sentiment analysis task

Cieliebak, Mark

Deriu, Jan Milan

Jan Deriu

Mark Cieliebak

Crossref

Sentiment Analysis using Convolutional Neural Networks with Multi-Task Training and Distant Supervision on Italian Tweets

ZHAW digitalcollection

Sentiment Analysis using Convolutional Neural Networks with Multi-Task
Training and Distant Supervision on Italian Tweets
Jan Deriu
Zurich University of Applied Sciences
Switzerland
deri@zhaw.ch
Mark Cieliebak
Zurich University of Applied Sciences
Switzerland
ciel@zhaw.ch
Abstract
English. In this paper, we propose a clas-
sifier for predicting sentiments of Italian
Twitter messages. This work builds upon
a deep learning approach where we lever-
age large amounts of weakly labelled data
to train a 2-layer convolutional neural net-
work. To train our network we apply a
form of multi-task training. Our system
participated in the EvalItalia-2016 com-
petition and outperformed all other ap-
proaches on the sentiment analysis task.
In questo articolo, presentiamo un sis-
tema per la classificazione di soggettività
e polarità di tweet in lingua italiana.
L’approccio descritto si basa su reti neu-
rali. In particolare, utilizziamo un dataset
di 300M di tweet per addestrare una con-
volutional neural network. Il sistema è
stato addestrato e valutato sui dati for-
niti dagli organizzatori di Sentipolc, task
di sentiment analysis su Twitter organiz-
zato nell’ambito di Evalita 2016..
1 Introduction
Sentiment analysis is a fundamental problem aim-
ing to give a machine the ability to understand the
emotions and opinions expressed in a written text.
This is an extremely challenging task due to the
complexity and variety of human language.
The sentiment polarity classification task of
EvalItalia-2016 1 (sentipolc) consists of three sub-
tasks which cover different aspects of sentiment
detection: T1 : Subjectivity detection: is the tweet
subjective or objective? T2 : Polarity detection: is
the sentiment of the tweet neutral, positive, nega-
tive or mixed?
1http://www.di.unito.it/˜tutreeb/sentipolc-
evalita16/index.html
T3 : Irony detection: is the tweet ironic?
The classic approaches to sentiment analysis usu-
ally consist of manual feature engineering and ap-
plying some sort of classifier on these features
(Liu, 2015). Deep neural networks have shown
great promises at capturing salient features for
these complex tasks (Mikolov et al., 2013b; Sev-
eryn and Moschitti, 2015a). Particularly success-
ful for sentiment classification were Convolutional
Neural Networks (CNN) (Kim, 2014; Kalchbren-
ner et al., 2014; Severyn and Moschitti, 2015b;
Johnson and Zhang, 2015), on which our work
builds upon. These networks typically have a large
number of parameters and are especially effective
when trained on large amounts of data.
In this work, we use a distant supervision ap-
proach to leverage large amounts of data in order
to train a 2-layer CNN 2. More specifically, we
train a neural network using the following three-
phase procedure: P1 : creation of word embed-
dings for the initialization of the first layer based
on an unsupervised corpus of 300M Italian tweets;
P2 : distant supervised phase, where the net-
work is pre-trained on a weakly labelled dataset of
40M tweets where the network weights and word
embeddings are trained to capture aspects related
to sentiment; and P3 : supervised phase, where
the network is trained on the provided supervised
training data consisting of 7410 manually labelled
tweets.
As the three tasks of EvalItalia-2016 are closely
related we apply a form of multitask training as
proposed by (Collobert et al., 2011), i.e. we train
one CNN for all the tasks simultaneously. This
has two advantages: i) we need to train only
one model instead of three models, and ii) the
CNN has access to more information which ben-
efits the score. The experiments indicate that the
multi-task CNN performs better than the single-
2We here refer to a layer as one convolutional and one
pooling layer.
task CNN. After a small bugfix regarding the data-
preprocessing our system outperforms all the other
systems in the sentiment polarity task.
2 Convolutional Neural Networks
We train a 2-layer CNN using 9-fold cross-
validation and combine the outputs of the 9 re-
sulting classifiers to increase robustness. The 9
classifiers differ in the data used for the super-
vised phase since cross-validation creates 9 differ-
ent training and validation sets.
The architecture of the CNN is shown in Fig-
ure 1 and described in detail below.
Sentence model. Each word in the input data is
associated to a vector representation, which con-
sists in a d-dimensional vector. A sentence (or
tweet) is represented by the concatenation of the
representations of its n constituent words. This
yields a matrix S ∈ Rd×n, which is used as input
to the convolutional neural network.
The first layer of the network consists of a lookup
table where the word embeddings are represented
as a matrix X ∈ Rd×|V|, where V is the vocabu-
lary. Thus the i-th column of X represents the i-th
word in the vocabulary V .
Convolutional layer. In this layer, a set ofm fil-
ters is applied to a sliding window of length h over
each sentence. Let S[i:i+h] denote the concatena-
tion of word vectors si to si+h. A feature ci is
generated for a given filter F by:
ci :=
∑
k,j
(S[i:i+h])k,j · Fk,j (1)
A concatenation of all vectors in a sentence pro-
duces a feature vector c ∈ Rn−h+1. The vectors
c are then aggregated over all m filters into a fea-
ture map matrix C ∈ Rm×(n−h+1). The filters
are learned during the training phase of the neu-
ral network using a procedure detailed in the next
section.
Max pooling. The output of the convolutional
layer is passed through a non-linear activation
function, before entering a pooling layer. The lat-
ter aggregates vector elements by taking the max-
imum over a fixed set of non-overlapping inter-
vals. The resulting pooled feature map matrix has
the form: Cpooled ∈ Rm×
n−h+1
s , where s is the
length of each interval. In the case of overlap-
ping intervals with a stride value st, the pooled
feature map matrix has the form Cpooled ∈
Rm×
n−h+1−s
st . Depending on whether the borders
are included or not, the result of the fraction is
rounded up or down respectively.
Hidden layer. A fully connected hidden layer
computes the transformation α(W∗x+b), where
W ∈ Rm×m is the weight matrix, b ∈ IRm the
bias, and α the rectified linear (relu) activation
function (Nair and Hinton, 2010). The output vec-
tor of this layer, x ∈ Rm, corresponds to the sen-
tence embeddings for each tweet.
Softmax. Finally, the outputs of the hidden layer
x ∈ Rm are fully connected to a soft-max regres-
sion layer, which returns the class ŷ ∈ [1,K] with
largest probability,
ŷ := argmax
j
ex
ᵀwj+aj∑K
k=1 e
xᵀwk+aj
, (2)
where wj denotes the weights vector of class j and
aj the bias of class j.
Network parameters. Training the neural net-
work consists in learning the set of parameters
Θ = {X,F1,b1,F2,b2,W,a}, where X is the
embedding matrix, with each row containing the
d-dimensional embedding vector for a specific
word; Fi,bi(i = {1, 2}) the filter weights and bi-
ases of the first and second convolutional layers;
W the concatenation of the weights wj for every
output class in the soft-max layer; and a the bias
of the soft-max layer.
Hyperparameters For both convolutional lay-
ers we set the length of the sliding window h to
5, the size of the pooling interval s is set to 3 in
both layers, where we use a striding of 2 in the
first layer, and the number of filtersm is set to 200
in both convolutional layers.
Dropout Dropout is an alternative technique
used to reduce overfitting (Srivastava et al.,
2014). In each training stage individual nodes are
dropped with probability p, the reduced neural net
is updated and then the dropped nodes are rein-
serted. We apply Dropout to the hidden layer and
to the input layer using p = 0.2 in both cases.
Optimization The network parameters are
learned using AdaDelta (Zeiler, 2012), which
adapts the learning rate for each dimension using
only first order information. We used the default
hyper-parameters.
Sentence Matrix Convolutional 
Feature Map
pooled 
repr.
Convolutional 
Feature Map
pooled 
repr.
Hidden
Layer
Softmax
Figure 1: The architecture of the CNN used in our approach.
Twitter
cat 0.1 0.9 0.3
cats 0.3 0.2 0.7
cute 0.2 0.3 0.1
…
Word Embeddings
Adapted Word Emb.
:-)
:-(
word2vec
GloVe
Distant
Supervision
2-Layer 
CNN
Raw Tweets
(200M)
Smiley 
Tweets
(90M) .
Annotated
Tweets
(18k)
Unknown
Tweet
cat 0.1 0.9 0.3
cats 0.3 0.2 0.7
cute 0.2 0.3 0.1
…
Predictive
Model
3
-P
h
as
e 
Tr
ai
n
in
g
A
p
p
lic
at
io
n
Figure 2: The overall architecture of our 3-phase approach.
3 Training
We train the parameters of the CNN using the
three-phase procedure as described in the intro-
duction. Figure 2 depicts the general flow of this
procedure.
3.1 Three-Phase Training
Preprocessing We apply standard preprocess-
ing procedures of normalizing URLs, hashtags
and usernames, and lowercasing the tweets. The
tweets are converted into a list of indices where
each index corresponds to the word position in the
vocabulary V . This representation is used as in-
put for the lookup table to assemble the sentence
matrix S.
Word Embeddings We create the word embed-
dings in phase P1 using word2vec (Mikolov et al.,
2013a) and train a skip-gram model on a corpus of
300M unlabelled Italian tweets. The window size
for the skip-gram model is 5, the threshold for the
minimal word frequency is set to 20 and the num-
ber of dimensions is d = 52 3. The resulting vo-
cabulary contains 890K unique words. The word
embeddings account for the majority of network
parameters (42.2M out of 46.6M parameters) and
are updated during the next two phases to intro-
duce sentiment specific information into the word
embeddings and create a good initialization for the
CNN.
Distant Supervised Phase We pre-train the
CNN for 1 epoch on an weakly labelled dataset
of 40M Italian tweets where each tweet contains
an emoticon. The label is inferred by the emoti-
cons inside the tweet, where we ignore tweets with
opposite emoticons. This results in 30M positive
tweets and 10M negative tweets. Thus, the classi-
fier is trained on a binary classification task.
Supervised Phase During the supervised phase
we train the pre-trained CNN with the provided
annotated data. The CNN is trained jointly on all
tasks of EvalItalia. There are four different binary
labels as well as some restrictions which result in
9 possible joint labels (for more details, see Sec-
tion 3.2). The multi-task classifier is trained to pre-
dict the most likely joint-label.
We apply 9-fold cross-validation on the dataset
generating 9 equally sized buckets. In each round
we train the CNN using early stopping on the held-
out set, i.e. we train it as long as the score im-
proves on the held-out set. For the multi-task train-
ing we monitor the scores for all 4 subtasks simul-
taneously and store the best model for each sub-
task. The training stops if there is no improvement
of any of the 4 monitored scores.
Meta Classifier We train the CNN using 9-fold
cross-validation, which results in 9 different mod-
els. Each model outputs nine real-value numbers
3According to the gensim implementation of word2vec
using d divisible by 4 speeds up the process.
ŷ corresponding to the probabilities for each of
the nine classes. To increase the robustness of the
system we train a random forest which takes the
outputs of the 9 models as its input. The hyper-
parameters were found via grid-search to obtain
the best overall performance over a development
set: Number of trees (100), maximum depth of the
forest (3) and the number of features used per ran-
dom selection (5).
3.2 Data
The supervised training and test data is provided
by the EvalItalia-2016 competition. Each tweet
contains four labels: L1 : is the tweet subjective
or objective? L2 : is the tweet positive? L3 : is the
tweet negative? L4 : is the tweet ironic? Further-
more an objective tweet implies that it is neither
positive nor negative as well as not ironic. There
are 9 possible combination of labels.
To jointly train the CNN for all three tasks T1, T2
and T3 we join the labels of each tweet into a sin-
gle label. In contrast, the single task training trains
a single model for each of the four labels sepa-
rately.
Table 1 shows an overview of the data available.
Table 1: Overview of datasets provided in EvalItalia-2016.
Label Training Set Test Set
Total 7410 2000
Subjective 5098 1305
Overall Positive 2051 352
Overall Negative 2983 770
Irony 868 235
3.3 Experiments & Results
We compare the performance of the multi-task
CNN with the performance of the single-task
CNNs. All the experiments start at the third-phase,
i.e. the supervised phase. Since there was no
predefined split in training and development set,
we generated a development set by sampling 10%
uniformly at random from the provided training
set. The development set is needed when assess-
ing the generalization power of the CNNs and the
meta-classifier. For each task we compute the av-
eraged F1-score (Barbieri et al., 2016). We present
the results achieved on the dev-set and the test-set
used for the competition. We refer to the set which
was held out during a cross validation iteration as
fold-set.
In Table 2 we show the average results obtained
by the 9 CNNs after the cross validation. The
scores show that the CNN is tuned too much to-
wards the held-out folds since the scores of the
held-out folds are significantly higher. For exam-
ple, the average score of the positivity task is 0.733
on the held-out sets but only 0.6694 on the dev-set
and 0.6601 on the test-set. Similar differences in
sores can be observed for the other tasks as well.
To mitigate this problem we apply a random for-
est on the outputs of the 9 classifiers obtained by
cross-validation. The results are shown in Table
3. The meta-classifier outperforms the average
scores obtained by the CNNs by up to 2 points
on the dev-set. The scores on the test-set show
a slightly lower increase in score. Especially the
single-task classifier did not benefit from the meta-
classifier as the scores on the test set decreased in
some cases.
The results show that the multi-task classifier out-
performs the single-task classifier in most cases.
There is some variation in the magnitude of the
difference: the multi-task classifier outperforms
the single-task classifier by 0.06 points in the neg-
ativity task in the test-set but only by 0.005 points
in the subjectivity task.
Set Task Subjective Positive Negative Irony
Fold-Set Single Task 0.723 0.738 0.721 0.646
Multi Task 0.729 0.733 0.737 0.657
Dev-Set Single Task 0.696 0.650 0.685 0.563
Multi Task 0.710 0.669 0.699 0.595
Test-Set Single Task 0.705 0.652 0.696 0.526
Multi Task 0.681 0.660 0.700 0.540
Table 2: Average F1-score obtained after applying cross val-
idation.
Set Task Subjective Positive Negative Irony
Dev-Set Single Task 0.702 0.693 0.695 0.573
Multi Task 0.714 0.686 0.717 0.604
Test-Set Single Task 0.712 0.650 0.643 0.501
Multi Task 0.714 0.653 0.713 0.536
Table 3: F1-Score obtained by the meta classifier.
4 Conclusion
In this work we presented a deep-learning ap-
proach for sentiment analysis. We described the
three-phase training approach to guarantee a high
quality initialization of the CNN and showed the
effects of using a multi-task training approach. To
increase the robustness of our system we applied
a meta-classifier on top of the CNN. The system
was evaluated in the EvalItalia-2016 competition
where it achieved 1st place in the polarity task and
high positions on the other two subtasks.
References
Francesco Barbieri, Valerio Basile, Danilo Croce,
Malvina Nissim, Nicole Novielli, and Viviana Patti.
2016. Overview of the EVALITA 2016 SENTi-
ment POLarity Classification Task. In Pierpaolo
Basile, Anna Corazza, Franco Cutugno, Simonetta
Montemagni, Malvina Nissim, Viviana Patti, Gio-
vanni Semeraro, and Rachele Sprugnoli, editors,
Proceedings of Third Italian Conference on Compu-
tational Linguistics (CLiC-it 2016) & Fifth Evalua-
tion Campaign of Natural Language Processing and
Speech Tools for Italian. Final Workshop (EVALITA
2016). Associazione Italiana di Linguistica Com-
putazionale (AILC).
Ronan Collobert, Jason Weston, Léon Bottou, Michael
Karlen, Koray Kavukcuoglu, and Pavel Kuksa.
2011. Natural Language Processing (Almost) from
Scratch. JMLR, 12:2493−2537.
Rie Johnson and Tong Zhang. 2015. Semi-supervised
Convolutional Neural Networks for Text Categoriza-
tion via Region Embedding. In NIPS 2015 - Ad-
vances in Neural Information Processing Systems
28, pages 919–927.
Nal Kalchbrenner, Edward Grefenstette, and Phil Blun-
som. 2014. A Convolutional Neural Network for
Modelling Sentences. In ACL - Proceedings of the
52nd Annual Meeting of the Association for Com-
putational Linguistics, pages 655–665, Baltimore,
Maryland, USA, April.
Yoon Kim. 2014. Convolutional Neural Networks for
Sentence Classification. In EMNLP 2014 - Empiri-
cal Methods in Natural Language Processing, pages
1746–1751, August.
Bing Liu. 2015. Sentiment Analysis.
Tomas Mikolov, Quoc V Le, and Ilya Sutskever.
2013a. Exploiting Similarities among Languages
for Machine Translation. arXiv, September.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Cor-
rado, and Jeff Dean. 2013b. Distributed represen-
tations of words and phrases and their composition-
ality. In Advances in neural information processing
systems, pages 3111–3119.
Vinod Nair and Geoffrey E Hinton. 2010. Rectified
linear units improve restricted boltzmann machines.
In Proceedings of the 27th International Conference
on Machine Learning (ICML-10), pages 807–814.
Aliaksei Severyn and Alessandro Moschitti. 2015a.
Twitter Sentiment Analysis with Deep Convolu-
tional Neural Networks. In 38th International
ACM SIGIR Conference, pages 959–962, New York,
USA, August. ACM.
Aliaksei Severyn and Alessandro Moschitti. 2015b.
UNITN: Training Deep Convolutional Neural Net-
work for Twitter Sentiment Classification. In Se-
mEval 2015 - Proceedings of the 9th International
Workshop on Semantic Evaluation.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky,
Ilya Sutskever, and Ruslan Salakhutdinov. 2014.
Dropout: A Simple Way to Prevent Neural Networks
from Overfitting. Journal of Machine Learning Re-
search, 15:1929–1958.
Matthew D. Zeiler. 2012. ADADELTA: An Adaptive
Learning Rate Method. arXiv, page 6.


Sentiment analysis using convolutional neural networks with multi-task training and distant supervision on italian tweets

Sentiment Analysis using Convolutional Neural Networks with Multi-TaskTraining and Distant Supervision on Italian TweetsJan DeriuZurich University of Applied SciencesSwitzerlandderi@zhaw.chMark CieliebakZurich University of Applied SciencesSwitzerlandciel@zhaw.chAbstractEnglish. In this paper, we propose a clas-sifier for predicting sentiments of ItalianTwitter messages. This work builds upona deep learning approach where we lever-age large amounts of weakly labelled datato train a 2-layer convolutional neural net-work. To train our network we apply aform of multi-task training. Our systemparticipated in the EvalItalia-2016 com-petition and outperformed all other ap-proaches on the sentiment analysis task.In questo articolo, presentiamo un sis-tema per la classificazione di soggettivita`e polarita` di tweet in lingua italiana.L’approccio descritto si basa su reti neu-rali. In particolare, utilizziamo un datasetdi 300M di tweet per addestrare una con-volutional neural network. Il sistema e`stato addestrato e valutato sui dati for-niti dagli organizzatori di Sentipolc, taskdi sentiment analysis su Twitter organiz-zato nell’ambito di Evalita 2016..1 IntroductionSentiment analysis is a fundamental problem aim-ing to give a machine the ability to understand theemotions and opinions expressed in a written text.This is an extremely challenging task due to thecomplexity and variety of human language.The sentiment polarity classification task ofEvalItalia-2016 1 (sentipolc) consists of three sub-tasks which cover different aspects of sentimentdetection: T1 : Subjectivity detection: is the tweetsubjective or objective? T2 : Polarity detection: isthe sentiment of the tweet neutral, positive, nega-tive or mixed?1http://www.di.unito.it/˜tutreeb/sentipolc-evalita16/index.htmlT3 : Irony detection: is the tweet ironic?The classic approaches to sentiment analysis usu-ally consist of manual feature engineering and ap-plying some sort of classifier on these features(Liu, 2015). Deep neural networks have showngreat promises at capturing salient features forthese complex tasks (Mikolov et al., 2013b; Sev-eryn and Moschitti, 2015a). Particularly success-ful for sentiment classification were ConvolutionalNeural Networks (CNN) (Kim, 2014; Kalchbren-ner et al., 2014; Severyn and Moschitti, 2015b;Johnson and Zhang, 2015), on which our workbuilds upon. These networks typically have a largenumber of parameters and are especially effectivewhen trained on large amounts of data.In this work, we use a distant supervision ap-proach to leverage large amounts of data in orderto train a 2-layer CNN 2. More specifically, wetrain a neural network using the following three-phase procedure: P1 : creation of word embed-dings for the initialization of the first layer basedon an unsupervised corpus of 300M Italian tweets;P2 : distant supervised phase, where the net-work is pre-trained on a weakly labelled dataset of40M tweets where the network weights and wordembeddings are trained to capture aspects relatedto sentiment; and P3 : supervised phase, wherethe network is trained on the provided supervisedtraining data consisting of 7410 manually labelledtweets.As the three tasks of EvalItalia-2016 are closelyrelated we apply a form of multitask training asproposed by (Collobert et al., 2011), i.e. we trainone CNN for all the tasks simultaneously. Thishas two advantages: i) we need to train onlyone model instead of three models, and ii) theCNN has access to more information which ben-efits the score. The experiments indicate that themulti-task CNN performs better than the single-2We here refer to a layer as one convolutional and onepooling layer.task CNN. After a small bugfix regarding the data-preprocessing our system outperforms all the othersystems in the sentiment polarity task.2 Convolutional Neural NetworksWe train a 2-layer CNN using 9-fold cross-validation and combine the outputs of the 9 re-sulting classifiers to increase robustness. The 9classifiers differ in the data used for the super-vised phase since cross-validation creates 9 differ-ent training and validation sets.The architecture of the CNN is shown in Fig-ure 1 and described in detail below.Sentence model. Each word in the input data isassociated to a vector representation, which con-sists in a d-dimensional vector. A sentence (ortweet) is represented by the concatenation of therepresentations of its n constituent words. Thisyields a matrix S ∈ Rd×n, which is used as inputto the convolutional neural network.The first layer of the network consists of a lookuptable where the word embeddings are representedas a matrix X ∈ Rd×|V|, where V is the vocabu-lary. Thus the i-th column of X represents the i-thword in the vocabulary V .Convolutional layer. In this layer, a set ofm fil-ters is applied to a sliding window of length h overeach sentence. Let S[i:i+h] denote the concatena-tion of word vectors si to si+h. A feature ci isgenerated for a given filter F by:ci :=∑k,j(S[i:i+h])k,j · Fk,j (1)A concatenation of all vectors in a sentence pro-duces a feature vector c ∈ Rn−h+1. The vectorsc are then aggregated over all m filters into a fea-ture map matrix C ∈ Rm×(n−h+1). The filtersare learned during the training phase of the neu-ral network using a procedure detailed in the nextsection.Max pooling. The output of the convolutionallayer is passed through a non-linear activationfunction, before entering a pooling layer. The lat-ter aggregates vector elements by taking the max-imum over a fixed set of non-overlapping inter-vals. The resulting pooled feature map matrix hasthe form: Cpooled ∈ Rm×n−h+1s , where s is thelength of each interval. In the case of overlap-ping intervals with a stride value st, the pooledfeature map matrix has the form Cpooled ∈Rm×n−h+1−sst . Depending on whether the bordersare included or not, the result of the fraction isrounded up or down respectively.Hidden layer. A fully connected hidden layercomputes the transformation α(W∗x+b), whereW ∈ Rm×m is the weight matrix, b ∈ IRm thebias, and α the rectified linear (relu) activationfunction (Nair and Hinton, 2010). The output vec-tor of this layer, x ∈ Rm, corresponds to the sen-tence embeddings for each tweet.Softmax. Finally, the outputs of the hidden layerx ∈ Rm are fully connected to a soft-max regres-sion layer, which returns the class yˆ ∈ [1,K] withlargest probability,yˆ := argmaxjexᵀwj+aj∑Kk=1 exᵀwk+aj, (2)where wj denotes the weights vector of class j andaj the bias of class j.Network parameters. Training the neural net-work consists in learning the set of parametersΘ = {X,F1,b1,F2,b2,W,a}, where X is theembedding matrix, with each row containing thed-dimensional embedding vector for a specificword; Fi,bi(i = {1, 2}) the filter weights and bi-ases of the first and second convolutional layers;W the concatenation of the weights wj for everyoutput class in the soft-max layer; and a the biasof the soft-max layer.Hyperparameters For both convolutional lay-ers we set the length of the sliding window h to5, the size of the pooling interval s is set to 3 inboth layers, where we use a striding of 2 in thefirst layer, and the number of filtersm is set to 200in both convolutional layers.Dropout Dropout is an alternative techniqueused to reduce overfitting (Srivastava et al.,2014). In each training stage individual nodes aredropped with probability p, the reduced neural netis updated and then the dropped nodes are rein-serted. We apply Dropout to the hidden layer andto the input layer using p = 0.2 in both cases.Optimization The network parameters arelearned using AdaDelta (Zeiler, 2012), whichadapts the learning rate for each dimension usingonly first order information. We used the defaulthyper-parameters.Sentence Matrix Convolutional Feature Mappooled repr.Convolutional Feature Mappooled repr.HiddenLayerSoftmaxFigure 1: The architecture of the CNN used in our approach.Twittercat 0.1 0.9 0.3cats 0.3 0.2 0.7cute 0.2 0.3 0.1…Word EmbeddingsAdapted Word Emb.:-):-(word2vecGloVeDistantSupervision2-Layer CNNRaw Tweets(200M)Smiley Tweets(90M) .AnnotatedTweets(18k)UnknownTweetcat 0.1 0.9 0.3cats 0.3 0.2 0.7cute 0.2 0.3 0.1…PredictiveModel3-Phase TrainingApplicationFigure 2: The overall architecture of our 3-phase approach.3 TrainingWe train the parameters of the CNN using thethree-phase procedure as described in the intro-duction. Figure 2 depicts the general flow of thisprocedure.3.1 Three-Phase TrainingPreprocessing We apply standard preprocess-ing procedures of normalizing URLs, hashtagsand usernames, and lowercasing the tweets. Thetweets are converted into a list of indices whereeach index corresponds to the word position in thevocabulary V . This representation is used as in-put for the lookup table to assemble the sentencematrix S.Word Embeddings We create the word embed-dings in phase P1 using word2vec (Mikolov et al.,2013a) and train a skip-gram model on a corpus of300M unlabelled Italian tweets. The window sizefor the skip-gram model is 5, the threshold for theminimal word frequency is set to 20 and the num-ber of dimensions is d = 52 3. The resulting vo-cabulary contains 890K unique words. The wordembeddings account for the majority of networkparameters (42.2M out of 46.6M parameters) andare updated during the next two phases to intro-duce sentiment specific information into the wordembeddings and create a good initialization for theCNN.Distant Supervised Phase We pre-train theCNN for 1 epoch on an weakly labelled datasetof 40M Italian tweets where each tweet containsan emoticon. The label is inferred by the emoti-cons inside the tweet, where we ignore tweets withopposite emoticons. This results in 30M positivetweets and 10M negative tweets. Thus, the classi-fier is trained on a binary classification task.Supervised Phase During the supervised phasewe train the pre-trained CNN with the providedannotated data. The CNN is trained jointly on alltasks of EvalItalia. There are four different binarylabels as well as some restrictions which result in9 possible joint labels (for more details, see Sec-tion 3.2). The multi-task classifier is trained to pre-dict the most likely joint-label.We apply 9-fold cross-validation on the datasetgenerating 9 equally sized buckets. In each roundwe train the CNN using early stopping on the held-out set, i.e. we train it as long as the score im-proves on the held-out set. For the multi-task train-ing we monitor the scores for all 4 subtasks simul-taneously and store the best model for each sub-task. The training stops if there is no improvementof any of the 4 monitored scores.Meta Classifier We train the CNN using 9-foldcross-validation, which results in 9 different mod-els. Each model outputs nine real-value numbers3According to the gensim implementation of word2vecusing d divisible by 4 speeds up the process.yˆ corresponding to the probabilities for each ofthe nine classes. To increase the robustness of thesystem we train a random forest which takes theoutputs of the 9 models as its input. The hyper-parameters were found via grid-search to obtainthe best overall performance over a developmentset: Number of trees (100), maximum depth of theforest (3) and the number of features used per ran-dom selection (5).3.2 DataThe supervised training and test data is providedby the EvalItalia-2016 competition. Each tweetcontains four labels: L1 : is the tweet subjectiveor objective? L2 : is the tweet positive? L3 : is thetweet negative? L4 : is the tweet ironic? Further-more an objective tweet implies that it is neitherpositive nor negative as well as not ironic. Thereare 9 possible combination of labels.To jointly train the CNN for all three tasks T1, T2and T3 we join the labels of each tweet into a sin-gle label. In contrast, the single task training trainsa single model for each of the four labels sepa-rately.Table 1 shows an overview of the data available.Table 1: Overview of datasets provided in EvalItalia-2016.Label Training Set Test SetTotal 7410 2000Subjective 5098 1305Overall Positive 2051 352Overall Negative 2983 770Irony 868 2353.3 Experiments & ResultsWe compare the performance of the multi-taskCNN with the performance of the single-taskCNNs. All the experiments start at the third-phase,i.e. the supervised phase. Since there was nopredefined split in training and development set,we generated a development set by sampling 10%uniformly at random from the provided trainingset. The development set is needed when assess-ing the generalization power of the CNNs and themeta-classifier. For each task we compute the av-eraged F1-score (Barbieri et al., 2016). We presentthe results achieved on the dev-set and the test-setused for the competition. We refer to the set whichwas held out during a cross validation iteration asfold-set.In Table 2 we show the average results obtainedby the 9 CNNs after the cross validation. Thescores show that the CNN is tuned too much to-wards the held-out folds since the scores of theheld-out folds are significantly higher. For exam-ple, the average score of the positivity task is 0.733on the held-out sets but only 0.6694 on the dev-setand 0.6601 on the test-set. Similar differences insores can be observed for the other tasks as well.To mitigate this problem we apply a random for-est on the outputs of the 9 classifiers obtained bycross-validation. The results are shown in Table3. The meta-classifier outperforms the averagescores obtained by the CNNs by up to 2 pointson the dev-set. The scores on the test-set showa slightly lower increase in score. Especially thesingle-task classifier did not benefit from the meta-classifier as the scores on the test set decreased insome cases.The results show that the multi-task classifier out-performs the single-task classifier in most cases.There is some variation in the magnitude of thedifference: the multi-task classifier outperformsthe single-task classifier by 0.06 points in the neg-ativity task in the test-set but only by 0.005 pointsin the subjectivity task.Set Task Subjective Positive Negative IronyFold-Set Single Task 0.723 0.738 0.721 0.646Multi Task 0.729 0.733 0.737 0.657Dev-Set Single Task 0.696 0.650 0.685 0.563Multi Task 0.710 0.669 0.699 0.595Test-Set Single Task 0.705 0.652 0.696 0.526Multi Task 0.681 0.660 0.700 0.540Table 2: Average F1-score obtained after applying cross val-idation.Set Task Subjective Positive Negative IronyDev-Set Single Task 0.702 0.693 0.695 0.573Multi Task 0.714 0.686 0.717 0.604Test-Set Single Task 0.712 0.650 0.643 0.501Multi Task 0.714 0.653 0.713 0.536Table 3: F1-Score obtained by the meta classifier.4 ConclusionIn this work we presented a deep-learning ap-proach for sentiment analysis. We described thethree-phase training approach to guarantee a highquality initialization of the CNN and showed theeffects of using a multi-task training approach. Toincrease the robustness of our system we applieda meta-classifier on top of the CNN. The systemwas evaluated in the EvalItalia-2016 competitionwhere it achieved 1st place in the polarity task andhigh positions on the other two subtasks.ReferencesFrancesco Barbieri, Valerio Basile, Danilo Croce,Malvina Nissim, Nicole Novielli, and Viviana Patti.2016. Overview of the EVALITA 2016 SENTi-ment POLarity Classification Task. In PierpaoloBasile, Anna Corazza, Franco Cutugno, SimonettaMontemagni, Malvina Nissim, Viviana Patti, Gio-vanni Semeraro, and Rachele Sprugnoli, editors,Proceedings of Third Italian Conference on Compu-tational Linguistics (CLiC-it 2016) & Fifth Evalua-tion Campaign of Natural Language Processing andSpeech Tools for Italian. Final Workshop (EVALITA2016). Associazione Italiana di Linguistica Com-putazionale (AILC).Ronan Collobert, Jason Weston, Le´on Bottou, MichaelKarlen, Koray Kavukcuoglu, and Pavel Kuksa.2011. Natural Language Processing (Almost) fromScratch. JMLR, 12:2493−2537.Rie Johnson and Tong Zhang. 2015. Semi-supervisedConvolutional Neural Networks for Text Categoriza-tion via Region Embedding. In NIPS 2015 - Ad-vances in Neural Information Processing Systems28, pages 919–927.Nal Kalchbrenner, Edward Grefenstette, and Phil Blun-som. 2014. A Convolutional Neural Network forModelling Sentences. In ACL - Proceedings of the52nd Annual Meeting of the Association for Com-putational Linguistics, pages 655–665, Baltimore,Maryland, USA, April.Yoon Kim. 2014. Convolutional Neural Networks forSentence Classification. In EMNLP 2014 - Empiri-cal Methods in Natural Language Processing, pages1746–1751, August.Bing Liu. 2015. Sentiment Analysis.Tomas Mikolov, Quoc V Le, and Ilya Sutskever.2013a. Exploiting Similarities among Languagesfor Machine Translation. arXiv, September.Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Cor-rado, and Jeff Dean. 2013b. Distributed represen-tations of words and phrases and their composition-ality. In Advances in neural information processingsystems, pages 3111–3119.Vinod Nair and Geoffrey E Hinton. 2010. Rectifiedlinear units improve restricted boltzmann machines.In Proceedings of the 27th International Conferenceon Machine Learning (ICML-10), pages 807–814.Aliaksei Severyn and Alessandro Moschitti. 2015a.Twitter Sentiment Analysis with Deep Convolu-tional Neural Networks. In 38th InternationalACM SIGIR Conference, pages 959–962, New York,USA, August. ACM.Aliaksei Severyn and Alessandro Moschitti. 2015b.UNITN: Training Deep Convolutional Neural Net-work for Twitter Sentiment Classification. In Se-mEval 2015 - Proceedings of the 9th InternationalWorkshop on Semantic Evaluation.Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky,Ilya Sutskever, and Ruslan Salakhutdinov. 2014.Dropout: A Simple Way to Prevent Neural Networksfrom Overfitting. Journal of Machine Learning Re-search, 15:1929–1958.Matthew D. Zeiler. 2012. ADADELTA: An AdaptiveLearning Rate Method. arXiv, page 6.

In this paper, we propose a clas-sifier for predicting sentiments of Italian Twitter messages. This work builds upon a deep learning approach where we leverage large amounts of weakly labelled data to train a 2-layer convolutional neural network. To train our network we apply a form of multi-task training. Our system participated in the EvalItalia-2016 competition and outperformed all other approaches on the sentiment analysis task

ZHAW Zürcher Hochschule für Angewandte Wissenschaften

https://digitalcollection.zhaw.ch/bitstream/11475/1853/1/paper_032.pdf

Sentiment analysis using convolutional neural networks with multi-task training and distant supervision on italian tweets

Abstract

Similar works

Full text

Available Versions

Crossref

ZHAW digitalcollection

ZHAW digitalcollection

ZHAW Zürcher Hochschule für Angewandte Wissenschaften