17,294 research outputs found
Reporting Score Distributions Makes a Difference: Performance Study of LSTM-networks for Sequence Tagging
In this paper we show that reporting a single performance score is
insufficient to compare non-deterministic approaches. We demonstrate for common
sequence tagging tasks that the seed value for the random number generator can
result in statistically significant (p < 10^-4) differences for
state-of-the-art systems. For two recent systems for NER, we observe an
absolute difference of one percentage point F1-score depending on the selected
seed value, making these systems perceived either as state-of-the-art or
mediocre. Instead of publishing and reporting single performance scores, we
propose to compare score distributions based on multiple executions. Based on
the evaluation of 50.000 LSTM-networks for five sequence tagging tasks, we
present network architectures that produce both superior performance as well as
are more stable with respect to the remaining hyperparameters.Comment: Accepted at EMNLP 201
Italian Event Detection Goes Deep Learning
This paper reports on a set of experiments with different word embeddings to
initialize a state-of-the-art Bi-LSTM-CRF network for event detection and
classification in Italian, following the EVENTI evaluation exercise. The net-
work obtains a new state-of-the-art result by improving the F1 score for
detection of 1.3 points, and of 6.5 points for classification, by using a
single step approach. The results also provide further evidence that embeddings
have a major impact on the performance of such architectures.Comment: to appear at CLiC-it 201
A Novel Distributed Representation of News (DRNews) for Stock Market Predictions
In this study, a novel Distributed Representation of News (DRNews) model is
developed and applied in deep learning-based stock market predictions. With the
merit of integrating contextual information and cross-documental knowledge, the
DRNews model creates news vectors that describe both the semantic information
and potential linkages among news events through an attributed news network.
Two stock market prediction tasks, namely the short-term stock movement
prediction and stock crises early warning, are implemented in the framework of
the attention-based Long Short Term-Memory (LSTM) network. It is suggested that
DRNews substantially enhances the results of both tasks comparing with five
baselines of news embedding models. Further, the attention mechanism suggests
that short-term stock trend and stock market crises both receive influences
from daily news with the former demonstrates more critical responses on the
information related to the stock market {\em per se}, whilst the latter draws
more concerns on the banking sector and economic policies.Comment: 25 page
Recommended from our members
Verifying baselines for crisis event information classification on Twitter
Social media are rich information sources during and in the aftermath of crisis events such as earthquakes and terrorist attacks. Despite myriad challenges, with the right tools, significant insight can be gained which can assist emergency responders and related applications. However, most extant approaches are incomparable, using bespoke definitions, models, datasets and even evaluation metrics. Furthermore, it is rare that code, trained models, or exhaustive parametrisation details are made openly available. Thus, even confirmation of self-reported performance is problematic; authoritatively determining the state of the art (SOTA) is essentially impossible. Consequently, to begin addressing such endemic ambiguity, this paper seeks to make 3 contributions: 1) the replication and results confirmation of a leading (and generalisable) technique; 2) testing straightforward modifications of the technique likely to improve performance; and 3) the extension of the technique to a novel and complimentary type of crisis-relevant information to demonstrate it’s generalisability
- …