How to evaluate sentiment classifiers for Twitter time-ordered data?

Social media are becoming an increasingly important source of information about the public mood regarding issues such as elections, Brexit, stock market, etc. In this paper we focus on sentiment classification of Twitter data. Construction of sentiment classifiers is a standard text mining task, but here we address the question of how to properly evaluate them as there is no settled way to do so. Sentiment classes are ordered and unbalanced, and Twitter produces a stream of time-ordered data. The problem we address concerns the procedures used to obtain reliable estimates of performance measures, and whether the temporal ordering of the training and test data matters. We collected a large set of 1.5 million tweets in 13 European languages. We created 138 sentiment models and out-of-sample datasets, which are used as a gold standard for evaluations. The corresponding 138 in-sample datasets are used to empirically compare six different estimation procedures: three variants of cross-validation, and three variants of sequential validation (where test set always follows the training set). We find no significant difference between the best cross-validation and sequential validation. However, we observe that all cross-validation variants tend to overestimate the performance, while the sequential methods tend to underestimate it. Standard cross-validation with random selection of examples is significantly worse than the blocked cross-validation, and should not be used to evaluate classifiers in time-ordered data scenarios

Distributed Dynamic Spectrum Access in Multichannel Random Access Networks with Selfish Users

Dynamic spectrum allocation schemes enable users to share spectrum resources by exploiting the variations in spectrum demand over time and space. Performing dynamic spectrum allocation centrally can be prohibitively complex. Therefore distributed schemes in which users can access the available channels independently may be preferable to centralized allocation. However, in distributed dynamic spectrum access, the lack of central coordination makes it difficult to utilize the system resources efficiently. Furthermore, if some or all of the users decide to deviate selfishly from the commonly agreed access procedure, this may have a decisive effect on system performance. In this paper we investigate the effect of incomplete information and selfish behavior on system performance in wireless access systems. We extend previous work by studying a distributed multichannel wireless random access system. Using a game-theoretic approach, we analyze the behavior of users in the selfish system and derive the transmission strategies at the Nash equilibrium. Our results show that lack of information leads to substantial degredation in performance of cooperative systems. We also show that there is a large incentive for selfish behavior in such cooperative systems. Selfish behavior of all users, however, causes further performance degradation, particularly in high load settings.QC 20111208. © 2010 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. QC 20111207</p