Asymmetric Classification: Constructing Channels from Sources in Real-Time

In this note we describe a system, developed under contract for major newspaper and newswire publishers (and currently deployed commercially), that constructs "topic channels" in real-time by simultaneously assigning one or more category codes to newspaper stories immediately upon publication and to newswire stories "on the run". The sources are diverse, but the category code taxonomy (developed by human domain experts) is unified. The system, named "Cogent" (COdinG ENgine Technology), is asymmetric in the following two senses: (1) speed of classification is far more important than speed of training, and (2) precision is far more important than recall. The last two statements must be taken in the extreme: a failure to classify in under a few milliseconds, or the inclusion of an irrelevant story in a channel, are both considered system failures of the first magnitude, not mere "glitches". The approach is statistical, and the thresholdadjustment used to favor precision over recall has direct interpretation as a likelihood ratio. Novel aspects include a new feature selection algorithm that drastically reduces dimensionality, and the use of publisherassigned metadata as features. Comparison with published results indicate that Cogent performs as well as the best available text categorizers for newswires but uses substantially fewer features and computational resources during classification

An Efficient Algorithm to Determine Stochastic Dominance Admissible Sets

Stochastic Dominance (SD) rules are playing an increasingly prominent role in the theory of choice under uncertainty. Its application areas include stock selection, capital budgeting, etc. The theory is important because it generates decision rules which are more generally applicable to these problems than are the traditional two parameter (mean-variance) rules employed in much of financial decision making. While they are theoretically sound, the SD rules are, until now, hard to implement because they require comparisons of probability distributions over their entire ranges. In this paper, we develop an algorithm that should remedy this situation. It exploits recent theoretical results from the Stochastic Dominance literature as well as several computational techniques to efficiently determine the SD admissible set of alternatives, which contains the optimal choices for all decision makers whose preferences satisfy reasonable economic criteria. As compared with the fastest code currently available, an implementation of our algorithm significantly reduces the computational time required to solve a problem of considerable size. These results indicate that, as a management tool, this algorithm can be applied to choice problems not previously thought solvable. For example, in the portfolio choice problem, which has an infinite choice set, the algorithm can provide reasonable approximations to the true set of optimal choices via the use of a suitably fine enough grid on the space of portfolios.stochastic dominance, algorithm