Fast and Extensible Phrase Scoring for Statistical Machine Translation

Abstract

Existing tools for generating phrase tables for phrase-based Statistical Machine Translation (SMT) are generally optimised towards low memory use to allow processing of large corpora with limited memory. Whilst being a reasonable design choice, this approach does not make optimal use of resources when the sufficient memory is available. We present memscore, a new open-source tool to score phrases in memory. Besides acting as a faster drop-in replacement for existing software, it implements a number of standard smoothing techniques and provides a platform for easy experimentation with new scoring methods.In Proceedings of MT Marathon 2010, Dublin (http://www.mtmarathon2010.info/web/Welcome.html).</p