On Asymptotically Optimal Methods of Prediction and Adaptive Coding for Markov Sources

Abstract

AbstractThe problem of predicting a sequence x1, x2, … generated by a discrete source with unknown statistics is considered. Each letter xt+1 is predicted using information on the word x1x2…xt only. In fact, this problem is a classical problem which has received much attention. Its history can be traced back to Laplace. To estimate the efficiency of a method of prediction, three quantities are considered: the precision as given by the Kullback–Leibler divergence, the memory size of the program needed to implement the method on a computer, and the time required, measured by the number of binary operations needed at each time instant. A method is presented for which the memory size and the average time are close to the minimum. The results can readily be translated to results about adaptive coding