Optimal algorithms for universal random number generation from finite memory sources

We study random number generators (RNGs), both in the fixed to variable-length (FVR) and the variable to fixed-length (VFR) regimes, in a universal setting in which the input is a finite memory source of arbitrary order and unknown parameters, with arbitrary input and output (finite) alphabet sizes. Applying the method of types, we characterize essentially unique optimal universal RNGs that maximize the expected output (respectively, minimize the expected input) length in the FVR (respectively, VFR) case. For the FVR case, the RNG studied is a generalization of Elias's scheme, while in the VFR case the general scheme is new. We precisely characterize, up to an additive constant, the corresponding expected lengths, which include second-order terms similar to those encountered in universal data compression and universal simulation. Furthermore, in the FVR case, we consider also a "twice-universal" setting, in which the Markov order k of the input source is also unknown.Comment: To appear in IEEE Transactions on Information Theor

Universal delay-limited simulation

We consider the problem of universal delay–limited simulation of an unknown information source of a certain parametric family (e.g., the family of memoryless sources or Markov sources), given a training sequence from that source and a stream of purely random bits. In the delay–limited setting, the simulation algorithm generates a random sequence sequentially, by delivering one symbol for each training symbol that is made available after a given initial delay, whereas the random bits are assumed to be available on demand. The goal of universal simulation is that the probability law of the generated sequence be identical to that of the training sequence, with minimum mutual information between the random processes generating both sequences. We characterize the optimal delay– limited simulation scheme and upper-bound the expected number of random bits it consumes. As in the non-sequential case, this upper bound is related to the entropy rate of the source

Universal delay-limited simulation

Abstract — Universal, delay–limited simulation of an unknown information source of a certain parametric family (e.g., the family of memoryless sources or Markov sources of a given order), given a training sequence from that source and a stream of purely random bits, is considered. In the delay–limited setting, the simulation algorithm generates a random sequence sequentially, by delivering one symbol for each training symbol that is made available after a given initial delay, whereas the random bits are assumed to be available on demand. The goal of universal simulation is that the probability law of the generated sequence be identical to that of the training sequence, with minimum mutual information between the random processes generating both sequences. In this paper, the optimal universal delay–limited simulation scheme is characterized, and an upper bound on the expected number of random bits it consumes is presented. As in the non-sequential case, the upper bound is related to the entropy rate of the source. The results are extended to a setting of variable delay. Index Terms: Random number generators, random process simulation, universal simulation, mutual information, method of types, enumeration. I