This short paper describes a simple coding technique, Sparse Sequential
Dirichlet Coding, for multi-alphabet memoryless sources. It is appropriate in
situations where only a small, unknown subset of the possible alphabet symbols
can be expected to occur in any particular data sequence. We provide a
competitive analysis which shows that the performance of Sparse Sequential
Dirichlet Coding will be close to that of a Sequential Dirichlet Coder that
knows in advance the exact subset of occurring alphabet symbols. Empirically we
show that our technique can perform similarly to the more computationally
demanding Sequential Sub-Alphabet Estimator, while using less computational
resources.Comment: 7 page

Hutter, Marcus

Veness, Joel

English

arXiv

This short paper describes a simple coding technique, Sparse Sequential Dirichlet
Coding, for multi-alphabet memoryless sources. It is appropriate in situations where
only a small, unknown subset of the possible alphabet symbols can be expected to
occur in any particular data sequence. We provide a competitive analysis which shows
that the performance of Sparse Sequential Dirichlet Coding will be close to that of
a Sequential Dirichlet Coder that knows in advance the exact subset of occurring
alphabet symbols. Empirically we show that our technique can perform similarly to
the more computationally demanding Sequential Sub-Alphabet Estimator, while using
less computational resources

The Australian National University

arXiv:1206.3618v1  [cs.IT]  16 Jun 2012Sparse Sequential Dirichlet CodingJoel Veness† Marcus Hutter‡†University of Alberta, Edmonton, Canada‡Australian National University, Canberra, AustraliaJune 19, 2012AbstractThis short paper describes a simple coding technique, Sparse Sequential DirichletCoding, for multi-alphabet memoryless sources. It is appropriate in situations whereonly a small, unknown subset of the possible alphabet symbols can be expected tooccur in any particular data sequence. We provide a competitive analysis which showsthat the performance of Sparse Sequential Dirichlet Coding will be close to that ofa Sequential Dirichlet Coder that knows in advance the exact subset of occurringalphabet symbols. Empirically we show that our technique can perform similarly tothe more computationally demanding Sequential Sub-Alphabet Estimator, while usingless computational resources.1 IntroductionSuppose we needed to code a sequence of symbols x1:n := x1x2 . . . xn from an unknownalphabet A generated by an unknown memoryless data generating source µ. If we knewan alphabet X such that A ⊆ X , one solution would be to code the sequence using theSequential Dirichlet EstimatorρX (x1:n) :=n∏i=1c(x1:i) +12i+ |X |2− 1, (1)where c(x1:n) :=∑n−1i=1 I[xn = xi], as suggested by Krichevsky and Trofimov [1981]. Thistechnique has the property [Tjalkens et al., 1993] that− log2ρX (x1:n)µ(x1:n)≤|X | − 12log2 n+ |X | − 1. (2)As Equation 2 suggests however, performance of this particular coding technique can bepoor for small values of n when |A| is much less than |X |. This problem occurs often when1using context-based techniques for data compression. This is because, for many contexts,only a small subset of the full alphabet symbols are possible. For example, when modelingEnglish text it is very rare to see any character other than the letter u immediately followingthe letter q. If we knew A in advance, we could code x1:n using ρA, which from Equation 2would of course give a redundancy no greater than|A| − 12log2 n+ |A| − 1. (3)The Sequential Sub-alphabet estimator proposed by Tjalkens et al. [1993] provides anatural Bayesian solution to this dilemma. Rather than using the superset alphabet X , theirtechnique weights over the set of all possible Sequential Dirichlet Estimators whose alphabetsare subsets of X . This leads to an elegant algorithm that has a coding redundancy no morethanlog2 |X |+ log2(|X ||A|)+|A| − 12log2 n + |A|+ 1, (4)when using a uniform prior over sub-alphabets. Unfortunately this method requires O(|X |)time to process each new symbol, and O(|X |) space. This can be prohibitive in situationswhere |X | is large. It would be better if the the time and space complexity were insteaddependent on at most |A|. This paper introduces a simple method, the Sparse SequentialDirichlet Estimator, which achieves similar redundancy properties to the Sequential Sub-alphabet Estimator whilst being able to process each symbol in O(1) time using at mostO(|A|) space.2 PreliminariesWe begin with some notation for data generating sources. An alphabet is a finite, non-emptyset of symbols, which will denote as either A or X . A binary string x1x2 . . . xn ∈ Xn of lengthn is denoted by x1:n. The prefix x1:j of x1:n, j ≤ n, is denoted by x≤j or x<j+1. The emptystring is denoted by ǫ. The concatenation of two strings s and r is denoted by sr.A probabilistic data generating source ρ is defined to be a sequence of probability massfunctions ρn : Xn → [0, 1], for n ∈ N, satisfying the constraint thatρn(x1:n) =∑y∈Xρn+1(x1:ny)for all x1:n ∈ Xn, with base case ρ0(ǫ) = 1. As the meaning is always clear from the argumentto ρ, we drop the subscripts on ρ from here onwards. Under this definition, the conditionalprobability of a symbol xn given previous data x<n is defined as ρ(xn|x<n) := ρ(x1:n)/ρ(x<n)if ρ(x<n) > 0, with the familiar chain rule ρ(x1:n) =∏ni=1 ρ(xi|x<i) now following.A source code c : X ∗ → X ∗ assigns to each possible data sequence x1:n a binary codewordc(x1:n) of length ℓc(x1:n). The typical goal when constructing a source code is to minimizethe lengths of each codeword while ensuring that the original data sequence x1:n is always2recoverable from c(x1:n). Given a data generating source µ, we know from Shannon’s SourceCoding Theorem that the optimal (in terms of expected code length) source code c uses code-words of length − log2 µ(x1:n) bits for all x1:n. This motivates the notion of the redundancyof a source code c given a sequence x1:n, which is defined as rc(x1:n) := ℓc(x1:n)+log2 µ(x1:n).Provided the data generating source is known, near optimal redundancy can essentially beachieved by using arithmetic encoding [Witten et al., 1987]. More precisely, using aµ to de-note the source code obtained by arithmetic coding using probabilistic model µ, the resultantcode lengths are known to satisfyℓaµ(x1:n) < − log2 µ(x1:n) + 2, (5)for all x1:n, which implies that the redundancy is always less than 2. In practice however,the true data generating source µ is typically unknown. The data can still be coded usingarithmetic encoding with an alternate coding distribution ρ, however now we expect to usean extra Eµ [log2 µ(x1:n)/ρ(x1:n)] bits to code the random sequence x1:n ∼ µ. From hereonwards, we restrict our attention to that of specifying a good coding distribution.3 Sparse Sequential Dirichlet DistributionWe now propose an adapted version of the Sequential Dirichlet Distribution, which willuse less computational resources than the Sequential Sub-Alphabet Estimator, while stillperforming well in situations where |A| is much less than |X |.Definition 1. Given an alphabet X , for all n ∈ N and for all x1:n ∈ Xn, the SparseSequential Dirichlet distribution ξ : X ∗ → (0, 1] is defined asξ(x1:n) :=n∏i=1I[c(x1:i) = 0]αi1|X | − |U(x<i)|+ I[c(x1:i) > 0] (1− αi)c(x1:i) +12i+ |U(x<i)|2− 1(6)where c(x1:n) :=∑n−1i=1 I[xn = xi], U(x1:n) := {s ∈ X : c(x1:ns) > 0} and αi :=1ifor i ∈ N.In the above, U(x1:n) is simply the number of distinct symbols occurring in x1:n. Further-more, one can easily verify that ξ is a valid probability measure over finite but arbitrarilylarge strings whose symbols are from the alphabet X .Computational Properties. Given a sequence x1:n ∈ An, ξ(x1:n) can be computed inO(n) time, with O(|A|) space required to store the counts for the seen symbols. Further-more, by using ξ(xn | x<n) = ξ(x1:n)/ξ(x<n) in combination with the chain rule ξ(x1:n) =ξ(xn | x<n)ξ(x<n), each symbol xn+1 can be processed in O(1) time, leading to a straight-forward incremental algorithm. As usual, numerical underflow issues can be addressed bystoring all probability values in log-space.3Analysis. We now show that Sparse Sequential Dirichlet Coding using an alphabet of Xperforms well provided there exists an alphabet A ⊂ X for which Sequential Dirichlet Codingperforms well. Our goal will be to provide a redundancy bound which does not exhibit alinear dependence on |X |.Theorem 1. Given alphabets X and A such that A ⊆ X , for all n ∈ N, for all x1:n ∈ An,we have − log2 ξ(x1:n) ≤ log2 n + |A| log2 |X | − log2 ρA(x1:n).Proof. First note that since |X | ≥ |X | − |U(x<i)| and |U(x1:n)| ≤ |A| for all x1:n ∈ An,ξ(x1:n) ≥n∏i=1I[c(x1:i) = 0]αi1|X |+ I[c(x1:i) > 0] (1− αi)c(x1:i) +12i+ |A|2− 1.Now, noting that αi =1i≥ 12/(i+ |A|/2− 1) for all i ∈ N, we getξ(x1:n) ≥n∏i=1c(x1:i) +12i+ |A|2− 1(I[c(x1:i) = 0]1|X |+ I[c(x1:i) > 0](1− αi)).Since there can be at most |A| new symbols, with the first symbol always being new, and∏1≤i≤n : I[c(x1:i)>0](1− αi) ≥n∏i=2(1− αi),we can concludeξ(x1:n) ≥ |X |−|A|n∏i=1c(x1:i) +12i+ |A|2− 1n∏i=2(1− αi). (7)Now, simplifying the telescoping product and applying the definition of ρA (see Equation 1)to the right-hand side of Equation 7 gives n−1|X |−|A|ρA(x1:n). Hence,− log2 ξ(x1:n) ≤ − log2 n−1|X |−|A|ρA(x1:n) = log2 n+ |A| log2 |X | − log2 ρA(x1:n).Thus, combining Theorem 1, Equation 5 and Equation 3, the overall coding redundancyof the Sparse Sequential Dirichlet Distribution is upper bounded by|A|+ 12log2 n + |A| log2 |X |+ |A|+ 1. (8)Discussion. A comparison of Equation 8 to Equation 2 suggests that the redundancy ofSparse Sequential Dirichlet Coding will be less than Sequential Dirichlet Coding when |A|is much smaller than |X |. Furthermore, by applying the inequalities|A| log2|X ||A|≤ log2(|X ||A|)≤ |A| log2e|X ||A|to bound Equation 4, we can see that our redundancy bound for Sparse Sequential DirichletCoding is competitive with the redundancy bound for the Sequential Sub-alphabet estimatorwhenever |X| is much larger than |A|, and worse when |A| is close to |X |.4Method Mean Min Maxoracle 185.048 12.3267 244.107sdc(A) 193.953 21.368 243.718sdc(X ) 236.343 63.7581 286.108ssd 210.844 23.7755 262.4ssa 212.257 24.4074 262.928ssd - ssa -1.41272 -5.77022 0.366856Table 1: Number of bits needed to encode 100 symbols when |A| = 5 and |X | = 26.4 Numerical ExperimentsWe now present some numerical results for Sparse Sequential Dirichlet Coding, by comparingand contrasting our technique using the experimental framework described below.Experimental Setup. Each different experiment consisted of evaluating the performanceof 5 different coding distributions on synthetically generated data, for various choices of Aand X . The first technique, oracle, used the true underlying data generating distributionto code the data. This is of course the optimal coding distribution in expectation, anda natural baseline. The second and third techniques, sdc(A) and sdc(X ), refer to usingSequential Dirichlet Coding using the alphabets A and X respectively. These two methodsallow us to measure the impact of knowing and not knowing A in advance. ssd refersto our Sparse Sequential Dirichlet Coding technique. Finally, ssa refers to the SequentialSub-Alphabet technique of Tjalkens et al. [1993].To evaluate each particular combination ofA and X , 100,000 parameter vectors, ai ∈ R|A|for 1 ≤ i ≤ 100, 000, were sampled from a Symmetric Dirichlet Distribution using a con-centration parameter of 1.0. These ai were used to define a set of Categorical Distributionsover the symbols in A. Each Categorical Distribution was used once to generate a datasequence of 100 independent and identically distributed random symbols, which were thencoded using each of the methods. The mean, min and max performance, measured in bits,for each different coding distribution on the generated data sequences was then summarisedin Tables 1, 2 and 3. Additionally, the last line of each table measured how many extra bitsSparse Sequential Dirichlet Coding needed compared with Sequential Sub-Alphabet Coding.Results. Table 1 and Table 2 compare the relative coding performance of Sparse SequentialDirichlet Coding when |A| is much less than |X |. In both situations we see that the SparseSequential Dirichlet technique is on average slightly superior to the Sequential Sub-Alphabetmethod, and never worse by more than 1.32 bits. Both techniques performed significantlybetter than the Sequential Dirichlet Coding method which used the alphabet X . Thisis consistent with the redundancy bounds we presented earlier. Lastly, Table 3 gives anexample of what can happen when the sparsity assumption doesn’t apply. Here we see thatthe Sparse Sequential Dirichlet method is outperformed by all other techniques, though not5Method Mean Min Maxoracle 278.363 131.529 340.359sdc(A) 293.969 146.716 349.882sdc(X ) 492.284 345.031 548.197ssd 349.169 181.365 412.766ssa 350.473 187.604 411.656ssd - ssa -1.30374 -7.4791 1.3234Table 2: Number of bits needed to encode 100 symbols when |A| = 10 and |X | = 256.Method Mean Min Maxoracle 360.053 234.325 422.467sdc(A) 382.911 258.392 440.005sdc(X ) 396.527 272.007 453.62ssd 410.573 277.942 476.754ssa 397.344 271.68 456.234ssd - ssa 13.2292 0.446927 20.5248Table 3: Number of bits needed to encode 100 symbols when |A| = 18 and |X | = 26.by a large margin.Discussion. In light of its superior computational properties, our results suggest that theSparse Sequential Dirichlet technique is a good alternative to the Sequential Sub-Alphabetmethod whenever |A| is much less than |X |. If issues of computation or limited memoryare not an issue, the Sequential Sub-Alphabet method is to be preferred due to its betterperformance when |A| is not much less than |X |.5 ConclusionThis short paper has described a simple and efficient coding technique for multi-alphabetmemoryless sources. It provably works well when only a small subset of possible alphabetsymbols are expected to occur in any given data sequence. As future work, it would beinteresting to explore the applicability of this technique as a building block within moresophisticated context modeling techniques.Acknowledgements The authors would like to thank Kee Siong Ng and Marc Bellemarefor comments that helped improve this paper.6ReferencesR. Krichevsky and V. Trofimov. The performance of universal encoding. Information Theory,IEEE Transactions on, 27(2):199–207, 1981.Tjalling J. Tjalkens, Yuri M. Shtarkov, and Frans M. J. Willems. Sequential weightingalgorithms for multialphabet sources. In 6th Joint Swedish-Russian Int. Worksh. Inform.Theory, pages 22–27, 1993.Ian H. Witten, Radford M. Neal, and John G. Cleary. Arithmetic coding for data compres-sion. Commun. ACM, 30:520–540, June 1987. ISSN 0001-0782.7

Sparse Sequential Dirichlet Coding

Abstract

Similar works

Full text

Available Versions

The Australian National University