477 research outputs found
On Hidden Markov Processes with Infinite Excess Entropy
We investigate stationary hidden Markov processes for which mutual
information between the past and the future is infinite. It is assumed that the
number of observable states is finite and the number of hidden states is
countably infinite. Under this assumption, we show that the block mutual
information of a hidden Markov process is upper bounded by a power law
determined by the tail index of the hidden state distribution. Moreover, we
exhibit three examples of processes. The first example, considered previously,
is nonergodic and the mutual information between the blocks is bounded by the
logarithm of the block length. The second example is also nonergodic but the
mutual information between the blocks obeys a power law. The third example
obeys the power law and is ergodic.Comment: 12 page
Mixing, Ergodic, and Nonergodic Processes with Rapidly Growing Information between Blocks
We construct mixing processes over an infinite alphabet and ergodic processes
over a finite alphabet for which Shannon mutual information between adjacent
blocks of length grows as , where . The processes
are a modification of nonergodic Santa Fe processes, which were introduced in
the context of natural language modeling. The rates of mutual information for
the latter processes are alike and also established in this paper. As an
auxiliary result, it is shown that infinite direct products of mixing processes
are also mixing.Comment: 21 page
On the Vocabulary of Grammar-Based Codes and the Logical Consistency of Texts
The article presents a new interpretation for Zipf-Mandelbrot's law in
natural language which rests on two areas of information theory. Firstly, we
construct a new class of grammar-based codes and, secondly, we investigate
properties of strongly nonergodic stationary processes. The motivation for the
joint discussion is to prove a proposition with a simple informal statement: If
a text of length describes independent facts in a repetitive way
then the text contains at least different words, under
suitable conditions on . In the formal statement, two modeling postulates
are adopted. Firstly, the words are understood as nonterminal symbols of the
shortest grammar-based encoding of the text. Secondly, the text is assumed to
be emitted by a finite-energy strongly nonergodic source whereas the facts are
binary IID variables predictable in a shift-invariant way.Comment: 24 pages, no figure
Variable-Length Coding of Two-Sided Asymptotically Mean Stationary Measures
We collect several observations that concern variable-length coding of
two-sided infinite sequences in a probabilistic setting. Attention is paid to
images and preimages of asymptotically mean stationary measures defined on
subsets of these sequences. We point out sufficient conditions under which the
variable-length coding and its inverse preserve asymptotic mean stationarity.
Moreover, conditions for preservation of shift-invariant -fields and
the finite-energy property are discussed and the block entropies for stationary
means of coded processes are related in some cases. Subsequently, we apply
certain of these results to construct a stationary nonergodic process with a
desired linguistic interpretation.Comment: 20 pages. A few typos corrected after the journal publicatio
Universal Coding and Prediction on Martin-L\"of Random Points
We perform an effectivization of classical results concerning universal
coding and prediction for stationary ergodic processes over an arbitrary finite
alphabet. That is, we lift the well-known almost sure statements to statements
about Martin-L\"of random sequences. Most of this work is quite mechanical but,
by the way, we complete a result of Ryabko from 2008 by showing that each
universal probability measure in the sense of universal coding induces a
universal predictor in the prequential sense. Surprisingly, the effectivization
of this implication holds true provided the universal measure does not ascribe
too low conditional probabilities to individual symbols. As an example, we show
that the Prediction by Partial Matching (PPM) measure satisfies this
requirement. In the almost sure setting, the requirement is superfluous.Comment: 12 page
- …