Our goal is to link two different formal notions of complexity: the complexity classes defined by Formal Language Theory (FLT)—in particular, the Sub-regular Hierarchy (Rogers et al., 2013; Lai, 2015; Heinz, 2018)—and Statistical Com- plexity Theory (Feldman and Crutchfield, 1998; Crutchfield and Marzen, 2015). The motivation for exploring this connection is that factors involving memory resources have been hypothesized to explain why phonological processes seem to inhabit the Sub-regular Hierarchy, and Statistical Complexity Theory gives an information-theoretic characterization of memory use. It is currently not known whether statistical complexity and FLT define equivalent complexity classes, or whether statistical complexity cross-cuts the usual FLT hierarchies. Our work begins to bridge the gap between FLT and Information Theory by presenting characterizations of certain Sub-regular languages in terms of statistical complexit

Dai, Huteng

Futrell, Richard

English

ScholarWorks@UMass Amherst

Proceedings of the Society for Computation in Linguistics Volume 3 Article 46 2020 Information-theoretic Characterization of the Sub-regular Hierarchy Huteng Dai Rutgers University, huteng.dai@rutgers.edu Richard Futrell University of California, Irvine, rfutrell@uci.edu Follow this and additional works at: https://scholarworks.umass.edu/scil  Part of the Computational Linguistics Commons Recommended Citation Dai, Huteng and Futrell, Richard (2020) "Information-theoretic Characterization of the Sub-regular Hierarchy," Proceedings of the Society for Computation in Linguistics: Vol. 3 , Article 46. DOI: https://doi.org/10.7275/c521-qn83 Available at: https://scholarworks.umass.edu/scil/vol3/iss1/46 This Extended Abstract is brought to you for free and open access by ScholarWorks@UMass Amherst. It has been accepted for inclusion in Proceedings of the Society for Computation in Linguistics by an authorized editor of ScholarWorks@UMass Amherst. For more information, please contact scholarworks@library.umass.edu. Information-theoretic characterization of the Sub-regular HierarchyHuteng DaiRutgers Universityhuteng.dai@rutgers.eduRichard FutrellUniversity of California, Irvinerfutrell@uci.eduOur goal is to link two different formal notionsof complexity: the complexity classes defined byFormal Language Theory (FLT)—in particular,the Sub-regular Hierarchy (Rogers et al., 2013;Lai, 2015; Heinz, 2018)—and Statistical Com-plexity Theory (Feldman and Crutchfield, 1998;Crutchfield and Marzen, 2015). The motivationfor exploring this connection is that factors involv-ing memory resources have been hypothesized toexplain why phonological processes seem to in-habit the Sub-regular Hierarchy, and StatisticalComplexity Theory gives an information-theoreticcharacterization of memory use. It is currently notknown whether statistical complexity and FLT de-fine equivalent complexity classes, or whether sta-tistical complexity cross-cuts the usual FLT hierar-chies. Our work begins to bridge the gap betweenFLT and Information Theory by presenting char-acterizations of certain Sub-regular languages interms of statistical complexity.Statistical complexity theory. Statistical com-plexity theory deals with stochastic processes:probabilistic models of infinitely-long sequences.For a process X generating sequences of sym-bols indexed as . . . Xt−2, Xt−1, Xt, Xt+1, . . . , wedefine the notation←−X (“the past”) to mean. . . Xt−2, Xt−1, and−→X (“the future”) to meanXt, Xt+1, . . . .The statistical complexity of a stochastic pro-cess is the minimal amount of information aboutthe past required to faithfully reproduce the fu-ture. Suppose that we want to simulate a stochasticprocess by generating each symbol based on somememory representationM of the past, and that wewant to find a memory representationM that sim-ulates the process as well as possible while hav-ing minimal information content, measured in bits.This quantity of minimal information is called sta-tistical complexity. Formally, the statistical com-plexity S of a process X is the minimum entropyof a memory representationM that perfectly sim-ulates the process:S ≡ minM :DKL[−→X |←−X ||−→X |M ]=0H[M ], (1)whereH[M ] is the entropy of the random variableM :H[M ] ≡ −∑xpM (x) log pM (x), (2)and DKL[·||·] is conditional Kullback-Leibler di-vergence (see Cover and Thomas, 2006), which iszero for identical conditional distributions. There-fore, Eq. 1 indicates the minimum entropy of anymemory representationM subject to the constraintthat M must allow us to generate a distributionover future sequences−→X which is identical to thedistribution we would have generated given thepast←−X .Further insight comes from considering the dif-ferent factors that contribute to statistical com-plexity. Using information-theoretic identities, webreak the statistical complexity into two terms:S = H[M ] = I[M :−→X ] +H[M |−→X ]= I[←−X :−→X ]︸ ︷︷ ︸E+H[M |−→X ]︸ ︷︷ ︸C,where I[· : ·] is mutual information, the amountof information in one random variable about an-other. The term E is called excess entropy andquantifies the amount of information in the pastwhich is useful for predicting the future. The termC is called crypticity and quantifies the amountof information stored inM which does not end upbeing useful for predicting the future.These quantities are easily understood in termsof memory resources used for incremental lan-guage production and comprehension. Statisti-cal complexity measures memory load or storage445Proceedings of the Society for Computation in Linguistics (SCiL) 2020, pages 445-448.New Orleans, Louisiana, January 2-5, 2020cost; it can be finite even for non-finite-state pro-cesses, as long as the sum in Eq. 2 converges.Excess entropy measures integration cost: it sayshow many bits of information from the past areused when processing the future. Crypticity is thedifference between statistical complexity and ex-cess entropy, and measures the amount of informa-tion stored in the minimal memory representationM which does not ultimately end up being used topredict the future.In order to study memory efficiency, we usethese quantities to define an efficiency metric, theE/S ratio, which is excess entropy divided by sta-tistical complexity. TheE/S ratio tells the propor-tion of bits stored in memory which end up beinguseful for predicting the future.Preliminaries. We study Sub-regular languagesdefined using Probabilistic Deterministic Finite-state Automata (PDFAs). A PDFA is character-ized by a set of internal states Q, an alphabet Σ,an emission distribution O of symbols ∈ Σ con-ditional on a state ∈ Q, a transition functionT : Q × Σ → Q defining which state the ma-chine transitions into after emitting a symbol, anddistinguished initial and final states. In a PDFA,the transition function T is deterministic; in a gen-eral Probabilistic Finite-state Automaton (PFA), itmay be stochastic, in which case we have a transi-tion distribution rather than a transition function.Our indexing convention is: at time t, the PDFAis in state qt; it generates symbol xt before transi-tioning into the next state qt+1. The time indexingconvention is shown in Figure 1.qt−2 qt−1 qt qt+1 qt+2Xt−2 Xt−1 Xt Xt+1−→X , Future←−X , PastFigure 1: Time-indexing conventions for a finite-statemachine.q0 q1b : 1/4c : 1/4# : 1/4 a : 1/3a : 1/4c : 1/3# : 1/3Figure 2: SL2 PDFA of ¬ab, Σ = {a, b, c}We use the following construction to gener-ate a stationary ergodic stochastic process from aPDFA: whenever the PDFA emits an end-of-wordsymbol #, it always transitions back into the ini-tial state. The resulting infinite stream of sym-bols is amenable to analysis using statistical com-plexity theory. In the literature on statistical com-plexity, a PDFA of this form is called a unifilarHidden Markov Model (Travers and Crutchfield,2011, unifilar HMM).Below, we describe how to calculate S,E, and C from the minimal trimmed PDFA(Heinz and Rogers, 2010) for Strictly k-Local(SLk) languages.Statistical complexity. For a unifilar HMM,the statistical complexity reduces to the entropyof the stationary distribution over internal states(Travers and Crutchfield, 2011). To get the sta-tionary distribution over internal states Q, we firstconstruct a state transition matrix: a stochasticmatrix whose entries represent the probability ofgoing into state qt+1 after being in state qt. For ageneral PFA, the entries of this matrix are given bymarginalizing over the emission distribution O:p(qt+1|qt) =∑xt∈ΣpO(xt|qt)pT (qt+1|xt, qt),where pT is the probability of transitioning intostate qt+1 after generating symbol xt from stateqt. In a PDFA, this probability is given by the de-terministic transition function T , so the transitionprobability pT reduces to a Kronecker delta func-tion:pT (qt+1|xt, qt) = δqt+1=T (xt,qt).Finally, the stationary distribution over states Q isgiven by the left eigenvector of the state transitionmatrix associated with eigenvalue 1.In general, the statistical complexity of a pro-cess depends on the minimal number of states re-quired to represent the process as a PDFA. Foran SLk language, statistical complexity is upperbounded as S ≤ (k − 1) log |Σ|.Excess entropy. For SLk languages,E = I[Xt−k+1, . . . , Xt−1 : Xt, . . . , Xt+k−2].In the case of SL2 languages, we computeE by constructing a symbol transition ma-trix, a stochastic matrix whose entries represent446p(xt+1|xt), marginalizing over qt and qt+1. Wealso need the stationary distribution over symbols,derived from the symbol transition matrix by thesame procedure as above.Crypticity. Crypticity C = S − E. In gen-eral, crypticity is bounded above by the uncer-tainty about the emitting state given a symbol:C ≤ H[Qt|Xt],with equality iff X is an SL2 language.Sub-regular Hierarchy. We consider two rela-tional structures, namely the successor (+1) andprecedence (<) relations. Languages with suc-cessor relation keep track of k-long sub-stringsof the input, such as {aa, ab, ac, ba, . . .} in anSL2 language. On the other hand, languageswith precedence relation keep track of k-long sub-sequences, such as {a . . . a, a . . . b, . . .} in an SP2language. Different sub-regular languages corre-spond to distinct PDFAs. For each relational struc-ture, languages with the higher logical power areconsidered to be more expressive. For example,SL languages are a subset of locally testable (LT)languages. The subset relations are indicated bylines connecting higher and lower regions in Fig-ure 3.RelationalstructuresLogical powerConjunctions ofNegative LiteralsPropositionalFirst OrderMonadicSecond Order+1 <SL (0.11) SP (0.18)LT (0.40) PT (0.20)LTT (0.43)SF (?)RegularFigure 3: Sub-regular Hierarchy, with E/S ratios cal-culated from the examples in the text.Table 1 shows calculated statistical complex-ity, excess entropy, and crypticity for the minimaltrimmed PDFAs of example languages in the Sub-regular Hierarchy, including Strictly Local (SL),Locally Testable (LT), Locally Threshold Testable(LTT), Strictly Piecewise (SP), Piecewise Testable(PT).The information quantities align with the hy-pothesis in FLT literature: the languages whichSL2 LT2 LTT2 SP2 PT2Statistical complexity 0.97 1.53 1.94 0.99 1.53Excess entropy 0.09 ≥0.61 ≥0.83 ≥0.18 ≥0.30Crypticity 0.75 ≤0.91 ≤1.10 ≤0.80 ≤1.22E/S ratio 0.11 ≥0.40 ≥0.43 ≥0.18 ≥0.20Table 1: Information quantities for PDFAs shown infigures. SL2 = Figure 2; LT2 = Figure 4; LTT2 = Fig-ure 5, SP2 = Figure 6; PT2 = Figure 7. Quantitiesmarked with ≤ or ≥ are bounds based on Markov ap-proximations.are more expressive have higher memory storagerequirements. E/S ratios characterize the sub-set relation in the Sub-regular Hierarchy, for bothsuccessor and precedence relations: the higher re-gions in the hierarchy have higher amount of E/Sratio, as illustrated in Figure 3.q0 q1 q2b : 1/3c : 1/3 a : 1/3a : 1/4b : 1/4c : 1/4a : 1/3c : 1/3b : 1/3# : 1/4Figure 4: LT2 PDFA of Some-ab, Σ = {a, b, c}q0q1 q2q3b : 1/3c : 1/3a : 1/3c : 1/3b : 1/4c : 1/4a : 1/3c : 1/3a : 1/3b : 1/3a : 1/4# : 1/4# : 1/3Figure 5: LTT2 PDFA of One-ab, Σ = {a, b, c}q0 q1b : 1/4c : 1/4# : 1/4a : 1/3c : 1/3a : 1/4# : 1/3Figure 6: SP2 PDFA of ¬a . . . b, Σ = {a, b, c}The information-theoretic characterization illu-minates the comparison across relational struc-tures. For example, SL and SP languages cor-respond to different types of phonotactics: SL447q0 q1 q2b : 1/3c : 1/3a : 1/3c : 1/3a : 1/4b : 1/4c : 1/4a: 1/3 b: 1/3#: 1/4Figure 7: PT2 PDFA of Some-a . . . b, Σ = {a, b, c}only describes local phonotactics, while SP corre-sponds to patterns of long-distance agreement. Inthe examples we have examined, SL and SP havesimilar information quantities when they sharethe same k-factor. We conjecture that SLk andSPk languages have similar memory efficiency be-cause they are both described by Conjunction ofNegative Literals (McNaughton and Papert, 1971,CNL; the combination of ¬ and ∧).Conclusion. We have investigated whether thereis a coherent relationship between complexitymetrics calculated using Statistical ComplexityTheory on one hand, and the Sub-regular hierar-chy of languages on the other hand. Our prelim-inary results, based on example languages repre-senting a number of Sub-regular classes, suggestthat increasing logical power corresponds to in-creasing information-theoretic memory storage re-quirements. Our current study is limited in thatwe have only calculated complexity metrics for se-lected examples of each language class. Futurework will work to establish general formal rela-tionships between language classes and statisticalcomplexity.Regardless of whether statistical complexityturns out to map cleanly onto FLT hierarchies,we believe it provides a promising frameworkfor characterizing bounds on complexity of hu-man languages and phonotactics in particular. Thetheory of statistical complexity provides a clearway to quantify and reason about memory stor-age cost and memory integration cost in a highlygeneral information-theoretic setting. Thereforeit is entirely reasonable to expect that there maybe bounds on the complexity of linguistic sub-systems, defined using the language of statisticalcomplexity.In this connection, we note that statistical com-plexity depends on a number of factors that arenot usually relevant in FLT, such as the transi-tion probabilities and number of states in a PDFA.Although these factors are not relevant in FLT,they may nonetheless be relevant for characteriz-ing constraints on the phonology and phonotacticsof human languages. By characterizing complex-ity using Statistical Complexity Theory, we cantake these factors into account in a principled way.Acknowledgement. We thank Jim Crutchfield,Jeff Heinz, Adam Jardine, and anonymous review-ers for their comments and insights.ReferencesThomas M. Cover and J.A. Thomas. 2006. Elements ofInformation Theory. John Wiley & Sons, Hoboken,NJ.James P Crutchfield and Sarah Marzen. 2015. Signa-tures of infinity: Nonergodicity and resource scal-ing in prediction, complexity, and learning. PhysicalReview E, 91(5):050106.David P. Feldman and James P. Crutchfield. 1998.Measures of statistical complexity: Why? PhysicsLetters A, 238(4-5):244–252.Jeffrey Heinz. 2018. The computational nature ofphonological generalizations. Phonological Typol-ogy, Phonetics and Phonology, pages 126–195.Jeffrey Heinz and James Rogers. 2010. Estimatingstrictly piecewise distributions. In Proceedings ofthe 48th Annual Meeting of the Association forComputational Linguistics, pages 886–896, Upp-sala, Sweden. Association for Computational Lin-guistics.Regine Lai. 2015. Learnable vs. unlearnable harmonypatterns. Linguistic Inquiry, 46(3):425–451.Robert McNaughton and Seymour A Papert. 1971.Counter-Free Automata (MIT research monographno. 65). The MIT Press.James Rogers, Jeffrey Heinz, Margaret Fero, JeremyHurst, Dakotah Lambert, and Sean Wibel. 2013.Cognitive and sub-regular complexity. In Formalgrammar, pages 90–108. Springer.Nicholas F. Travers and James P. Crutchfield. 2011.Asymptotic synchronization for finite-state sources.Journal of Statistical Physics, 145(5):1202–1223.448

Information-theoretic Characterization of the Sub-regular Hierarchy

https://scholarworks.umass.edu/cgi/viewcontent.cgi?article=1141&amp;context=scil

Information-theoretic Characterization of the Sub-regular Hierarchy

Abstract

Similar works

Full text

Available Versions

ScholarWorks@UMass Amherst