Analysing Information Distribution in Complex Systems

Informatsiooniteooria on populaarne tööriist, mida kasutatakse tihti nii lineaarsete kui ka mittelineaarsete seoste tuvastamiseks dünaamilistes komplekssetes süsteemides.Hiljuti välja töötatud osaline informatsiooni dekompositsioon on täiendus harilikule informatsiooniteooriale, mis võimaldab partitsioneerida kahe sisendi ja ühe väljundi vahelise informatsiooni kolmeks komponendiks: unikaalseks, liiaseks ning sünergiliseks informatsiooniks. Nende suuruste praktiliseks arvutamiseks on Tartu Ülikoolis välja töötatud numbriline lahendaja.Käesolev bakalaureusetöö on esimene omalaadne, pakkudes kolme mudeli näolesimesi näiteid osalise informatsiooni dekompositsiooni praktilisest rakendamisest komplekssete süsteemide analüüsimisel. Esiteks leiti, et Isingu mudelis saavutab sünergia maksimumi korratus demagnetiseerunud režiimis enne faasinihet. Teiseks pakuti välja kvantitatiivne, informatsiooni jaotusel põhinev elementaarsete rakuautomaatide karakterisatsioon. Kolmandaks arutleti, et kuigi pärileviga tehisnärvivõrkude analüüsimine ei osutunud osalist informatsiooni dekompositsiooni kasutades viljakaks, võib informatsiooni jaotuse analüüsimine rekurrentsetes tehisnärvivõrkudes pakkuda huvitavamaid tulemusi.Information theory is a popular tool that is often utilized to capture both linearas well as non-linear relationships between different parts of dynamical complex systems. Recently, an extension to classical information theory called partial information decomposition has been developed, which allows one to partition the information that two subsystems have about a third one into unique, redundant and synergetic information terms. To calculate these novel quantities in practice, a numerical estimator has been developed at the University of Tartu.This thesis provides the very first examples of applying partial information de-composition in complex systems research. Three complex systems are empirically analysed in terms of partial information decomposition using the numerical estimator. First, the synergy in the Ising model was found to peak while the system wasstill in the demagnetized, disorder regime. Second, a novel automatic and quantitative characterization of elementary cellular automata based on the information distribution in the automata was obtained. Last, feedforward neural networks were discovered not to be amenable to analysis with the current tools.However, it was argued that analysing recurrent neural networks could yield more interesting results

PLCMOS -- a data-driven non-intrusive metric for the evaluation of packet loss concealment algorithms

Speech quality assessment is a problem for every researcher working on models that produce or process speech. Human subjective ratings, the gold standard in speech quality assessment, are expensive and time-consuming to acquire in a quantity that is sufficient to get reliable data, while automated objective metrics show a low correlation with gold standard ratings. This paper presents PLCMOS, a non-intrusive data-driven tool for generating a robust, accurate estimate of the mean opinion score a human rater would assign an audio file that has been processed by being transmitted over a degraded packet-switched network with missing packets being healed by a packet loss concealment algorithm. Our new model shows a model-wise Pearson's correlation of ~0.97 and rank correlation of ~0.95 with human ratings, substantially above all other available intrusive and non-intrusive metrics. The model is released as an ONNX model for other researchers to use when building PLC systems.Comment: to appear: INTERSPEECH 2023, associated model release: https://aka.ms/PLCMO

Code Llama: Open Foundation Models for Code

We release Code Llama, a family of large language models for code based on Llama 2 providing state-of-the-art performance among open models, infilling capabilities, support for large input contexts, and zero-shot instruction following ability for programming tasks. We provide multiple flavors to cover a wide range of applications: foundation models (Code Llama), Python specializations (Code Llama - Python), and instruction-following models (Code Llama - Instruct) with 7B, 13B and 34B parameters each. All models are trained on sequences of 16k tokens and show improvements on inputs with up to 100k tokens. 7B and 13B Code Llama and Code Llama - Instruct variants support infilling based on surrounding content. Code Llama reaches state-of-the-art performance among open models on several code benchmarks, with scores of up to 53% and 55% on HumanEval and MBPP, respectively. Notably, Code Llama - Python 7B outperforms Llama 2 70B on HumanEval and MBPP, and all our models outperform every other publicly available model on MultiPL-E. We release Code Llama under a permissive license that allows for both research and commercial use

Analyzing Information Distribution in Complex Systems

Information theory is often utilized to capture both linear as well as nonlinear relationships between any two parts of a dynamical complex system. Recently, an extension to classical information theory called partial information decomposition has been developed, which allows one to partition the information that two subsystems have about a third one into unique, redundant and synergistic contributions. Here, we apply a recent estimator of partial information decomposition to characterize the dynamics of two different complex systems. First, we analyze the distribution of information in triplets of spins in the 2D Ising model as a function of temperature. We find that while redundant information obtains a maximum at the critical point, synergistic information peaks in the disorder phase. Secondly, we characterize 1D elementary cellular automata rules based on the information distribution between neighboring cells. We describe several clusters of rules with similar partial information decomposition. These examples illustrate how the partial information decomposition provides a characterization of the emergent dynamics of complex systems in terms of the information distributed across their interacting units