The Tsallis entropy and the Shannon entropy of a universal probability

We study the properties of Tsallis entropy and Shannon entropy from the point of view of algorithmic randomness. In algorithmic information theory, there are two equivalent ways to define the program-size complexity K(s) of a given finite binary string s. In the standard way, K(s) is defined as the length of the shortest input string for the universal self-delimiting Turing machine to output s. In the other way, the so-called universal probability m is introduced first, and then K(s) is defined as -log_2 m(s) without reference to the concept of program-size. In this paper, we investigate the properties of the Shannon entropy, the power sum, and the Tsallis entropy of a universal probability by means of the notion of program-size complexity. We determine the convergence or divergence of each of these three quantities, and evaluate its degree of randomness if it converges.Comment: 5 pages, to appear in the Proceedings of the 2008 IEEE International Symposium on Information Theory, Toronto, ON, Canada, July 6 - 11, 200

Paradigms of Cognition

An abstract, quantitative theory which connects elements of information —key ingredients in the cognitive proces—is developed. Seemingly unrelated results are thereby unified. As an indication of this, consider results in classical probabilistic information theory involving information projections and so-called Pythagorean inequalities. This has a certain resemblance to classical results in geometry bearing Pythagoras’ name. By appealing to the abstract theory presented here, you have a common point of reference for these results. In fact, the new theory provides a general framework for the treatment of a multitude of global optimization problems across a range of disciplines such as geometry, statistics and statistical physics. Several applications are given, among them an “explanation” of Tsallis entropy is suggested. For this, as well as for the general development of the abstract underlying theory, emphasis is placed on interpretations and associated philosophical considerations. Technically, game theory is the key tool

Combining Data-Driven 2D and 3D Human Appearance Models

Detailed 2D and 3D body estimation of humans has many applications in our everyday life: interaction with machines, virtual try-on of fashion or product adjustments based on a body size estimate are just some examples. Two key components of such systems are: (1) detailed pose and shape estimation and (2) generation of images. Ideally, they should use 2D images as input signal so that they can be applied easily and on arbitrary digital images. Due to the high complexity of human appearance and the depth ambiguities in 2D space, data driven models are the tool at hand to design such methods. In this work, we consider two aspects of such systems: in the first part, we propose general optimization and implementation techniques for machine learning models and make them available in the form of software packages. In the second part, we present in multiple steps, how the detailed analysis and generation of human appearance based on digital 2D images can be realized. We work with two machine learning methods: Decision Forests and Artificial Neural Networks. The contribution of this thesis to the theory of Decision Forests consists of the introduction of a generalized entropy function that is efficient to evaluate and tunable to specific tasks and allows us to establish relations to frequently used heuristics. For both, Decision Forests and Neural Networks, we present methods for implementation and a software package. Existing methods for 3D body estimation from images usually estimate the 14 most important, pose defining points in 2D and convert them to a 3D `skeleton'. In this work we show that a carefully crafted energy function is sufficient to recover a full 3D body shape automatically from the keypoints. In this way, we devise the first fully automatic method estimating 3D body pose and shape from a 2D image. While this method successfully recovers a coarse 3D pose and shape, it is still a challenge to recover details such as body part rotations. However, for more detailed models, it would be necessary to annotate data with a very rich set of cues. This approach does not scale to large datasets, since the effort per image as well as the required quality could not be reached due to how hard it is to estimate the position of keypoints on the body surface. To solve this problem, we develop a method that can alternate between optimizing the 2D and 3D models, improving them iteratively. The labeling effort for humans remains low. At the same time, we create 2D models reasoning about factors more items than existing methods and we extend the 3D pose and body shape estimation to rotation and body extent. To generate images of people, existing methods usually work with 3D models that are hard to adjust and to use. In contrast, we develop a method that builds on the possibilities for automatic 3D body estimation: we use it to create a dataset of 3D bodies together with 2D clothes and cloth segments. With this information, we develop a data driven model directly producing 2D images of people. Only the broad interplay of 2D and 3D body and appearance models in different forms makes it possible to achieve a high level of detail for analysis and generation of human appearance. The developed techniques can in principle also be used for the analysis and generation of images of other creatures and objects.Detaillierte 2D und 3D Körperschätzung von Menschen hat vielfältige Anwendungen in unser aller Alltag: Interaktion mit Maschinen, virtuelle "Anprobe" von Kleidung oder direkte Produktanpassungen durch Schätzung der Körpermaße sind nur einige Beispiele. Dazu sind Methoden zur (1) detaillierten Posen- und Körpermaßschätzung und (2) Körperdarstellung notwendig. Idealerweise sollten sie digitale 2D Bilder als Ein- und Ausgabemedium verwenden, damit die einfache und allgemeine Anwendbarkeit gewährleistet bleibt. Aufgrund der hohen Komplexität des menschlichen Erscheinungsbilds und der Tiefenmehrdeutigkeit im 2D Raum sind datengetriebene Modelle ein naheliegendes Werkzeug, um solche Methoden zu entwerfen. In dieser Arbeit betrachten wir zwei Aspekte solcher Systeme: im ersten Teil entwickeln wir allgemein anwendbare Techniken für die Optimierung und Implementierung maschineller Lernmethoden und stellen diese in Form von Softwarepaketen bereit. Im zweiten Teil präsentieren wir in mehreren Schritten, wie die detaillierte Analyse und Darstellung von Menschen basierend auf digitalen 2D Bildern bewerkstelligt werden kann. Wir arbeiten dabei mit zwei Methoden zum maschinellen Lernen: Entscheidungswäldern und Künstlichen Neuronalen Netzen. Der Beitrag dieser Dissertation zur Theorie der Entscheidungswälder besteht in der Einführung einer verallgemeinerten Entropiefunktion, die effizient auswertbar und anpassbar ist und es ermöglicht, häufig verwendete Heuristiken besser einzuordnen. Für Entscheidungswälder und für Neuronale Netze beschreiben wir Methoden zur Implementierung und stellen jeweils ein Softwarepaket bereit, welches diese umsetzt. Die bisherigen Methoden zur 3D Körperschätzung aus Bildern beschränken sich auf die automatische Bestimmung der 14 wichtigsten 2D Punkte, welche die Pose definieren und deren Konvertierung in ein 3D "Skelett" Wir zeigen, dass durch die Optimierung einer fein abgestimmten Energiefunktion auch ein voller 3D Körper, nicht nur dessen Skelett, aus automatisch bestimmten 14 Punkten geschätzt werden kann. Damit beschreiben wir die erste vollautomatische Methode, die einen 3D Körper aus einem digitalen 2D Bild schätzt. Die detaillierte 3D Pose, beispielsweise mit Rotationen der Körperteile und die Beschaffenheit des untersuchten Körpers, ist damit noch nicht bestimmbar. Um detailliertere Modelle zu erstellen wäre es notwendig, Daten mit einem hohen Detailgrad zu annotieren. Dies skaliert jedoch nicht zu großen Datenmengen, da sowohl der Zeitaufwand pro Bild, als auch die notwendige Qualität aufgrund der schwierig einzuschätzenden exakten Positionen von Punkten auf der Körperoberfläche nicht erreicht werden können. Um dieses Problem zu lösen entwickeln wir eine Methode, die zwischen der Optimierung der 2D und 3D Modelle alterniert und diese wechselseitig verbessert. Dabei bleibt der Annotationsaufwand für Menschen gering. Gleichzeitig gelingt es, 2D Modelle mit einem Vielfachen an Details bisheriger Methoden zu erstellen und die Schätzung der 3D Pose und des Körpers auf Rotationen und Körperumfang zu erweitern. Um Bilder von Menschen zu generieren, beschränken sich existierende Methoden auf 3D Modelle, die schwer anzupassen und zu verwenden sind. Im Gegensatz dazu nutzen wir in dieser Arbeit einen Ansatz, der auf den Möglichkeiten zur automatischen 3D Posenschätzung basiert: wir nutzen sie, um einen Datensatz aus 3D Körpern mit dazugehörigen 2D Kleidungen und Kleidungssegmenten zu erstellen. Dies erlaubt es uns, ein datengetriebenes Modell zu entwickeln, welches direkt 2D Bilder von Menschen erzeugt. Erst das vielfältige Zusammenspiel von 2D und 3D Körper- und Erscheinungsmodellen in verschiedenen Formen ermöglicht es uns, einen hohen Detailgrad sowohl bei der Analyse als auch der Generierung menschlicher Erscheinung zu erzielen. Die hierfür entwickelten Techniken sind prinzipiell auch für die Analyse und Generierung von Bildern anderer Lebewesen und Objekte anwendbar

Complexity in Economic and Social Systems

There is no term that better describes the essential features of human society than complexity. On various levels, from the decision-making processes of individuals, through to the interactions between individuals leading to the spontaneous formation of groups and social hierarchies, up to the collective, herding processes that reshape whole societies, all these features share the property of irreducibility, i.e., they require a holistic, multi-level approach formed by researchers from different disciplines. This Special Issue aims to collect research studies that, by exploiting the latest advances in physics, economics, complex networks, and data science, make a step towards understanding these economic and social systems. The majority of submissions are devoted to financial market analysis and modeling, including the stock and cryptocurrency markets in the COVID-19 pandemic, systemic risk quantification and control, wealth condensation, the innovation-related performance of companies, and more. Looking more at societies, there are papers that deal with regional development, land speculation, and the-fake news-fighting strategies, the issues which are of central interest in contemporary society. On top of this, one of the contributions proposes a new, improved complexity measure

Privacy-aware publication and utilization of healthcare data

textOpen access to health data can bring enormous social and economical benefits. However, such access can also lead to privacy breaches, which may result in discrimination in insurance and employment markets. Privacy is a subjective and contextual concept, thus it should be interpreted from both systemic and information perspectives to clearly understand potential breaches and consequences. This dissertation investigates three popular use cases of healthcare data: specifically, 1) synthetic data publication, 2) aggregate data utilization, and 3) privacy-aware API implementation. For each case, we develop statistical models that improve the privacy-utility Pareto frontier by leveraging a variety of machine learning techniques such as information theoretic privacy measures, Bayesian graphical models, non-parametric modeling, and low-rank factorization techniques. It shows that much utility can be extracted from health records while maintaining strong privacy guarantees and protection of sensitive health information.Electrical and Computer Engineerin

Information theoretic refinement criteria for image synthesis

Aquest treball està enmarcat en el context de gràfics per computador partint de la intersecció de tres camps: rendering, teoria de la informació, i complexitat.Inicialment, el concepte de complexitat d'una escena es analitzat considerant tres perspectives des d'un punt de vista de la visibilitat geomètrica: complexitat en un punt interior, complexitat d'una animació, i complexitat d'una regió. L'enfoc principal d'aquesta tesi és l'exploració i desenvolupament de nous criteris de refinament pel problema de la il·luminació global. Mesures de la teoria de la informació basades en la entropia de Shannon i en la entropia generalitzada de Harvda-Charvát-Tsallis, conjuntament amb les f-divergències, són analitzades com a nuclis del refinement. Mostrem com ens aporten una rica varietat d'eficients i altament discriminatòries mesures que són aplicables al rendering en els seus enfocs de pixel-driven (ray-tracing) i object-space (radiositat jeràrquica).Primerament, basat en la entropia de Shannon, es defineixen un conjunt de mesures de qualitat i contrast del pixel. S'apliquen al supersampling en ray-tracing com a criteris de refinement, obtenint un algorisme nou de sampleig adaptatiu basat en entropia, amb un alt rati de qualitat versus cost. En segon lloc, basat en la entropia generalitzada de Harvda-Charvát-Tsallis, i en la informació mutua generalitzada, es defineixen tres nous criteris de refinament per la radiositat jeràrquica. En correspondencia amb tres enfocs clàssics, es presenten els oracles basats en la informació transportada, el suavitzat de la informació, i la informació mutua, amb resultats molt significatius per aquest darrer. Finalment, tres membres de la familia de les f-divergències de Csiszár's (divergències de Kullback-Leibler, chi-square, and Hellinger) son analitzats com a criteris de refinament mostrant bons resultats tant pel ray-tracing com per la radiositat jeràrquica.This work is framed within the context of computer graphics starting out from the intersection of three fields: rendering, information theory, and complexity.Initially, the concept of scene complexity is analysed considering three perspectives from a geometric visibility point of view: complexity at an interior point, complexity of an animation, and complexity of a region. The main focus of this dissertation is the exploration and development of new refinement criteria for the global illumination problem. Information-theoretic measures based on Shannon entropy and Harvda-Charvát-Tsallis generalised entropy, together with f-divergences, are analysed as kernels of refinement. We show how they give us a rich variety of efficient and highly discriminative measures which are applicable to rendering in its pixel-driven (ray-tracing) and object-space (hierarchical radiosity) approaches.Firstly, based on Shannon entropy, a set of pixel quality and pixel contrast measures are defined. They are applied to supersampling in ray-tracing as refinement criteria, obtaining a new entropy-based adaptive sampling algorithm with a high rate quality versus cost. Secondly, based on Harvda-Charvát-Tsallis generalised entropy, and generalised mutual information, three new refinement criteria are defined for hierarchical radiosity. In correspondence with three classic approaches, oracles based on transported information, information smoothness, and mutual information are presented, with very significant results for the latter. And finally, three members of the family of Csiszár's f-divergences (Kullback-Leibler, chi-square, and Hellinger divergences) are analysed as refinement criteria showing good results for both ray-tracing and hierarchical radiosity