Entropy and inference, revisited

We study properties of popular near-uniform (Dirichlet) priors for learning undersampled probability distributions on discrete nonmetric spaces and show that they lead to disastrous results. However, an Occam-style phase space argument expands the priors into their infinite mixture and resolves most of the observed problems. This leads to a surprisingly good estimator of entropies of discrete distributions.Comment: LaTex2e, 9 pages, 5 figures; references added, minor revisions introduced, formatting errors correcte

Unsupervised metrics for unsupervised image denoising

En els darrers anys, el rapid desenvolupament dels recursos computacionals i les tècniques d'aprenentatge profund han donat lloc a un conjunt de nous mètodes per al "denoising" d'imatges. Aquests mètodes es basen principalment en l'entrenament supervisat de xarxes neuronals convolucionals, utilitzant imatges netes per generar parelles (sorollosa, neta) com a dades d'entrada i sortida durant l'entrenament. Aquest enfocament restringeix l'aplicació d'aquests mètodes als camps d'imatge on existeixen imatges netes. Això exclou la majoria de les dades d'imatge científica, on la recollida d'imatges netes és impossible emprant les tècniques actuals. Recentment, diferents autors han introduït un conjunt de tècniques que no requereixen dades netes en el que s'ha anomenat "Denoising" No Supervisat. No obstant això, no s'han realitzat comparacions importants entre aquests mètodes, de manera que no hi ha un mètode de referència inequívoc. A més, l'avaluació d'aquests mètodes sempre s'ha dut a terme de manera supervisada, comparant els resultats del "denoising" amb imatges netes de referència, a causa de la manca de mètriques fiables no supervisades. Tenint en compte aquests fets, en aquest treball dissenyem i testem un conjunt de mètriques no supervisades per a la comparació de models en absència de dades netes i les utilitzem per dur a terme una comparació global entre els diferents mètodes fins a la data utilitzant un conjunt d'imatges comú del camp de la microscòpia electrònica de transmissió.En los últimos años, la extraordinaria mejora de los recursos computacionales y las técnicas de Deep Learning han dado lugar a un conjunto de nuevos métodos para eliminar el ruido de las imágenes. Estos métodos se basan principalmente en el entrenamiento supervisado de redes neuronales convolucionales, utilizando imágenes limpias para generar pares (ruidosa, limpia) como datos de (entrada, salida) durante el entrenamiento. Este enfoque restringe la aplicación de estos métodos a los campos de imágenes donde hay imágenes limpias disponibles. Esto excluye la mayor parte de los datos de imágenes científicas, en los que la recopilación de imágenes limpias es imposible utilizando las técnicas actuales. Recientemente, diferentes autores han introducido un conjunto de nuevas técnicas que no requieren datos limpios en lo que se ha denominado Unsupervised Denoising. Sin embargo no se han realizado comparaciones importantes entre estos métodos, por lo que no existe un criterio claro sobre el método de referencia. Además, la evaluación de estos métodos siempre se ha realizado de forma supervisada, comparando los resultados sin ruido con imágenes de referencia limpias, debido a la falta de métricas fiables no supervisadas. Atendiendo a estos hechos, en el presente trabajo diseñamos y probamos un conjunto de métricas no supervisadas para la comparación de modelos en ausencia de datos limpios y los usamos para llevar a cabo una comparación global entre los diferentes métodos hasta la fecha utilizando un conjunto de imágenes común de Microscopio Electrónico de Transmisión.In the recent years the extraordinary improvement of the computational resources and the Deep Learning techniques have led to a set of new methods for image denoising. These methods are mostly based on the supervised training of Convolutional Neural Networks, using clean images to generate (noisy, clean) pairs as (input, output) data during the training. This approach restricts the application of these methods to imaging fields where clean images are available. This excludes most of the scientific imaging data, where the collection of clean images is impossible using the present techniques. Recently, different authors have introduced a set of new techniques that do not require clean data in what has been named Unsupervised Denoising. Nevertheless, no major comparisons among these methods have been performed, so there is no clear state of the art approach. Furthermore, the evaluation of these methods has always been performed in a supervised way, comparing the denoised results with true clean ground truth images, due to the lack of unsupervised reliable metrics. Attending to these facts, in the present work we design and test a set of unsupervised metrics for model comparison in the absence of clean data and use them to drive a global comparison among the different methods to the date using a common dataset of Transmission Electron Microscope images.Outgoin

Beyond the Quadratic Time Barrier for Network Unreliability

Karger (STOC 1995) gave the first FPTAS for the network (un)reliability problem, setting in motion research over the next three decades that obtained increasingly faster running times, eventually leading to a $\tilde{O}(n^2)$-time algorithm (Karger, STOC 2020). This represented a natural culmination of this line of work because the algorithmic techniques used can enumerate $\Theta(n^2)$ (near)-minimum cuts. In this paper, we go beyond this quadratic barrier and obtain a faster algorithm for the network unreliability problem. Our algorithm runs in $m^{1+o(1)} + \tilde{O}(n^{1.5})$ time. Our main contribution is a new estimator for network unreliability in very reliable graphs. These graphs are usually the bottleneck for network unreliability since the disconnection event is elusive. Our estimator is obtained by defining an appropriate importance sampling subroutine on a dual spanning tree packing of the graph. To complement this estimator for very reliable graphs, we use recursive contraction for moderately reliable graphs. We show that an interleaving of sparsification and contraction can be used to obtain a better parametrization of the recursive contraction algorithm that yields a faster running time matching the one obtained for the very reliable case

A Weighted Model Counting Approach for Critical Infrastructure Reliability

Reliability assessment of engineered systems such as telecommunication networks, power grids, and railroads is an important step towars supporting resilient communities. However, calculating the reliability of a network is computationally intensive. Thus, simulation methods are often preferred over exact methods in practice. Unfortunately, highly reliable and large scale systems can challenge common assumptions in simulation techniques, rendering reliability estimates—as well as reported error and confidence—unreliable themselves. A new generation of techniques, termed probably approximately correct (PAC) methods, delivers provable network reliability calculations with user-specified error and confidence. In this paper we focus on RelNet, a model counting-based method for network reliability estimation endowed with rigorous PAC guarantees. Despite previous success in power transmission network applications, small edge failure probabilities and dependent failures can challenge the current methodology. We put forward Weighted RelNet, a general importance sampling-based extension that treats the systems joint probability distribution as a black box. Empirical evaluations suggest the new approach is competitive across challenging rare-event benchmarks.The authors gratefully acknowledge the support by the U.S. Department of Defense (Grant W911NF-13-1-0340) and the U.S. National Science Foundation (Grants CMMI-1436845 and CMMI-1541033)

Investigation of Non-coherent Discrete Target Range Estimation Techniques for High-precision Location

Ranging is an essential and crucial task for radar systems. How to solve the range-detection problem effectively and precisely is massively important. Meanwhile, unambiguity and high resolution are the points of interest as well. Coherent and non-coherent techniques can be applied to achieve range estimation, and both of them have advantages and disadvantages. Coherent estimates offer higher precision but are more vulnerable to noise and clutter and phase wrap errors, particularly in a complex or harsh environment, while the non-coherent approaches are simpler but provide lower precision. With the purpose of mitigating inaccuracy and perturbation in range estimation, miscellaneous techniques are employed to achieve optimally precise detection. Numerous elegant processing solutions stemming from non-coherent estimate are now introduced into the coherent realm, and vice versa. This thesis describes two non-coherent ranging estimate techniques with novel algorithms to mitigate the instinct deficit of non-coherent ranging approaches. One technique is based on peak detection and realised by Kth-order Polynomial Interpolation, while another is based on Z-transform and realised by Most-likelihood Chirp Z-transform. A two-stage approach for the fine ranging estimate is applied to the Discrete Fourier transform domain of both algorithms. An N-point Discrete Fourier transform is implemented to attain a coarse estimation; an accurate process around the point of interest determined in the first stage is conducted. For KPI technique, it interpolates around the peak of Discrete Fourier transform profiles of the chirp signal to achieve accurate interpolation and optimum precision. For Most-likelihood Chirp Z-transform technique, the Chirp Z-transform accurately implements the periodogram where only a narrow band spectrum is processed. Furthermore, the concept of most-likelihood estimator is introduced to combine with Chirp Z-transform to acquire better ranging performance. Cramer-Rao lower bound is presented to evaluate the performance of these two techniques from the perspective of statistical signal processing. Mathematical derivation, simulation modelling, theoretical analysis and experimental validation are conducted to assess technique performance. Further research will be pushed forward to algorithm optimisation and system development of a location system using non-coherent techniques and make a comparison to a coherent approach

Learning latent variable models : efficient algorithms and applications

Learning latent variable models is a fundamental machine learning problem, and the models belonging to this class - which include topic models, hidden Markov models, mixture models and many others - have a variety of real-world applications, like text mining, clustering and time series analysis. For many practitioners, the decade-old Expectation Maximization method (EM) is still the tool of choice, despite its known proneness to local minima and long running times. To overcome these issues, algorithms based on the spectral method of moments have been recently proposed. These techniques recover the parameters of a latent variable model by solving - typically via tensor decomposition - a system of non-linear equations relating the low-order moments of the observable data with the parameters of the model to be learned. Moment-based algorithms are in general faster than EM as they require a single pass over the data, and have provable guarantees of learning accuracy in polynomial time. Nevertheless, methods of moments have room for improvements: their ability to deal with real-world data is often limited by a lack of robustness to input perturbations. Also, almost no theory studies their behavior when some of the model assumptions are violated by the input data. Extending the theory of methods of moments to learn latent variable models and providing meaningful applications to real-world contexts is the focus of this thesis. ssuming data to be generated by a certain latent variable model, the standard approach of methods of moments consists of two steps: first, finding the equations that relate the moments of the observable data with the model parameters and then, to solve these equations to retrieve estimators of the parameters of the model. In Part I of this thesis we will focus on both steps, providing and analyzing novel and improved model-specific moments estimators and techniques to solve the equations of the moments. In both the cases we will introduce theoretical results, providing guarantees on the behavior of the proposed methods, and we will perform experimental comparisons with existing algorithms. In Part II, we will analyze the behavior of methods of moments when data violates some of the model assumptions performed by a user. First, we will observe that in this context most of the theoretical infrastructure underlying methods of moments is not valid anymore, and consequently we will develop a theoretical foundation to methods of moments in the misspecified setting, developing efficient methods, guaranteed to provide meaningful results even when some of the model assumptions are violated. During all the thesis, we will apply the developed theoretical results to challenging real-world applications, focusing on two main domains: topic modeling and healthcare analytics. We will extend the existing theory of methods of moments to learn models that are traditionally used to do topic modeling – like the single-topic model and Latent Dirichlet Allocation – providing improved learning techniques and comparing them with existing methods, which we prove to outperform in terms of speed and learning accuracy. Furthermore, we will propose applications of latent variable models to the analysis of electronic healthcare records, which, similarly to text mining, are very likely to become massive datasets; we will propose a method to discover recurrent phenotypes in populations of patients and to cluster them in groups with similar clinical profiles - a task where the efficiency properties of methods of moments will constitute a competitive advantage over traditional approaches.Aprender modelos de variable latente es un problema fundamental de machine learning, y los modelos que pertenecen a esta clase, como topic models, hidden Markov models o mixture models, tienen muchas aplicaciones en el mundo real, por ejemplo text mining, clustering y time series analysis. El método de Expectation Maximization (EM) sigue siendo la herramienta más utilizada, a pesar de su conocida tendencia a producir soluciones subóptimas y sus largos tiempos de ejecución. Para superar estos problemas, se han propuesto recientemente algoritmos basados en el método de los momentos. Estas técnicas aprenden los parámetros de un modelo resolviendo un sistema de ecuaciones no lineales que relacionan los momentos de los datos observables con los parámetros del modelo que se debe aprender. Los métodos de los momentos son en general más rápidos que EM, ya que requieren una sola pasada sobre los datos y tienen garantías de producir estimadores consistentes en tiempo polinomial. A pesar de estas ventajas, los métodos de los momentos todavía tienen margen de mejora: cuando se utilizan con datos reales, los métodos de los momentos se revelan inestables, con una fuerte sensibilidad a las perturbaciones. Además, las garantías de estos métodos son válidas solo si el usuario conoce el modelo probabilístico que genera los datos, y no existe alguna teoría que estudie lo que pasa cuando ese modelo es desconocido o no correctamente especificado. El objetivo de esta tesis es ampliar la teoría de métodos de los momentos, estudiar sus aplicaciones para aprender modelos de variable latente, extendiendo la teoría actual. Además se proporcionarán aplicaciones significativas a contextos reales. Típicamente, el método de los momentos consta de de dos partes: primero, encontrar las ecuaciones que relacionan los momentos de los datos observables con los parámetros del modelo y segundo, resolver estas ecuaciones para recuperar estimadores consistentes de los parámetros del modelo. En la Parte I de esta tesis, nos centraremos en ambos pasos, proporcionando y analizando nuevos estimadores de momentos para una variedad de modelos, y técnicas para resolver las ecuaciones de los momentos. En ambos casos, introduciremos resultados teóricos, proporcionaremos garantías sobre el comportamiento de los métodos propuestos y realizaremos comparaciones experimentales con algoritmos existentes. En la Parte II, analizaremos el comportamiento de los métodos de los momentos cuando algunas de las hipótesis de modelo se encuentran violadas por los datos. Como primera cosa, observaremos que en este contexto, la mayoría de la infraestructura teórica que subyace a estos métodos pierde de validez y, por lo tanto, desarrollaremos una base teórica nueva, presentando métodos eficientes, garantizados para proporcionar resultados razonables incluso cuando algunas de las hipótesis del modelo son violadas. En toda la tesis aplicamos los resultados obtenidos a nivel teórico a aplicaciones del mundo real, centrándonos en dos áreas principales: topic modeling y healthcare analytics. Ampliamos la teoría existente de los métodos de momentos para aprender los modelos que se usan tradicionalmente en el ámbito de topic modeling, como el single-topic model y la Latent Dirichlet Allocation, proporcionando nuevas técnicas de aprendizaje y comparándolas con los métodos existentes. Además, estudiamos aplicaciones de modelos de variable latente en el análisis de datos del ámbito healthcare; proponemos un método para descubrir fenotipos recurrentes en poblaciones de pacientes y agruparlos en clusters con perfiles clínicos similares, una tarea donde las propiedades de eficiencia de los métodos de los momentos constituyen una ventaja competitiva sobre los métodos tradicionales.Postprint (published version

Spatial Dependence and Heterogeneity in Empirical Analyses of Regional Labour Market Dynamics

Are regions within a country really independent islands? Do economic relations and effects really have a homogenous, unique size across an entire country? These two assumptions are often imposed implicitly in empirical economic and social research. In his doctoral thesis, the author discusses how statistical methods can deviate from this unrealistic model structure through employing spatial patterns in both observable variables and presumed relations. Opportunities to improve our understanding of the economy as well as chances and perils in the application of such methods are demonstrated in a number of studies on aspects of regional labour market dynamics.Warum sollen Regionen innerhalb eines Landes unabhängige Inseln sein? Und warum sollen, über das gesamte Land hinweg, einheitlich starke ökonomische oder soziale Wirkungszusammenhänge bestehen? Diese zwei Annahmen werden in der angewandten empirischen Wirtschafts- und Sozialforschung üblicherweise implizit unterstellt. Wie in statistischen Verfahren von dieser unrealistischen Modellstruktur unter Ausnutzung der räumlichen Strukturen in beobachteten Variablen und unterstellten Zusammenhängen abgewichen werden kann, diskutiert der Autor im vorliegenden Band. Möglichkeiten, unser Verständnis der Ökonomie zu vertiefen, werden ebenso verdeutlicht, wie Chancen und Tücken beim Einsatz der Methoden in Studien zu verschiedenen Aspekten der Arbeitsmarktdynamik

Questions and Concerns About Google's Quantum Supremacy Claim

In October 2019, Nature published a paper [6] describing an experimental work that was performed at Google. The paper claims to demonstrate quantum (computational) supremacy on a 53-qubit quantum computer. Since then we have been involved in a long-term project to study various statistical aspects of the Google experiment. In [30] we studied Google's statistical framework that we found to be very sound and offered some technical improvements. This document describes three main concerns (based on statistical analysis) about the Google 2019 experiment. The first concern is that the data do not agree with Google's noise model (or any other specific model). The second concern is that a crucial simple formula for a priori estimation of the fidelity seems to involve an unexpected independence assumption, and yet it gives very accurate predictions. The third concern is about statistical properties of the calibration process.Comment: 49 pages, 13 Figures, 7 Table