411 research outputs found
An Introduction to Variational Autoencoders
Variational autoencoders provide a principled framework for learning deep
latent-variable models and corresponding inference models. In this work, we
provide an introduction to variational autoencoders and some important
extensions
Scalable Hierarchical Gaussian Process Models for Regression and Pattern Classification
Gaussian processes, which are distributions over functions, are powerful nonparametric tools for the two major machine learning tasks: regression and classification. Both tasks are concerned with learning input-output mappings from example input-output pairs. In Gaussian process (GP) regression and classification, such mappings are modeled by Gaussian processes. In GP regression, the likelihood is Gaussian for continuous outputs, and hence closed-form solutions for prediction and model selection can be obtained. In GP classification, the likelihood is non-Gaussian for discrete/categorical outputs, and hence closed-form solutions are not available, and approximate inference methods must be resorted
Distributed Statistical Learning under Communication Constraints
In this thesis, we study distributed statistical learning, in which multiple terminals, connected by links with limited capacity, cooperate to perform a learning task. As the links connecting the terminals have limited capacity, the messages exchanged between the terminals have to be compressed. The goal of this thesis is to investigate how to compress the data observations at multiple terminals and how to use the compressed data for inference. We first focus on the distributed parameter estimation problem, in which terminals send messages related to their local observations using limited rates to a fusion center that will obtain an estimate of a parameter related to the observations of all terminals. It is well known that if the transmission rates are in the Slepian-Wolf region, the fusion center can fully recover all observations and hence can construct an estimator having the same performance as that of the centralized case. One natural question is whether Slepian-Wolf rates are necessary to achieve the same estimation performance as that of the centralized case. In this thesis, we show that the answer to this question is negative. We then examine the optimality of data dimensionality reduction via sufficient statistics compression in distributed parameter estimation problems. The data dimensionality reduction step is often needed especially if the data has a very high dimension and the communication rate is not as high as the one characterized above. We show that reducing the dimensionality by extracting sufficient statistics of the parameter to be estimated does not degrade the overall estimation performance in the presence of communication constraints. We further analyze the optimal estimation performance in the presence of communication constraints and we verify the derived bound using simulations. Finally, we study distributed optimization problems, for which we examine the randomized distributed coordinate descent algorithm with quantized updates. In the literature, the iteration complexity of the randomized distributed coordinate descent algorithm has been characterized under the assumption that machines can exchange updates with an infinite precision. We consider a practical scenario in which the messages exchange occurs over channels with finite capacity, and hence the updates have to be quantized. We derive sufficient conditions on the quantization error such that the algorithm with quantized update still converge
Approximate inference in massive MIMO scenarios with moment matching techniques
Mención Internacional en el título de doctorThis Thesis explores low-complexity inference probabilistic algorithms in
high-dimensional Multiple-Input Multiple-Output (MIMO) systems and high order
M-Quadrature Amplitude Modulation (QAM) constellations. Several
modern communications systems are using more and more antennas to maximize
spectral efficiency, in a new phenomena call Massive MIMO. However,
as the number of antennas and/or the order of the constellation grow several
technical issues have to be tackled, one of them is that the symbol
detection complexity grows fast exponentially with the system dimension.
Nowadays the design of massive MIMO low-complexity receivers is one important
research line in MIMO because symbol detection can no longer rely
on conventional approaches such as Maximum a Posteriori (MAP) due to
its exponential computation complexity. This Thesis proposes two main results.
On one hand a hard decision low-complexity MIMO detector based on
Expectation Propagation (EP) algorithm which allows to iteratively approximate
within polynomial cost the posterior distribution of the transmitted
symbols. The receiver is named Expectation Propagation Detector (EPD)
and its solution evolves from Minimum Mean Square Error (MMSE) solution
and keeps per iteration the MMSE complexity which is dominated by
a matrix inversion. Hard decision Symbol Error Rate (SER) performance is
shown to remarkably improve state-of-the-art solutions of similar complexity.
On the other hand, a soft-inference algorithm, more suitable to modern
communication systems with channel codification techniques such as Low-
Density Parity-Check (LDPC) codes, is also presented. Modern channel
decoding techniques need as input Log-Likehood Ratio (LLR) information
for each coded bit. In order to obtain that information, firstly a soft bit
inference procedure must be performed. In low-dimensional scenarios, this
can be done by marginalization over the symbol posterior distribution. However,
this is not feasible at high-dimension. While EPD could provide this
probabilistic information, it is shown that its probabilistic estimates are in
general poor in the low Signal-to-Noise Ratio (SNR) regime. In order to
solve this inconvenience a new algorithm based on the Expectation Consistency
(EC) algorithm, which generalizes several algorithms such as Belief.
Propagation (BP) and EP itself, was proposed. The proposed algorithm
called Expectation Consistency Detector (ECD) maps the inference problem
as an optimization over a non convex function. This new approach
allows to find stationary points and tradeoffs between accuracy and convergence,
which leads to robust update rules. At the same complexity cost than
EPD, the new proposal achieves a performance closer to channel capacity at
moderate SNR. The result reveals that the probabilistic detection accuracy
has a relevant impact in the achievable rate of the overall system. Finally,
a modified ECD algorithm is presented, with a Turbo receiver structure
where the output of the decoder is fed back to ECD, achieving performance
gains in all block lengths simulated.
The document is structured as follows. In Chapter I an introduction
to the MIMO scenario is presented, the advantages and challenges are exposed
and the two main scenarios of this Thesis are set forth. Finally, the
motivation behind this work, and the contributions are revealed. In Chapters
II and III the state of the art and our proposal are presented for Hard
Detection, whereas in Chapters IV and V are exposed for Soft Inference Detection.
Eventually, a conclusion and future lines can be found in Chapter
VI.Esta Tesis aborda algoritmos de baja complejidad para la estimación probabilística en sistemas de Multiple-Input Multiple-Output (MIMO) de grandes
dimensiones con constelaciones M-Quadrature Amplitude Modulation (QAM)
de alta dimensionalidad. Son diversos los sistemas de comunicaciones que en
la actualidad están utilizando más y más antenas para maximizar la eficiencia
espectral, en un nuevo fenómeno denominado Massive MIMO. Sin embargo
los incrementos en el número de antenas y/o orden de la constelación
presentan ciertos desafíos tecnológicos que deben ser considerados. Uno de
ellos es la detección de los símbolos transmitidos en el sistema debido a que
la complejidad aumenta más rápido que las dimensiones del sistema. Por
tanto el diseño receptores para sistemas Massive MIMO de baja complejidad
es una de las importantes líneas de investigación en la actualidad en
MIMO, debido principalmente a que los métodos tradicionales no se pueden
implementar en sistemas con decenas de antenas, cuando lo deseable serían
centenas, debido a que su coste es exponencial.
Los principales resultados en esta Tesis pueden clasificarse en dos. En
primer lugar un receptor MIMO para decisión dura de baja complejidad
basado en el algoritmo Expectation Propagation (EP) que permite de manera
iterativa, con un coste computacional polinómico por iteración, aproximar
la distribución a posteriori de los símbolos transmitidos. El algoritmo,
denominado Expectation Propagation Detector (EPD), es inicializado con
la solución del algoritmo Minimum Mean Square Error (MMSE) y mantiene
el coste de este para todas las iteraciones, dominado por una inversión de
matriz. El rendimiento del decisor en probabilidad de error de símbolo muestra
ganancias remarcables con respecto a otros métodos en la literatura con
una complejidad similar. En segundo lugar, un algoritmo que provee una
estimación blanda, información que es más apropiada para los actuales sistemas
de comunicaciones que utilizan codificación de canal, como pueden
ser códigos Low-Density Parity-Check (LDPC). La información necesaria
para estos decodificadores de canal es Log-Likehood Ratio (LLR) para cada
uno de los bits codificados.
En escenarios de bajas dimensiones se pueden calcular las marginales de la distribución a posteriori, pero en escenarios de grandes dimensiones
no es viable, aunque EPD puede proporcionar este tipo de información a la
entrada del decodificador, dicha información no es la mejor al estar el algoritmo
pensado para detección dura, sobre todo se observa este fenómeno en
el rango de baja Signal-to-Noise Ratio (SNR). Para solucionar este problema
se propone un nuevo algoritmo basado en Expectation Consistency
(EC) que engloba diversos algoritmos como pueden ser Belief Propagation
(BP) y el algoritmo EP propuesto con anterioridad. El nuevo algoritmo
llamado Expectation Consistency Detector (ECD), trata el problema como
una optimización de una función no convexa. Esta aproximación permite
encontrar los puntos estacionarios y la relación entre precisión y convergencia,
que permitirán reglas de actualización más robustas y eficaces. Con
la misma compleja que el algoritmo propuesto inicialmente, ECD permite
rendimientos más próximos a la capacidad del canal en regímenes moderados
de SNR. Los resultados muestran que la precisión tiene un gran efecto
en la tasa que alcanza el sistema. Finalmente una versión modificada de
ECD es propuesta en una arquitectura típica de los Turbo receptores, en
la que la salida del decodificador es la entrada del receptor, y que permite
ganancias en el rendimiento en todas las longitudes de código simuladas.
El presente documento está estructurado de la siguiente manera. En el
primer Capítulo I, se realiza una introducción a los sistemas MIMO, presentando
sus ventajas, desventajas, problemas abiertos. Los modelos que se
utilizaran en la tesis y la motivación con la que se inició esta tesis son expuestos
en este primer capítulo. En los Capítulos II y III el estado del arte y
nuestra propuesta para detección dura son presentados, mientras que en los
Capítulos IV y V se presentan para detección suave. Finalmente las conclusiones
que pueden obtenerse de esta Tesis y futuras líneas de investigación
son expuestas en el Capítulo VI.Programa Oficial de Doctorado en Multimedia y ComunicacionesPresidente: Juan José Murillo Fuentes.- Secretario: Gonzalo Vázquez Vilar.- Vocal: María Isabel Valera Martíne
Multi-modal dictionary learning for image separation with application in art investigation
In support of art investigation, we propose a new source separation method
that unmixes a single X-ray scan acquired from double-sided paintings. In this
problem, the X-ray signals to be separated have similar morphological
characteristics, which brings previous source separation methods to their
limits. Our solution is to use photographs taken from the front and back-side
of the panel to drive the separation process. The crux of our approach relies
on the coupling of the two imaging modalities (photographs and X-rays) using a
novel coupled dictionary learning framework able to capture both common and
disparate features across the modalities using parsimonious representations;
the common component models features shared by the multi-modal images, whereas
the innovation component captures modality-specific information. As such, our
model enables the formulation of appropriately regularized convex optimization
procedures that lead to the accurate separation of the X-rays. Our dictionary
learning framework can be tailored both to a single- and a multi-scale
framework, with the latter leading to a significant performance improvement.
Moreover, to improve further on the visual quality of the separated images, we
propose to train coupled dictionaries that ignore certain parts of the painting
corresponding to craquelure. Experimentation on synthetic and real data - taken
from digital acquisition of the Ghent Altarpiece (1432) - confirms the
superiority of our method against the state-of-the-art morphological component
analysis technique that uses either fixed or trained dictionaries to perform
image separation.Comment: submitted to IEEE Transactions on Images Processin
Disentangled Variational Autoencoder based Multi-Label Classification with Covariance-Aware Multivariate Probit Model
Multi-label classification is the challenging task of predicting the presence
and absence of multiple targets, involving representation learning and label
correlation modeling. We propose a novel framework for multi-label
classification, Multivariate Probit Variational AutoEncoder (MPVAE), that
effectively learns latent embedding spaces as well as label correlations. MPVAE
learns and aligns two probabilistic embedding spaces for labels and features
respectively. The decoder of MPVAE takes in the samples from the embedding
spaces and models the joint distribution of output targets under a Multivariate
Probit model by learning a shared covariance matrix. We show that MPVAE
outperforms the existing state-of-the-art methods on a variety of application
domains, using public real-world datasets. MPVAE is further shown to remain
robust under noisy settings. Lastly, we demonstrate the interpretability of the
learned covariance by a case study on a bird observation dataset
Iterative Receiver Techniques for Data-Driven Channel Estimation and Interference Mitigation in Wireless Communications
Wireless mobile communications were initially a way for people to communicate through low data rate voice call connections. As data enabled devices allow users the ability to do much more with their mobile devices, so to will the demand for more reliable and pervasive wireless data. This is being addressed by so-called 4th generation wireless systems based on orthogonal frequency division multiplexing (OFDM) and multiple-input multiple-output (MIMO) antenna systems. Mobile wireless customers are becoming more demanding and expecting to have a great user experience over high speed broadband access at any time and anywhere, both indoor and outdoor. However, these promising improvements cannot be realized without an e±cient design of the receiver. Recently, receivers utilizing iterative detection and decoding have changed the fundamental receiver design paradigm from traditional separated parameter estimation and data detection blocks to an integrated iterative parameter estimator and data detection unit. Motivated by this iterative data driven approach, we develop low complexity iterative receivers with improved sensitivity compared to the conventional receivers, this brings potential benefits for the wireless communication system, such as improving the overall system throughput, increasing the macro cell coverage, and reducing the cost of the equipments in both the base station and mobile terminal. It is a challenge to design receivers that have good performance in a highly dynamic mobile wireless environment. One of the challenges is to minimize overhead reference signal energy (preamble, pilot symbols) without compromising the performance. We investigate this problem, and develop an iterative receiver with enhanced data-driven channel estimation. We discuss practical realizations of the iterative receiver for SISO-OFDM system. We utilize the channel estimation from soft decoded data (the a priori information) through frequency-domain combining and time-domain combining strategies in parallel with limited pilot signals. We analyze the performance and complexity of the iterative receiver, and show that the receiver's sensitivity can be improved even with this low complexity solution. Hence, seamless communications can be achieved with better macro cell coverage and mobility without compromising the overall system performance. Another challenge is that a massive amount of interference caused by MIMO transmission (spatial multiplexing MIMO) reduces the performance of the channel estimation, and further degrades data detection performance. We extend the iterative channel estimation from SISO systems to MIMO systems, and work with linear detection methods to perform joint interference mitigation and channel estimation. We further show the robustness of the iterative receivers in both indoor and outdoor environment compared to the conventional receiver approach. Finally, we develop low complexity iterative spatial multiplexed MIMO receivers for nonlinear methods based on two known techniques, that is, the Sphere Decoder (SD) method and the Markov Chain Monte Carlo (MCMC) method. These methods have superior performance, however, they typically demand a substantial increase in computational complexity, which is not favorable in practical realizations. We investigate and show for the first time how to utilize the a priori information in these methods to achieve performance enhancement while simultaneously substantially reducing the computational complexity. In our modified sphere decoder method, we introduce a new accumulated a priori metric in the tree node enumeration process. We show how we can improve the performance by obtaining the reliable tree node candidate from the joint Maximum Likelihood (ML) metric and an approximated a priori metric. We also show how we can improve the convergence speed of the sphere decoder (i.e., reduce the com- plexity) by selecting the node with the highest a priori probability as the starting node in the enumeration process. In our modified MCMC method, the a priori information is utilized for the firrst time to qualify the reliably decoded bits from the entire signal space. Two new robust MCMC methods are developed to deal with the unreliable bits by using the reliably decoded bit information to cancel the interference that they generate. We show through complexity analysis and performance comparison that these new techniques have improved performance compared to the conventional approaches, and further complexity reduction can be obtained with the assistance of the a priori information. Therefore, the complexity and performance tradeoff of these nonlinear methods can be optimized for practical realizations
- …