Tree-AMP: Compositional Inference with Tree Approximate Message Passing

We introduce Tree-AMP, standing for Tree Approximate Message Passing, a python package for compositional inference in high-dimensional tree-structured models. The package provides a unifying framework to study several approximate message passing algorithms previously derived for a variety of machine learning tasks such as generalized linear models, inference in multi-layer networks, matrix factorization, and reconstruction using non-separable penalties. For some models, the asymptotic performance of the algorithm can be theoretically predicted by the state evolution, and the measurements entropy estimated by the free entropy formalism. The implementation is modular by design: each module, which implements a factor, can be composed at will with other modules to solve complex inference tasks. The user only needs to declare the factor graph of the model: the inference algorithm, state evolution and entropy estimation are fully automated.Comment: Source code available at https://github.com/sphinxteam/tramp and documentation at https://sphinxteam.github.io/tramp.doc

Expectation propagation as a solution for digital communication systems.

In the context of digital communications, a digital receiver is required to provide an estimation of the transmitted symbols. Nowadays channel decoders highly benefit from soft (probabilistic) estimates for each transmitted symbol rather than from hard decisions. For this reason, digital receivers must be designed to provide the probability that each possible symbol was transmitted based on the received corrupted signal. Since exact inference might be unfeasible in terms of complexity for high-order scenarios, it is necessary to resort to approximate inference, such as the linear minimum mean square error (LMMSE) criterion. The LMMSE approximates the discrete prior information of the transmitted symbols with a Gaussian distribution, which causes a degradation in its performance. In this thesis, an alternative approximate statistical technique is applied to the design of a digital probabilistic receiver in digital communications. Specifically, the expectation propagation (EP) algorithm is investigated to find the Gaussian posterior probability density function (pdf) that minimizes the Kullback-Leibler (KL) divergence with respect to the true posterior pdf. Two different communication system scenarios are studied: a single-input singleoutput (SISO) digital communication system with memory channel and a multipleinput multiple-output (MIMO) system with memoryless channel. In the SISO scenario, three different designs of a soft standalone and turbo equalizer based on the EP algorithm are developed: the block or batch approach, the filter-type version that emulates theWiener filter behavior and the smoothing equalizer which proceeds similarly to a Kalman smoother. Finally, the block EP implementation is also adapted to MIMO scenarios with feedback from the decoder. In both scenarios, the EP is applied iteratively, including a damping mechanism and a control to avoid negative values of variances, which would lead to instabilities (specially for high-order constellations). Experimental results included through the thesis show that the EP algorithm applied to communication systems greatly improves the performance of previous approaches found in the literature with a complexity slightly increased but still proportional to that of the LMMSE. These results also show the robustness of the algorithm even for high-order modulations, large memory channels and high number of antennas. Major contributions of this dissertation have been published in four journal (one of them is still under review) and two conference papers. One more paper will be submitted to a journal soon. All these papers are listed below: • Irene Santos, Juan José Murillo-Fuentes, Rafael Boloix-Tortosa, Eva Arias de Reyna and Pablo M. Olmos, "Expectation Propagation as Turbo Equalizer in ISI Channels," IEEE Transactions on Communications, vol. 65, no.1, pp. 360-370, Jan 2017. • Irene Santos, Juan José Murillo-Fuentes, Eva Arias de Reyna and Pablo M. Olmos, "Turbo EP-based Equalization: a Filter-Type Implementation," IEEE Transactions on Communications, Sep 2017, Accepted. [Online] Available: https://ieeexplore.ieee.org/document/8353388/ • Irene Santos, Juan José Murillo-Fuentes, Eva Arias-de-Reyna and Pablo M. Olmos, "Probabilistic Equalization With a Smoothing Expectation Propagation Approach," IEEE Transactions on Wireless Communications, vol. 16, no. 5, pp. 2950-2962, May 2017. • Irene Santos, Juan José Murillo-Fuentes and Eva Arias-de-Reyna, "Equalization with Expectation Propagation at Smoothing Level," To be submitted. [Online] Available: https://arxiv.org/abs/1809.00806 • Irene Santos and Juan José Murillo-Fuentes, "EP-based turbo detection for MIMO receivers and large-scale systems," IEEE Transactions on Vehicular Technology, May 2018, Under review. [Online] Available: https://arxiv.org/abs/1805.05065 • Irene Santos, Juan José Murillo-Fuentes, and Pablo M. Olmos, "Block expectation propagation equalization for ISI channels," 23rd European Signal Processing Conference (EUSIPCO 2015), Nice, 2015, pp. 379-383. • Irene Santos, and Juan José Murillo-Fuentes, "Improved probabilistic EPbased receiver for MIMO systems and high-order modulations," XXXIII Simposium Nacional de la Unión Científica Internacional de Radio (URSI 2018), Granada, 2018.En el ámbito de las comunicaciones digitales, es necesario un receptor digital que proporcione una estimación de los símbolos transmitidos. Los decodificadores de canal actuales se benefician enormemente de estimaciones suaves (probabilísticas) de cada símbolo transmitido, en vez de estimaciones duras. Por este motivo, los receptores digitales deben diseñarse para proporcionar la probabilidad de cada posible símbolo que fue transmitido en base a la señal recibida y corrupta. Dado que la inferencia exacta puede no ser posible en términos de complejidad para escenarios de alto orden, es necesario recurrir a inferencia aproximada, como por ejemplo el criterio de linear minimum-mean-square-error (LMMSE). El LMMSE aproxima la información a priori discreta de los símbolos transmitidos con una distribución Gaussiana, lo cual provoca una degradación en su resultado. En esta tesis, se aplica una técnica alternativa de inferencia estadística para diseñar un receptor digital probabilístico de comunicaciones digitales. En concreto, se investiga el algoritmo expectation propagation (EP) con el objetivo de encontrar la función densidad de probabilidad (pdf) a posteriori Gaussiana que minimiza la divergencia de Kullback-Leibler (KL) con respecto a la pdf a posteriori verdadera. Se estudian dos escenarios de comunicaciones digitales diferentes: un sistema de comunicaciones single-input single-output (SISO) con canales con memoria y un sistema multiple-input multiple-output (MIMO) con canales sin memoria. Para el escenario SISO se proponen tres diseños diferentes para un igualador probabilístico, tanto simple como turbo, que está basado en el algoritmo EP: una versión bloque, una versión filtrada que emula el comportamiento de un filtroWiener y una versión smoothing que funciona de forma similar a un Kalman smoother. Finalmente, la implementación del EP en bloque se adapta también para escenarios MIMO con realimentación desde el decodificador. En ambos escenarios, el EP se aplica de forma iterativa, incluyendo un mecanismo de damping y un control para evitar valores de varianzas negativas, que darían lugar a inestabilidades (especialmente, en constelaciones de alto orden). Los resultados experimentales que se incluyen en la tesis muestran que, cuando el algoritmo EP se aplica a sistemas de comunicaciones, se mejora notablemente el resultado de otras propuestas anteriores que existen en la literatura, con un pequeño incremento de la complejidad que es proporcional a la carga del LMMSE. Estos resultados también demuestran la robustez del algoritmo incluso para modulaciones de alto orden, canales con bastante memoria y un gran número de antenas. Las principales contribuciones de esta tesis se han publicado en cuatro artículos de revista (uno de ellos todavía bajo revisión) y dos artículos de conferencia. Otro artículo adicional se encuentra en preparación y se enviaría próximamente a una revista. Estos se citan a continuación: • Irene Santos, Juan José Murillo-Fuentes, Rafael Boloix-Tortosa, Eva Arias de Reyna and Pablo M. Olmos, "Expectation Propagation as Turbo Equalizer in ISI Channels," IEEE Transactions on Communications, vol. 65, no.1, pp. 360-370, Jan 2017. • Irene Santos, Juan José Murillo-Fuentes, Eva Arias de Reyna and Pablo M. Olmos, "Turbo EP-based Equalization: a Filter-Type Implementation," IEEE Transactions on Communications, Sep 2017, Aceptado. [Online] Disponible: https://ieeexplore.ieee.org/document/8353388/ • Irene Santos, Juan José Murillo-Fuentes, Eva Arias-de-Reyna and Pablo M. Olmos, "Probabilistic Equalization With a Smoothing Expectation Propagation Approach," IEEE Transactions on Wireless Communications, vol. 16, no. 5, pp. 2950-2962, May 2017. • Irene Santos, Juan José Murillo-Fuentes and Eva Arias-de-Reyna, "Equalization with Expectation Propagation at Smoothing Level," En preparación. [Online] Disponible: https://arxiv.org/abs/1809.00806 • Irene Santos and Juan José Murillo-Fuentes, "EP-based turbo detection for MIMO receivers and large-scale systems," IEEE Transactions on Vehicular Technology, May 2018, En revisión. [Online] Disponible: https://arxiv.org/abs/1805.05065 • Irene Santos, Juan José Murillo-Fuentes, and Pablo M. Olmos, "Block expectation propagation equalization for ISI channels," 23rd European Signal Processing Conference (EUSIPCO 2015), Nice, 2015, pp. 379-383. • Irene Santos, and Juan José Murillo-Fuentes, "Improved probabilistic EPbased receiver for MIMO systems and high-order modulations," XXXIII Simposium Nacional de la Unión Científica Internacional de Radio (URSI 2018), Granada, 2018

Channel Estimation in Coded Modulation Systems

With the outstanding performance of coded modulation techniques in fading channels, much research efforts have been carried out on the design of communication systems able to operate at low signal-to-noise ratios (SNRs). From this perspective, the so-called iterative decoding principle has been applied to many signal processing tasks at the receiver: demodulation, detection, decoding, synchronization and channel estimation. Nevertheless, at low SNRs, conventional channel estimators do not perform satisfactorily. This thesis is mainly concerned with channel estimation issues in coded modulation systems where different diversity techniques are exploited to combat fading in single or multiple antenna systems. First, for single antenna systems in fast time-varying fading channels, the thesis focuses on designing a training sequence by exploiting signal space diversity (SSD). Motivated by the power/bandwidth efficiency of the SSD technique, the proposed training sequence inserts pilot bits into the coded bits prior to constellation mapping and signal rotation. This scheme spreads the training sequence during a transmitted codeword and helps the estimator to track fast variation of the channel gains. A comprehensive comparison between the proposed training scheme and the conventional training scheme is then carried out, which reveals several interesting conclusions with respect to both error performance of the system and mean square error of the channel estimator. For multiple antenna systems, different schemes are examined in this thesis for the estimation of block-fading channels. For typical coded modulation systems with multiple antennas, the first scheme makes a distinction between the iteration in the channel estimation and the iteration in the decoding. Then, the estimator begins iteration when the soft output of the decoder at the decoding iteration meets some specified reliability conditions. This scheme guarantees the convergence of the iterative receiver with iterative channel estimator. To accelerate the convergence process and decrease the complexity of successive iterations, in the second scheme, the channel estimator estimates channel state information (CSI) at each iteration with a combination of the training sequence and soft information. For coded modulation systems with precoding technique, in which a precoder is used after the modulator, the training sequence and data symbols are combined using a linear precoder to decrease the required training overhead. The power allocation and the placement of the training sequence to be precoded are obtained based on a lower bound on the mean square error of the channel estimation. It is demonstrated that considerable performance improvement is possible when the training symbols are embedded within data symbols with an equi-spaced pattern. In the last scheme, a joint precoder and training sequence is developed to maximize the achievable coding gain and diversity order under imperfect CSI. In particular, both the asymptotic performance behavior of the system with the precoded training scheme under imperfect CSI and the mean square error of the channel estimation are derived to obtain achievable diversity order and coding gain. Simulation results demonstrate that the joint optimized scheme outperforms the existing training schemes for systems with given precoders in terms of error rate and the amount of training overhead

Impact of signaling schemes on iterative linear minimum-mean-square-error detection

In this paper, we study the iterative detection problem for a coded system with multi-ary modulation. We show that, with iterative linear minimum-mean-square-error (LMMSE) detection, superposition coded modulation (SCM) can provide performance superior to that with other traditional signaling schemes used in trellis coded modulation (TCM) and bit-interleaved coded modulation (BICM). This finding provides a useful guideline for system design considering inter-symbol interference (ISI) and other forms of interference. Simulation results are provided to illustrate the efficiency of the iterative LMMSE detection with different signaling schemes. © 2008 IEEE

Reduced Receivers for Faster-than-Nyquist Signaling and General Linear Channels

Fast and reliable data transmission together with high bandwidth efficiency are important design aspects in a modern digital communication system. Many different approaches exist but in this thesis bandwidth efficiency is obtained by increasing the data transmission rate with the faster-than-Nyquist (FTN) framework while keeping a fixed power spectral density (PSD). In FTN consecutive information carrying symbols can overlap in time and in that way introduce a controlled amount of intentional intersymbol interference (ISI). This technique was introduced already in 1975 by Mazo and has since then been extended in many directions. Since the ISI stemming from practical FTN signaling can be of significant duration, optimum detection with traditional methods is often prohibitively complex, and alternative equalization methods with acceptable complexity-performance tradeoffs are needed. The key objective of this thesis is therefore to design reduced-complexity receivers for FTN and general linear channels that achieve optimal or near-optimal performance. Although the performance of a detector can be measured by several means, this thesis is restricted to bit error rate (BER) and mutual information results. FTN signaling is applied in two ways: As a separate uncoded narrowband communication system or in a coded scenario consisting of a convolutional encoder, interleaver and the inner ISI mechanism in serial concatenation. Turbo equalization where soft information in the form of log likelihood ratios (LLRs) is exchanged between the equalizer and the decoder is a commonly used decoding technique for coded FTN signals. The first part of the thesis considers receivers and arising stability problems when working within the white noise constraint. New M-BCJR algorithms for turbo equalization are proposed and compared to reduced-trellis VA and BCJR benchmarks based on an offset label idea. By adding a third low-complexity M-BCJR recursion, LLR quality is improved for practical values of M. M here measures the reduced number of BCJR computations for each data symbol. An improvement of the minimum phase conversion that sharpens the focus of the ISI model energy is proposed. When combined with a delayed and slightly mismatched receiver, the decoding allows a smaller M without significant loss in BER. The second part analyzes the effect of the internal metric calculations on the performance of Forney- and Ungerboeck-based reduced-complexity equalizers of the M-algorithm type for both ISI and multiple-input multiple-output (MIMO) channels. Even though the final output of a full-complexity equalizer is identical for both models, the internal metric calculations are in general different. Hence, suboptimum methods need not produce the same final output. Additionally, new models working in between the two extremes are proposed and evaluated. Note that the choice of observation model does not impact the detection complexity as the underlying algorithm is unaltered. The last part of the thesis is devoted to a different complexity reducing approach. Optimal channel shortening detectors for linear channels are optimized from an information theoretical perspective. The achievable information rates of the shortened models as well as closed form expressions for all components of the optimal detector of the class are derived. The framework used in this thesis is more general than what has been previously used within the area