Non-linear Convolution Filters for CNN-based Learning

During the last years, Convolutional Neural Networks (CNNs) have achieved state-of-the-art performance in image classification. Their architectures have largely drawn inspiration by models of the primate visual system. However, while recent research results of neuroscience prove the existence of non-linear operations in the response of complex visual cells, little effort has been devoted to extend the convolution technique to non-linear forms. Typical convolutional layers are linear systems, hence their expressiveness is limited. To overcome this, various non-linearities have been used as activation functions inside CNNs, while also many pooling strategies have been applied. We address the issue of developing a convolution method in the context of a computational model of the visual cortex, exploring quadratic forms through the Volterra kernels. Such forms, constituting a more rich function space, are used as approximations of the response profile of visual cells. Our proposed second-order convolution is tested on CIFAR-10 and CIFAR-100. We show that a network which combines linear and non-linear filters in its convolutional layers, can outperform networks that use standard linear filters with the same architecture, yielding results competitive with the state-of-the-art on these datasets.Comment: 9 pages, 5 figures, code link, ICCV 201

Multi Look-Up Table Digital Predistortion for RF Power Amplifier Linearization

Premi extraordinari doctorat curs 2007-2008, àmbit d’Enginyeria de les TICAquesta Tesi Doctoral se centra en el disseny d'un nou linealitzador de Predistorsió Digital (Digital Predistortion - DPD) capaç de compensar la dinàmica i els efectes no lineals introduïts pels Amplificadors de Potència (Power Amplifiers - PAs). Un dels trets més rellevants d'aquest nou predistorsionador digital i adaptatiu consisteix en ser deduïble a partir d'un model de PA anomenat Nonlinear Auto-Regressive Moving Average (NARMA). A més, la seva arquitectura multi-LUT (multi-Taula) permet la implementació en un dispositiu Field Programmable Gate Array (FPGA).La funció de predistorsió es realitza en banda base, per tant, és independent de la banda freqüencial on es durà a terme l'amplificació del senyal de RF, el que pot resultar útil si tenim en compte escenaris multibanda o reconfigurables. D'altra banda, el fet que aquest DPD tingui en compte els efectes de memòria introduïts pel PA, representa una clara millora de les prestacions aconseguides per un simple DPD sense memòria. En comparació amb d'altres DPDs basats en models més computacionalment complexos, com és el cas de les xarxes neuronals amb memòria (Time-Delayed Neural Networks - TDNN), la estructura recursiva del DPD proposat permet reduir el nombre de LUTs necessàries per compensar els efectes de memòria del PA. A més, la seva estructura multi-LUT permet l'escalabilitat, és a dir, activar or desactivar les LUTs que formen el DPD en funció de la dinàmica que presenti el PA.En una primera aproximació al disseny del DPD, és necessari identificar el model NARMA del PA. Un dels majors avantatges que presenta el model NARMA és la seva capacitat per trobar un compromís entre la fidelitat en l'estimació del PA i la complexitat computacional introduïda. Per reforçar aquest compromís, l' ús d'algoritmes heurístics de cerca, com són el Simulated Annealing o els Genetic Algorithms, s'utilitzen per trobar els retards que millor caracteritzen la memòria del PA i per tant, permeten la reducció del nombre de coeficients necessaris per caracteritzar-la. Tot i així, la naturalesa recursiva del model NARMA comporta que, de cara a garantir l'estabilitat final del DPD, cal dur a terme un estudi previ sobre l'estabilitat del model.Una vegada s'ha obtingut el model NARMA del PA i s'ha verificat l'estabilitat d'aquest, es procedeix a l'obtenció de la funció de predistorsió a través del mètode d'identificació predictiu. Aquest mètode es basa en la continua identificació del model NARMA del PA i posteriorment, a partir del model obtingut, es força al PA perquè es comporti de manera lineal. Per poder implementar la funció de predistorsió en la FPGA, cal primer expressar-la en forma de combinacions en paral·lel i cascada de les anomenades Cel·les Bàsiques de Predistorsió (BPCs), que són les unitats fonamentals que composen el DPD. Una BPC està formada per un multiplicador complex, un port RAM dual que actua com a LUT (taula de registres) i un calculador d'adreces. Les LUTs s'omplen tenint en compte una distribució uniforme dels continguts i l'indexat d'aquestes es duu a terme mitjançant el mòdul de l'envoltant del senyal. Finalment, l'adaptació del DPD consisteix en monitoritzar els senyals d'entrada i sortida del PA i anar duent a terme actualitzacions periòdiques del contingut de les LUTs que formen les BPCs. El procés d'adaptació del contingut de les LUTs es pot dur a terme en la mateixa FPGA encarregada de fer la funció de predistorsió, o de manera alternativa, pot ser duta a terme per un dispositiu extern (com per exemple un DSP - Digital Signal Processor) en una escala de temps més relaxada. Per validar l'exposició teòrica i provar el bon funcionalment del DPD proposat en aquesta Tesi, es proporcionen resultats tant de simulació com experimentals que reflecteixen els objectius assolits en la linealització del PA. A més, certes qüestions derivades de la implementació pràctica, tals com el consum de potència o la eficiència del PA, són també tractades amb detall.This Ph.D. thesis addresses the design of a new Digital Predistortion (DPD) linearizer capable to compensate the unwanted nonlinear and dynamic behavior of power amplifiers (PAs). The distinctive characteristic of this new adaptive DPD is its deduction from a Nonlinear Auto Regressive Moving Average (NARMA) PA behavioral model and its particular multi look-up table (LUT) architecture that allows its implementation in a Field Programmable Gate Array (FPGA) device.The DPD linearizer presented in this thesis operates at baseband, thus becoming independent on the final RF frequency band and making it suitable for multiband or reconfigurable scenarios. Moreover, the proposed DPD takes into account PA memory effects compensation which representsan step forward in overcoming classical limitations of memoryless predistorters. Compared to more computational complex DPDs with dynamic compensation, such Time-Delayed Neural Networks (TDNN), this new DPD takes advantage of the recursive nature of the NARMA structure to relax the number of LUTs required to compensate memory effects in PAs. Furthermore, its parallel multi-LUT architecture is scalable, that is, permits enabling or disabling the contribution of specific LUTs depending on the dynamics presented by a particular PA.In a first approach, it is necessary to identify a NARMA PA behavioral model. The extraction of PA behavioral models for DPD linearization purposes is carried out by means of input and output complex envelope signal observations. One of the major advantages of the NARMA structure regards its capacity to deal with the existing trade-off between computational complexity and accuracy in PA behavioral modeling. To reinforce this compromise, heuristic search algorithms such the Simulated Annealing or Genetic Algorithms are utilized to find the best sparse delays that permit accurately reproducing the PA nonlinear dynamic behavior. However, due to the recursive nature of the NARMA model, an stability test becomes a previous requisite before advancing towards DPD linearization.Once the PA model is identified and its stability verified, the DPD function is extracted applying a predictive predistortion method. This identification method relies just on the PA NARMA model and consists in adaptively forcing the PA to behave as a linear device. Focusing in the DPD implementation, it is possible to map the predistortion function in a FPGA, but to fulfill this objective it is first necessary to express the predistortion function as a combined set of LUTs.In order to store the DPD function into a FPGA, it has to be stated in terms of parallel and cascade Basic Predistortion Cells (BPCs), which are the fundamental building blocks of the NARMA based DPD. A BPC is formed by a complex multiplier, a dual port RAM memory block acting as LUT and an address calculator. The LUT contents are filled following an uniform spacing procedure and its indexing is performed with the amplitude (modulus) of the signal's envelope.Finally, the DPD adaptation consists in monitoring the input-output data and performing frequent updates of the LUT contents that conform the BPCs. This adaptation process can be carried out in the same FPGA in charge of performing the DPD function, or alternatively can be performed by an external device (i.e. a DSP device) in a different time-scale than real-time operation.To support all the theoretical design and to prove the linearization performance achieved by this new DPD, simulation and experimental results are provided. Moreover, some issues derived from practical experimentation, such as power consumption and efficiency, are also reported and discussed within this thesis.Award-winningPostprint (published version

Method and system for training dynamic nonlinear adaptive filters which have embedded memory

Described herein is a method and system for training nonlinear adaptive filters (or neural networks) which have embedded memory. Such memory can arise in a multi-layer finite impulse response (FIR) architecture, or an infinite impulse response (IIR) architecture. We focus on filter architectures with separate linear dynamic components and static nonlinear components. Such filters can be structured so as to restrict their degrees of computational freedom based on a priori knowledge about the dynamic operation to be emulated. The method is detailed for an FIR architecture which consists of linear FIR filters together with nonlinear generalized single layer subnets. For the IIR case, we extend the methodology to a general nonlinear architecture which uses feedback. For these dynamic architectures, we describe how one can apply optimization techniques which make updates closer to the Newton direction than those of a steepest descent method, such as backpropagation. We detail a novel adaptive modified Gauss-Newton optimization technique, which uses an adaptive learning rate to determine both the magnitude and direction of update steps. For a wide range of adaptive filtering applications, the new training algorithm converges faster and to a smaller value of cost than both steepest-descent methods such as backpropagation-through-time, and standard quasi-Newton methods. We apply the algorithm to modeling the inverse of a nonlinear dynamic tracking system 5, as well as a nonlinear amplifier 6

NONLINEAR SYSTEM MODELING UTILIZING NEURAL NETWORKS: AN APPLICATION TO THE DOUBLE SIDED ARC WELDING PROCESS

The need and desire to create robust and accurate joining of materials has been one of up most importance throughout the course of history. Many forms have often been employed, but none exhibit the strength or durability as the weld. This study endeavors to explore some of the aspects of welding, more specifically relating to the Double Sided Arc Welding process and how best to model the dynamic non-linear response of such a system. Concepts of the Volterra series, NARMAX approximation and neural networks are explored. Fundamental methods of the neural network, including radial basis functions, and Back-propagation are investigated

Extended evaluation of the Volterra-Neural Network for model compression

When large models are used for a classification task, model compression is necessary because there are transmission, space, time or computing constraints that have to be fulfilled. Multilayer Perceptron (MLP) models are traditionally used as a classifier, but depending on the problem, they may need a large number of parameters (neuron functions, weights and bias) to obtain an acceptable performance. This work extends the evaluation of a technique to compress an array of MLPs, through the outputs of a Volterra-Neural Network (Volterra-NN), maintaining its classification performance. The obtained results show that these outputs can be used to build an array of (Volterra-NN) that needs significantly less parameters than the original array of MLPs, furthermore having the same high accuracy in most of the cases. The Volterra-NN compression capabilities have been tested by solving several kind of classification problems. Experimental results are presented on three well-known databases: Letter Recognition, Pen-Based Recognition of Handwritten Digits, and Face recognition databases.Sociedad Argentina de Informática e Investigación Operativ