A Low-complexity Complex-valued Activation Function for Fast and Accurate Spectral Domain Convolutional Neural Network

Conventional Convolutional Neural Networks (CNNs), which are realized in spatial domain, exhibit high computational complexity. This results in high resource utilization and memory usage and makes them unsuitable for implementation in resource and energy-constrained embedded systems. A promising approach for low-complexity and high-speed solution is to apply CNN modeled in the spectral domain. One of the main challenges in this approach is the design of activation functions. Some of the proposed solutions perform activation functions in spatial domain, necessitating multiple and computationally expensive spatial-spectral domain switching. On the other hand, recent work on spectral activation functions resulted in very computationally intensive solutions. This paper proposes a complex-valued activation function for spectral domain CNNs that only transmits input values that have positive-valued real or imaginary component. This activation function is computationally inexpensive in both forward and backward propagation and provides sufficient nonlinearity that ensures high classification accuracy. We apply this complex-valued activation function in a LeNet-5 architecture and achieve an accuracy gain of up to 7% for MNIST and 6% for Fashion MNIST dataset, while providing up to 79% and 85% faster inference times, respectively, over state-of-the-art activation functions for spectral domain

Accelerating fully spectral CNNs with adaptive activation functions on FPGA

Computing convolutional layers in frequency domain can largely reduce the computation overhead for training and inference of convolutional neural networks (CNNs). However, existing designs with such an idea require repeated spatial- and frequency-domain transforms due to the absence of nonlinear functions in the frequency domain, as such it makes the benefit less attractive for low-latency inference. This paper presents a fully spectral CNN approach by proposing a novel adaptive Rectified Linear Unit (ReLU) activation in spectral domain. The proposed design maintains the non-linearity in the network while taking into account the hardware efficiency in algorithm level. The spectral model size is further optimized by merging and fusing layers. Then, a customized hardware architecture is proposed to implement the designed spectral network on FPGA device with DSP optimizations for 8-bit fixed point multipliers. Our hardware accelerator is implemented on Intel's Arria 10 device and applied to the MNIST, SVHN, AT&T and CIFAR-10 datasets. Experimental results show a speed improvement of 6 × ~ 10 × and 4 × ~ 5.7 × compared to state-of-the-art spatial or FFT-based designs respectively, while achieving similar accuracy across the benchmark datasets

Designing energy-efficient computing systems using equalization and machine learning

As technology scaling slows down in the nanometer CMOS regime and mobile computing becomes more ubiquitous, designing energy-efficient hardware for mobile systems is becoming increasingly critical and challenging. Although various approaches like near-threshold computing (NTC), aggressive voltage scaling with shadow latches, etc. have been proposed to get the most out of limited battery life, there is still no “silver bullet” to increasing power-performance demands of the mobile systems. Moreover, given that a mobile system could operate in a variety of environmental conditions, like different temperatures, have varying performance requirements, etc., there is a growing need for designing tunable/reconfigurable systems in order to achieve energy-efficient operation. In this work we propose to address the energy- efficiency problem of mobile systems using two different approaches: circuit tunability and distributed adaptive algorithms. Inspired by the communication systems, we developed feedback equalization based digital logic that changes the threshold of its gates based on the input pattern. We showed that feedback equalization in static complementary CMOS logic enabled up to 20% reduction in energy dissipation while maintaining the performance metrics. We also achieved 30% reduction in energy dissipation for pass-transistor digital logic (PTL) with equalization while maintaining performance. In addition, we proposed a mechanism that leverages feedback equalization techniques to achieve near optimal operation of static complementary CMOS logic blocks over the entire voltage range from near threshold supply voltage to nominal supply voltage. Using energy-delay product (EDP) as a metric we analyzed the use of the feedback equalizer as part of various sequential computational blocks. Our analysis shows that for near-threshold voltage operation, when equalization was used, we can improve the operating frequency by up to 30%, while the energy increase was less than 15%, with an overall EDP reduction of ≈10%. We also observe an EDP reduction of close to 5% across entire above-threshold voltage range. On the distributed adaptive algorithm front, we explored energy-efficient hardware implementation of machine learning algorithms. We proposed an adaptive classifier that leverages the wide variability in data complexity to enable energy-efficient data classification operations for mobile systems. Our approach takes advantage of varying classification hardness across data to dynamically allocate resources and improve energy efficiency. On average, our adaptive classifier is ≈100× more energy efficient but has ≈1% higher error rate than a complex radial basis function classifier and is ≈10× less energy efficient but has ≈40% lower error rate than a simple linear classifier across a wide range of classification data sets. We also developed a field of groves (FoG) implementation of random forests (RF) that achieves an accuracy comparable to Convolutional Neural Networks (CNN) and Support Vector Machines (SVM) under tight energy budgets. The FoG architecture takes advantage of the fact that in random forests a small portion of the weak classifiers (decision trees) might be sufficient to achieve high statistical performance. By dividing the random forest into smaller forests (Groves), and conditionally executing the rest of the forest, FoG is able to achieve much higher energy efficiency levels for comparable error rates. We also take advantage of the distributed nature of the FoG to achieve high level of parallelism. Our evaluation shows that at maximum achievable accuracies FoG consumes ≈1.48×, ≈24×, ≈2.5×, and ≈34.7× lower energy per classification compared to conventional RF, SVM-RBF , Multi-Layer Perceptron Network (MLP), and CNN, respectively. FoG is 6.5× less energy efficient than SVM-LR, but achieves 18% higher accuracy on average across all considered datasets

Quantum information outside quantum information

Quantum theory, as counter-intuitive as a theory can get, has turned out to make predictions of the physical world that match observations so precisely that it has been described as the most accurate physical theory ever devised. Viewing quantum entanglement, superposition and interference not as undesirable necessities but as interesting resources paved the way to the development of quantum information science. This area studies the processing, transmission and storage of information when one accounts that information is physical and subjected to the laws of nature that govern the systems it is encoded in. The development of the consequences of this idea, along with the great advances experienced in the control of individual quantum systems, has led to what is now known as the second quantum revolution, in which quantum information science has emerged as a fully-grown field. As such, ideas and tools developed within the framework of quantum information theory begin to permeate to other fields of research. This Ph.D. dissertation is devoted to the use of concepts and methods akin to the field of quantum information science in other areas of research. In the same way, it also considers how encoding information in quantum degrees of freedom may allow further development of well-established research fields and industries. This is, this thesis aims to the study of quantum information outside the field of quantum information. Four different areas are visited. A first question posed is that of the role of quantum information in quantum field theory, with a focus in the quantum vacuum. It is known that the quantum vacuum contains entanglement, but it remains unknown whether it can be accessed and exploited in experiments. We give crucial steps in this direction by studying the extraction of vacuum entanglement in realistic models of light-matter interaction, and by giving strict mathematical conditions of general applicability that must be fulfilled for extraction to be possible at all. Another field where quantum information methods can offer great insight is in that of quantum thermodynamics, where the idealizations made in macroscopic thermodynamics break down. Making use of a quintessential framework of quantum information and quantum optics, we study the cyclic operation of a microscopic heat engine composed by a single particle reciprocating between two finite-size baths, focusing on the consequences of the removal of the macroscopic idealizations. One more step down the stairs to applications in society, we analyze the impact that encoding information in quantum systems and processing it in quantum computers may have in the field of machine learning. A great desideratum in this area, largely obstructed by computational power, is that of explainable models which not only make predictions but also provide information about the decision process that triggers them. We develop an algorithm to train neural networks using explainable techniques that exploits entanglement and superposition to execute efficiently in quantum computers, in contrast with classical counterparts. Furthermore, we run it in state-of-the-art quantum computers with the aim of assessing the viability of realistic implementations. Lastly, and encompassing all the above, we explore the notion of causality in quantum mechanics from an information-theoretic point of view. While it is known since the work of John S. Bell in 1964 that, for a same causal pattern, quantum systems can generate correlations between variables that are impossible to obtain employing only classical systems, there is an important lack of tools to study complex causal effects whenever a quantum behavior is expected. We fill this gap by providing general methods for the characterization of the quantum correlations achievable in complex causal patterns. Closing the circle, we make use of these tools to find phenomena of fundamental and experimental relevance back in quantum information.La teoría cuántica, la más extraña y antiintuitiva de las teorías físicas, es también considerada como la teoría más precisa jamás desarrollada. La interpretación del entrelazamiento, la superposición y la interferencia como interesantes recursos aprovechables cimentó el desarrollo de la teoría cuántica de la información (QIT), que estudia el procesado, transmisión y almacenamiento de información teniendo en cuenta que ésta es física, en tanto a que está sujeta a las leyes de la naturaleza que gobiernan los sistemas en que se codifica. El desarrollo de esta idea, en conjunción con los recientes avances en el control de sistemas cuánticos individuales, ha dado lugar a la conocida como segunda revolución cuántica, en la cual la QIT ha emergido como un área de estudio con denominación propia. A consecuencia de su desarrollo actual, ideas y herramientas creadas en su seno comienzan a permear a otros ámbitos de investigación. Esta tesis doctoral está dedicada a la utilización de conceptos y métodos originales del campo de información cuántica en otras áreas. También considera cómo la codificación de información en grados de libertad cuánticos puede afectar el futuro desarrollo de áreas de investigación e industrias bien establecidas. Es decir, esta tesis tiene como objetivo el estudio de la información cuántica fuera de la información cuántica, haciendo hincapié en cuatro ámbitos diferentes. Una primera cuestión propuesta es la del papel de la información cuántica en la teoría cuántica de campos, con especial énfasis en el vacío cuántico. Es conocido que el vacío cuántico contiene entrelazamiento, pero aún se desconoce éste es accesible para su uso en realizaciones experimentales. En esta tesis se dan pasos cruciales en esta dirección mediante el estudio de la extracción de entrelazamiento en modelos realistas de la interacción materia-radiación, y dando condiciones matemáticas estrictas que deben ser satisfechas para que dicha extracción sea posible. Otro campo en el cual métodos propios de QIT pueden ofrecer nuevos puntos de vista es en termodinámica cuántica. A través del uso de un marco de trabajo ampliamente utilizado en información y óptica cuánticas, estudiamos la operación cíclica de un motor térmico microscópico que alterna entre dos baños térmicos de tamaño finito, prestando especial atención a las consecuencias de la eliminación de las idealizaciones macroscópicas utilizadas en termodinámica macroscópica. Acercándonos a aplicaciones industriales, analizamos el potencial impacto de codificar y procesar información en sistemas cuánticos en el ámbito del aprendizaje automático. Un fin codiciado en esta área, inaccesible debido a su coste computacional, es el de modelos explicativos que realicen predicciones, y además ofrezcan información acerca del proceso de decisión que las genera. Presentamos un algoritmo de entrenamiento de redes neuronales con técnicas explicativas que hace uso del entrelazamiento y la superposición para tener una ejecución eficiente en ordenadores cuánticos, en comparación con homólogos clásicos. Además, ejecutamos el algoritmo en ordenadores cuánticos contemporáneos con el objetivo de evaluar la viabilidad de implementaciones realistas. Finalmente, y englobando todo lo anterior, exploramos la noción de causalidad en mecánica cuántica desde el punto de vista de la teoría de la información. A pesar de que es conocido que para un mismo patrón causal existen sistemas cuánticos que dan lugar a correlaciones imposibles de generar por mediación de sistemas clásicos, existe una notable falta de herramientas para estudiar efectos causales cuánticos complejos. Cubrimos esta falta mediante métodos generales para la caracterización de las correlaciones cuánticas que pueden ser generadas en estructuras causales complejas. Cerrando el círculo, usamos estas herramientas para encontrar fenómenos de relevancia fundamental y experimental en la información cuántic

Problems in Control, Estimation, and Learning in Complex Robotic Systems

In this dissertation, we consider a range of different problems in systems, control, and learning theory and practice. In Part I, we look at problems in control of complex networks. In Chapter 1, we consider the performance analysis of a class of linear noisy dynamical systems. In Chapter 2, we look at the optimal design problems for these networks. In Chapter 3, we consider dynamical networks where interactions between the networks occur randomly in time. And in the last chapter of this part, in Chapter 4, we look at dynamical networks wherein coupling between the subsystems (or agents) changes nonlinearly based on the difference between the state of the subsystems. In Part II, we consider estimation problems wherein we deal with a large body of variables (i.e., at large scale). This part starts with Chapter 5, in which we consider the problem of sampling from a dynamical network in space and time for initial state recovery. In Chapter 6, we consider a similar problem with the difference that the observations instead of point samples become continuous observations that happen in Lebesgue measurable observations. In Chapter 7, we consider an estimation problem in which the location of a robot during the navigation is estimated using the information of a large number of surrounding features and we would like to select the most informative features using an efficient algorithm. In Part III, we look at active perception problems, which are approached using reinforcement learning techniques. This part starts with Chapter 8, in which we tackle the problem of multi-agent reinforcement learning where the agents communicate and classify as a team. In Chapter 9, we consider a single agent version of the same problem, wherein a layered architecture replaces the architectures of the previous chapter. Then, we use reinforcement learning to design the meta-layer (to select goals), action-layer (to select local actions), and perception-layer (to conduct classification)