Lightweight Probabilistic Deep Networks

Even though probabilistic treatments of neural networks have a long history, they have not found widespread use in practice. Sampling approaches are often too slow already for simple networks. The size of the inputs and the depth of typical CNN architectures in computer vision only compound this problem. Uncertainty in neural networks has thus been largely ignored in practice, despite the fact that it may provide important information about the reliability of predictions and the inner workings of the network. In this paper, we introduce two lightweight approaches to making supervised learning with probabilistic deep networks practical: First, we suggest probabilistic output layers for classification and regression that require only minimal changes to existing networks. Second, we employ assumed density filtering and show that activation uncertainties can be propagated in a practical fashion through the entire network, again with minor changes. Both probabilistic networks retain the predictive power of the deterministic counterpart, but yield uncertainties that correlate well with the empirical error induced by their predictions. Moreover, the robustness to adversarial examples is significantly increased.Comment: To appear at CVPR 201

Multiple description video coding for stereoscopic 3D

In this paper, we propose an MDC schemes for stereoscopic 3D video. In the literature, MDC has previously been applied in 2D video but not so much in 3D video. The proposed algorithm enhances the error resilience of the 3D video using the combination of even and odd frame based MDC while retaining good temporal prediction efficiency for video over error-prone networks. Improvements are made to the original even and odd frame MDC scheme by adding a controllable amount of side information to improve frame interpolation at the decoder. The side information is also sent according to the video sequence motion for further improvement. The performance of the proposed algorithms is evaluated in error free and error prone environments especially for wireless channels. Simulation results show improved performance using the proposed MDC at high error rates compared to the single description coding (SDC) and the original even and odd frame MDC

Evidence accumulation in a Laplace domain decision space

Evidence accumulation models of simple decision-making have long assumed that the brain estimates a scalar decision variable corresponding to the log-likelihood ratio of the two alternatives. Typical neural implementations of this algorithmic cognitive model assume that large numbers of neurons are each noisy exemplars of the scalar decision variable. Here we propose a neural implementation of the diffusion model in which many neurons construct and maintain the Laplace transform of the distance to each of the decision bounds. As in classic findings from brain regions including LIP, the firing rate of neurons coding for the Laplace transform of net accumulated evidence grows to a bound during random dot motion tasks. However, rather than noisy exemplars of a single mean value, this approach makes the novel prediction that firing rates grow to the bound exponentially, across neurons there should be a distribution of different rates. A second set of neurons records an approximate inversion of the Laplace transform, these neurons directly estimate net accumulated evidence. In analogy to time cells and place cells observed in the hippocampus and other brain regions, the neurons in this second set have receptive fields along a "decision axis." This finding is consistent with recent findings from rodent recordings. This theoretical approach places simple evidence accumulation models in the same mathematical language as recent proposals for representing time and space in cognitive models for memory.Comment: Revised for CB

MorphIC: A 65-nm 738k-Synapse/mm2^2 Quad-Core Binary-Weight Digital Neuromorphic Processor with Stochastic Spike-Driven Online Learning

Recent trends in the field of neural network accelerators investigate weight quantization as a means to increase the resource- and power-efficiency of hardware devices. As full on-chip weight storage is necessary to avoid the high energy cost of off-chip memory accesses, memory reduction requirements for weight storage pushed toward the use of binary weights, which were demonstrated to have a limited accuracy reduction on many applications when quantization-aware training techniques are used. In parallel, spiking neural network (SNN) architectures are explored to further reduce power when processing sparse event-based data streams, while on-chip spike-based online learning appears as a key feature for applications constrained in power and resources during the training phase. However, designing power- and area-efficient spiking neural networks still requires the development of specific techniques in order to leverage on-chip online learning on binary weights without compromising the synapse density. In this work, we demonstrate MorphIC, a quad-core binary-weight digital neuromorphic processor embedding a stochastic version of the spike-driven synaptic plasticity (S-SDSP) learning rule and a hierarchical routing fabric for large-scale chip interconnection. The MorphIC SNN processor embeds a total of 2k leaky integrate-and-fire (LIF) neurons and more than two million plastic synapses for an active silicon area of 2.86mm$^2$ in 65nm CMOS, achieving a high density of 738k synapses/mm$^2$. MorphIC demonstrates an order-of-magnitude improvement in the area-accuracy tradeoff on the MNIST classification task compared to previously-proposed SNNs, while having no penalty in the energy-accuracy tradeoff.Comment: This document is the paper as accepted for publication in the IEEE Transactions on Biomedical Circuits and Systems journal (2019), the fully-edited paper is available at https://ieeexplore.ieee.org/document/876400

Dynamic bandwidth allocation in ATM networks

Includes bibliographical references.This thesis investigates bandwidth allocation methodologies to transport new emerging bursty traffic types in ATM networks. However, existing ATM traffic management solutions are not readily able to handle the inevitable problem of congestion as result of the bursty traffic from the new emerging services. This research basically addresses bandwidth allocation issues for bursty traffic by proposing and exploring the concept of dynamic bandwidth allocation and comparing it to the traditional static bandwidth allocation schemes

Error resilience and concealment techniques for high-efficiency video coding

This thesis investigates the problem of robust coding and error concealment in High Efficiency Video Coding (HEVC). After a review of the current state of the art, a simulation study about error robustness, revealed that the HEVC has weak protection against network losses with significant impact on video quality degradation. Based on this evidence, the first contribution of this work is a new method to reduce the temporal dependencies between motion vectors, by improving the decoded video quality without compromising the compression efficiency. The second contribution of this thesis is a two-stage approach for reducing the mismatch of temporal predictions in case of video streams received with errors or lost data. At the encoding stage, the reference pictures are dynamically distributed based on a constrained Lagrangian rate-distortion optimization to reduce the number of predictions from a single reference. At the streaming stage, a prioritization algorithm, based on spatial dependencies, selects a reduced set of motion vectors to be transmitted, as side information, to reduce mismatched motion predictions at the decoder. The problem of error concealment-aware video coding is also investigated to enhance the overall error robustness. A new approach based on scalable coding and optimally error concealment selection is proposed, where the optimal error concealment modes are found by simulating transmission losses, followed by a saliency-weighted optimisation. Moreover, recovery residual information is encoded using a rate-controlled enhancement layer. Both are transmitted to the decoder to be used in case of data loss. Finally, an adaptive error resilience scheme is proposed to dynamically predict the video stream that achieves the highest decoded quality for a particular loss case. A neural network selects among the various video streams, encoded with different levels of compression efficiency and error protection, based on information from the video signal, the coded stream and the transmission network. Overall, the new robust video coding methods investigated in this thesis yield consistent quality gains in comparison with other existing methods and also the ones implemented in the HEVC reference software. Furthermore, the trade-off between coding efficiency and error robustness is also better in the proposed methods

딥 스파이킹 뉴럴 네트워크의 빠르고 정확한 정보 전달

학위논문 (박사) -- 서울대학교 대학원 : 공과대학 전기·컴퓨터공학부, 2021. 2. 윤성로.오늘 날 딥러닝의 큰 성공은 고성능 병렬 컴퓨팅 시스템의 발전과 복잡한 모델을 학습하기 위해 필요한 많은 양의 데이터가 수집되어 접근이 가능해진 점이라고 할 수 있다. 하지만 실제 세상에 존재하는 더 어려운 문제들을 풀고자할 때는 더욱 더 섬세하고 복잡한 모델과 이 모델을 성공적으로 학습할 수 있는 방대한 양의 데이터를 필요한다. 하지만 이러한 점들은 모델 수행 시 연산 오버헤드와 전력 소모를 급격하게 증가시킬 수 밖에 없다. 이러한 문제점들을 극복하는 여러 방법들 중 하나로 스파이킹 뉴럴 네트워크가 최근 많은 주목을 받고 있다. 스파이킹 뉴럴 네트워크는 제 3세대 인공 신경망으로 불리며 이벤트 중심의 동작을 기반으로 하여 저전력이 가장 큰 장점이다. 스파이킹 뉴럴 네트워크는 실제 인간의 뇌에서 뉴런들 간 정보를 전달하는 방식을 모방하며 스파이킹 뉴런을 연산 단위로 사용하고 있다. 스파이킹 뉴럴 네트워크는 생물학적 신경계와 동일하게 시간적 정보를 활용할 수 있기 때문에 매우 뛰어난 연산 능력을 가지고 있다. 하지만 스파이킹 뉴럴 네트워크는 이미지 분류와 같은 비교적 쉬운 응용에만 주로 사용되고 있으며 얕은 인공 신경망과 간단한 데이터셋에서만 주로 수행되고 있다. 이러한 제약이 존재하는 가장 큰 요인 중 하나는 스파이크 뉴럴 네트워크에 적합한 학습 알고리즘이 아직 존재하지 않기 때문이다. 스파이크로 정보를 전달하고 연산을 수행하기 때문에 미분이 불가능하다. 따라서 딥 뉴럴 네트워크에서 주로 사용되는 역전파 알고리즘의 사용이 불가능하다. 본 논문에서 딥 스파이킹 뉴럴 네트워크를 이미지 분류보다 더 어려운 회귀 문제 (객체 인식)에 적용해 보고, 딥 뉴럴 네트워크의 성능에 버금가는 객체 인식 모델을 스파이킹 뉴럴 네트워에서 처음으로 제안한다. 더 나아가, 객체 인식 모델의 성능과 지연시간, 에너지 효율성을 향상 시킬 수 있는 여러 방법들을 제안한다. 본 논문은 크게 두 가지 주제로 나누어 설명한다: (a) 딥 스파이킹 뉴럴 네트워크에서의 객체 인식 모델, (b) 딥 스파이킹 뉴럴 네트워크에서의 객체 인식 모델의 성능 및 효율성 향상. 제안하는 방법들을 통해 빠르고 정확한 객체 인식 모델을 딥 스파이킹 뉴럴 네트워크에서 수행할 수 있다. 첫 번째 방법은 딥 스파이킹 뉴럴 네트워크에서의 객체 인식 모델이다. 객체 인식 모델은 Spiking-YOLO로 부르고, 저자들이 아는 바에 의하면 PASCAL VOC, MS COCO와 같은 데이터 셋에서 딥 뉴럴 네트워크의 성능에 버금가는 결과를 보여준 첫 번째 스파이킹 뉴럴 네트워크를 기반으로 하는 객체 인식 모델이다. Spiking-YOLO에서는 크게 두 가지 방법을 제안한다. 첫번 째는 채널 별 가중치 정규화이고 두번째는 불균형 한계 전압을 가지는 양음수 뉴런이다. 두 가지 방법을 통해 빠르고 정확한 정보를 딥 스파이킹 뉴럴 네트워크에서 전달 가능하게 한다. 실험 결과, Spiking-YOLO는 PASCAL VOC와 MS COCO 데이터셋에서 딥 뉴럴 네트워크의 객체 인식률의 98%에 뛰어난 성능을 보였다. 또한 Spiking-YOLO가 뉴로모픽 칩에 구현되었음 가정하였을 때, Tiny YOLO보다 약 280의 에너지를 적게 소모하였고 기존의 DNN-to-SNN 전환 방법들 보다 2.3배에서 4배 더 빠르게 수렴하는 것을 확인할 수 있었다. 두 번째 방법은 스파이킹 뉴럴 네트워크에 조금 더 효율적인 연산 능력을 부여하는데 중점을 주고 있다. 비록 스파이킹 뉴럴 네트워크가 희박한 양의 스파이크로 정보를 효율적으로 전달하며 연산 오버헤드와 에너지 소모가 적지만, 두 가지 매우 중요한 문제들이 존재한다: (a) 지연속도: 좋은 성능을 내기 위해 필요한 타임스탭, (b) 시냅틱 연산수: 추론 시 생성된 총 스파이크의 수. 이러한 문제들을 적절히 해결하지 못한다면 스파이킹 뉴럴 네트워크의 큰 장점이라고 할 수 있는 에너지와 전력 효율성이 크게 저하될 수 있다. 이를 해결하기 위해 본 논문에서는 한계 전압 균형 방법론을 새로 제안한다. 제안하는 방법론은 베이시안 최적화 알고리즘을 사용하여 가장 최적의 한계전압 값을 찾는다. 또한 베이시안 최적화 알고리즘을 지연속도나 시냅틱 연산수 등의 스파이킹 뉴럴 네트워크의 특성을 고려할 수 있게 디자인한다. 더 나아가, 두 단계의 한계 전압을 제안하여 높은 에너지 효율을 가지며 더 빠르고 더 정확한 객체 인식 모델을 가능하게 한다. 실험 결과에 따르면 제안하는 방법들을 통해 state-of-the-art 객체 인식률을 달성하였고 기존의 방법들보다 PASCAL VOC에서는 2배, MS COCO에서는 1.85배 빠르게 수렴하는 것을 확인하였다. 또한 시냅틱 연산수도 PASCAL VOC에서는 40.33%, MS COCO에서는 45.31%를 줄일 수 있었다.One of the primary reasons behind the recent success of deep neural networks (DNNs) lies in the development of high-performance parallel computing systems and the availability of enormous amounts of data for training a complex model. Nonetheless, solving such advanced machine learning problems in real world applications requires a more sophisticated model with a vast number of parameters and training data, which leads to substantial amounts of computational overhead and power consumption. Given these circumstances, spiking neural networks (SNNs) have attracted growing interest as the third generation of neural networks due to their event-driven and low-powered nature. SNNs were introduced to mimic how information is encoded and processed in the human brain by employing spiking neurons as computation units. SNNs utilize temporal aspects in information transmission as in biological neural systems, thus providing sparse yet powerful computing ability. SNNs have been successfully applied in several applications, but these applications only include relatively simple tasks such as image classification, and are limited to shallow neural networks and datasets. One of the primary reasons for the limited application scope is the lack of scalable training algorithms attained from non-differential spiking neurons. In this dissertation, we investigate deep SNNs in a much more challenging regression problem (i.e., object detection), and propose a first object detection model in deep SNNs which is able to achieve comparable results to those of DNNs in non-trivial datasets. Furthermore, we introduce novel approaches to improve performance of the object detection model in terms of accuracy, latency and energy efficiency. This dissertation contains mainly two approaches: (a) object detection model in deep SNNs, and (b) improving performance of object detection model in deep SNNs. Consequently, the two approaches enable fast and accurate object detection in deep SNNs. The first approach is an object detection model in deep SNNs. We present a spiked-based object detection model, called Spiking-YOLO. To the best of our knowledge, Spiking-YOLO is the first spiked-based object detection model that is able to achieve comparable results to those of DNNs on a non-trivial dataset, namely PASCAL VOC and MS COCO. In doing so, we introduce two novel methods: a channel-wise weight normalization and a signed neuron with imbalanced threshold, both of which provide fast and accurate information transmission in deep SNNs. Our experiments show that Spiking-YOLO achieves remarkable results that are comparable (up to 98%) to those of Tiny YOLO (DNNs) on PASCAL VOC and MS COCO. Furthermore, Spiking-YOLO on a neuromorphic chip consumes approximately 280 times less energy than Tiny YOLO, and converges 2.3 to 4 times faster than previous DNN-to-SNN conversion methods. The second approach aims to provide a more effective form of computational capabilities in SNNs. Even though, SNNs enable sparse yet efficient information transmission through spike trains, leading to exceptional computational and energy efficiency, the critical challenges in SNNs to date are two-fold: (a) latency: the number of time steps required to achieve competitive results and (b) synaptic operations: the total number of spikes generated during inference. Without addressing these challenges properly, the potential impact of SNNs may be diminished in terms of energy and power efficiency. We present a threshold voltage balancing method for object detection in SNNs, which utilizes Bayesian optimization to find optimal threshold voltages in SNNs. We specifically design Bayesian optimization to consider important characteristics of SNNs, such as latency and number of synaptic operations. Furthermore, we introduce two-phase threshold voltages to provide faster and more accurate object detection, while providing high energy efficiency. According to experimental results, the proposed methods achieve the state-of-the-art object detection accuracy in SNNs, and converge 2x and 1.85x faster than conventional methods on PASCAL VOC and MS COCO, respectively. Moreover, the total number of synaptic operations is reduced by 40.33% and 45.31% on PASCAL VOC and MS COCO, respectively.Abstract i List of Figures ix List of Tables x 1 Introduction 1 2 Background 10 2.1 Object detection 10 2.2 Spiking Neural Networks 16 2.3 DNN-to-SNN conversion 18 2.4 Hyper-parameter optimization 21 3 Object detection model in deep SNNs 25 3.1 Introduction 25 3.2 Channel-wise weight normalization 27 3.2.1 Conventional weight normalization methods 27 3.2.2 Analysis of limitations in layer-wise weight normalization 29 3.2.3 Proposed weight normalization method 30 3.2.4 Analysis of the improved firing rate 38 3.3 Signed neuron with imbalanced threshold 39 3.3.1 Limitation of leaky-ReLU implementation in SNNs 39 3.3.2 The notion of imbalanced threshold 41 3.4 Evaluation 43 3.4.1 Spiking-YOLO detection results 43 3.4.2 Spiking-YOLO energy efficiency 57 4 Improving performance and efficiency of deep SNNs 60 4.1 Introduction 60 4.2 Threshold voltage balancing through Bayesian optimization 62 4.2.1 Motivation 62 4.2.2 Overall process and setup 67 4.2.3 Design of Bayesian optimization for SNNs 69 4.3 Fast and accurate object detection with two-phase threshold voltages 74 4.3.1 Motivation 74 4.3.2 Phase-1 threshold voltages: fast object detection 76 4.3.3 Phase-2 threshold voltages: accurate detection 76 4.4 Evaluation 79 4.4.1 Experimental setup 79 4.4.2 Experimental results 79 5 Conclusion 85 5.1 Dissertation summary 86 5.2 Discussion 88 5.2.1 Overview of the proposed methods and their usages 88 5.3 Challenges in SNNs 90 5.4 Future Work 92 5.4.1 Extension to various applications and DNN models 92 5.4.2 Further improve efficiency of SNNs 93 5.4.3 Optimization of deep SNNs 94 Bibliography 95 Abstract (In Korean) 110Docto