Predictive Coding for Dynamic Visual Processing: Development of Functional Hierarchy in a Multiple Spatio-Temporal Scales RNN Model

The current paper proposes a novel predictive coding type neural network model, the predictive multiple spatio-temporal scales recurrent neural network (P-MSTRNN). The P-MSTRNN learns to predict visually perceived human whole-body cyclic movement patterns by exploiting multiscale spatio-temporal constraints imposed on network dynamics by using differently sized receptive fields as well as different time constant values for each layer. After learning, the network becomes able to proactively imitate target movement patterns by inferring or recognizing corresponding intentions by means of the regression of prediction error. Results show that the network can develop a functional hierarchy by developing a different type of dynamic structure at each layer. The paper examines how model performance during pattern generation as well as predictive imitation varies depending on the stage of learning. The number of limit cycle attractors corresponding to target movement patterns increases as learning proceeds. And, transient dynamics developing early in the learning process successfully perform pattern generation and predictive imitation tasks. The paper concludes that exploitation of transient dynamics facilitates successful task performance during early learning periods.Comment: Accepted in Neural Computation (MIT press

Error resilience and concealment techniques for high-efficiency video coding

This thesis investigates the problem of robust coding and error concealment in High Efficiency Video Coding (HEVC). After a review of the current state of the art, a simulation study about error robustness, revealed that the HEVC has weak protection against network losses with significant impact on video quality degradation. Based on this evidence, the first contribution of this work is a new method to reduce the temporal dependencies between motion vectors, by improving the decoded video quality without compromising the compression efficiency. The second contribution of this thesis is a two-stage approach for reducing the mismatch of temporal predictions in case of video streams received with errors or lost data. At the encoding stage, the reference pictures are dynamically distributed based on a constrained Lagrangian rate-distortion optimization to reduce the number of predictions from a single reference. At the streaming stage, a prioritization algorithm, based on spatial dependencies, selects a reduced set of motion vectors to be transmitted, as side information, to reduce mismatched motion predictions at the decoder. The problem of error concealment-aware video coding is also investigated to enhance the overall error robustness. A new approach based on scalable coding and optimally error concealment selection is proposed, where the optimal error concealment modes are found by simulating transmission losses, followed by a saliency-weighted optimisation. Moreover, recovery residual information is encoded using a rate-controlled enhancement layer. Both are transmitted to the decoder to be used in case of data loss. Finally, an adaptive error resilience scheme is proposed to dynamically predict the video stream that achieves the highest decoded quality for a particular loss case. A neural network selects among the various video streams, encoded with different levels of compression efficiency and error protection, based on information from the video signal, the coded stream and the transmission network. Overall, the new robust video coding methods investigated in this thesis yield consistent quality gains in comparison with other existing methods and also the ones implemented in the HEVC reference software. Furthermore, the trade-off between coding efficiency and error robustness is also better in the proposed methods

A New H.264/AVC Error Resilience Model Based on Regions of Interest

International audienceVideo transmission over the Internet can sometimes be subject to packet loss which reduces the end-user's Quality of Experience (QoE). Solutions aiming at improving the robustness of a video bitstream can be used to subdue this problem. In this paper, we propose a new Region of Interest-based error resilience model to protect the most important part of the picture from distortions. We conduct eye tracking tests in order to collect the Region of Interest (RoI) data. Then, we apply in the encoder an intra-prediction restriction algorithm to the macroblocks belonging to the RoI. Results show that while no significant overhead is noted, the perceived quality of the video's RoI, measured by means of a perceptual video quality metric, increases in the presence of packet loss compared to the traditional encoding approach

Understanding user experience of mobile video: Framework, measurement, and optimization

Since users have become the focus of product/service design in last decade, the term User eXperience (UX) has been frequently used in the field of Human-Computer-Interaction (HCI). Research on UX facilitates a better understanding of the various aspects of the user’s interaction with the product or service. Mobile video, as a new and promising service and research field, has attracted great attention. Due to the significance of UX in the success of mobile video (Jordan, 2002), many researchers have centered on this area, examining users’ expectations, motivations, requirements, and usage context. As a result, many influencing factors have been explored (Buchinger, Kriglstein, Brandt & Hlavacs, 2011; Buchinger, Kriglstein & Hlavacs, 2009). However, a general framework for specific mobile video service is lacking for structuring such a great number of factors. To measure user experience of multimedia services such as mobile video, quality of experience (QoE) has recently become a prominent concept. In contrast to the traditionally used concept quality of service (QoS), QoE not only involves objectively measuring the delivered service but also takes into account user’s needs and desires when using the service, emphasizing the user’s overall acceptability on the service. Many QoE metrics are able to estimate the user perceived quality or acceptability of mobile video, but may be not enough accurate for the overall UX prediction due to the complexity of UX. Only a few frameworks of QoE have addressed more aspects of UX for mobile multimedia applications but need be transformed into practical measures. The challenge of optimizing UX remains adaptations to the resource constrains (e.g., network conditions, mobile device capabilities, and heterogeneous usage contexts) as well as meeting complicated user requirements (e.g., usage purposes and personal preferences). In this chapter, we investigate the existing important UX frameworks, compare their similarities and discuss some important features that fit in the mobile video service. Based on the previous research, we propose a simple UX framework for mobile video application by mapping a variety of influencing factors of UX upon a typical mobile video delivery system. Each component and its factors are explored with comprehensive literature reviews. The proposed framework may benefit in user-centred design of mobile video through taking a complete consideration of UX influences and in improvement of mobile videoservice quality by adjusting the values of certain factors to produce a positive user experience. It may also facilitate relative research in the way of locating important issues to study, clarifying research scopes, and setting up proper study procedures. We then review a great deal of research on UX measurement, including QoE metrics and QoE frameworks of mobile multimedia. Finally, we discuss how to achieve an optimal quality of user experience by focusing on the issues of various aspects of UX of mobile video. In the conclusion, we suggest some open issues for future study

Recognizing recurrent neural networks (rRNN): Bayesian inference for recurrent neural networks

Recurrent neural networks (RNNs) are widely used in computational neuroscience and machine learning applications. In an RNN, each neuron computes its output as a nonlinear function of its integrated input. While the importance of RNNs, especially as models of brain processing, is undisputed, it is also widely acknowledged that the computations in standard RNN models may be an over-simplification of what real neuronal networks compute. Here, we suggest that the RNN approach may be made both neurobiologically more plausible and computationally more powerful by its fusion with Bayesian inference techniques for nonlinear dynamical systems. In this scheme, we use an RNN as a generative model of dynamic input caused by the environment, e.g. of speech or kinematics. Given this generative RNN model, we derive Bayesian update equations that can decode its output. Critically, these updates define a 'recognizing RNN' (rRNN), in which neurons compute and exchange prediction and prediction error messages. The rRNN has several desirable features that a conventional RNN does not have, for example, fast decoding of dynamic stimuli and robustness to initial conditions and noise. Furthermore, it implements a predictive coding scheme for dynamic inputs. We suggest that the Bayesian inversion of recurrent neural networks may be useful both as a model of brain function and as a machine learning tool. We illustrate the use of the rRNN by an application to the online decoding (i.e. recognition) of human kinematics

Dynamic bandwidth allocation in ATM networks

Includes bibliographical references.This thesis investigates bandwidth allocation methodologies to transport new emerging bursty traffic types in ATM networks. However, existing ATM traffic management solutions are not readily able to handle the inevitable problem of congestion as result of the bursty traffic from the new emerging services. This research basically addresses bandwidth allocation issues for bursty traffic by proposing and exploring the concept of dynamic bandwidth allocation and comparing it to the traditional static bandwidth allocation schemes

딥 스파이킹 뉴럴 네트워크의 빠르고 정확한 정보 전달

학위논문 (박사) -- 서울대학교 대학원 : 공과대학 전기·컴퓨터공학부, 2021. 2. 윤성로.오늘 날 딥러닝의 큰 성공은 고성능 병렬 컴퓨팅 시스템의 발전과 복잡한 모델을 학습하기 위해 필요한 많은 양의 데이터가 수집되어 접근이 가능해진 점이라고 할 수 있다. 하지만 실제 세상에 존재하는 더 어려운 문제들을 풀고자할 때는 더욱 더 섬세하고 복잡한 모델과 이 모델을 성공적으로 학습할 수 있는 방대한 양의 데이터를 필요한다. 하지만 이러한 점들은 모델 수행 시 연산 오버헤드와 전력 소모를 급격하게 증가시킬 수 밖에 없다. 이러한 문제점들을 극복하는 여러 방법들 중 하나로 스파이킹 뉴럴 네트워크가 최근 많은 주목을 받고 있다. 스파이킹 뉴럴 네트워크는 제 3세대 인공 신경망으로 불리며 이벤트 중심의 동작을 기반으로 하여 저전력이 가장 큰 장점이다. 스파이킹 뉴럴 네트워크는 실제 인간의 뇌에서 뉴런들 간 정보를 전달하는 방식을 모방하며 스파이킹 뉴런을 연산 단위로 사용하고 있다. 스파이킹 뉴럴 네트워크는 생물학적 신경계와 동일하게 시간적 정보를 활용할 수 있기 때문에 매우 뛰어난 연산 능력을 가지고 있다. 하지만 스파이킹 뉴럴 네트워크는 이미지 분류와 같은 비교적 쉬운 응용에만 주로 사용되고 있으며 얕은 인공 신경망과 간단한 데이터셋에서만 주로 수행되고 있다. 이러한 제약이 존재하는 가장 큰 요인 중 하나는 스파이크 뉴럴 네트워크에 적합한 학습 알고리즘이 아직 존재하지 않기 때문이다. 스파이크로 정보를 전달하고 연산을 수행하기 때문에 미분이 불가능하다. 따라서 딥 뉴럴 네트워크에서 주로 사용되는 역전파 알고리즘의 사용이 불가능하다. 본 논문에서 딥 스파이킹 뉴럴 네트워크를 이미지 분류보다 더 어려운 회귀 문제 (객체 인식)에 적용해 보고, 딥 뉴럴 네트워크의 성능에 버금가는 객체 인식 모델을 스파이킹 뉴럴 네트워에서 처음으로 제안한다. 더 나아가, 객체 인식 모델의 성능과 지연시간, 에너지 효율성을 향상 시킬 수 있는 여러 방법들을 제안한다. 본 논문은 크게 두 가지 주제로 나누어 설명한다: (a) 딥 스파이킹 뉴럴 네트워크에서의 객체 인식 모델, (b) 딥 스파이킹 뉴럴 네트워크에서의 객체 인식 모델의 성능 및 효율성 향상. 제안하는 방법들을 통해 빠르고 정확한 객체 인식 모델을 딥 스파이킹 뉴럴 네트워크에서 수행할 수 있다. 첫 번째 방법은 딥 스파이킹 뉴럴 네트워크에서의 객체 인식 모델이다. 객체 인식 모델은 Spiking-YOLO로 부르고, 저자들이 아는 바에 의하면 PASCAL VOC, MS COCO와 같은 데이터 셋에서 딥 뉴럴 네트워크의 성능에 버금가는 결과를 보여준 첫 번째 스파이킹 뉴럴 네트워크를 기반으로 하는 객체 인식 모델이다. Spiking-YOLO에서는 크게 두 가지 방법을 제안한다. 첫번 째는 채널 별 가중치 정규화이고 두번째는 불균형 한계 전압을 가지는 양음수 뉴런이다. 두 가지 방법을 통해 빠르고 정확한 정보를 딥 스파이킹 뉴럴 네트워크에서 전달 가능하게 한다. 실험 결과, Spiking-YOLO는 PASCAL VOC와 MS COCO 데이터셋에서 딥 뉴럴 네트워크의 객체 인식률의 98%에 뛰어난 성능을 보였다. 또한 Spiking-YOLO가 뉴로모픽 칩에 구현되었음 가정하였을 때, Tiny YOLO보다 약 280의 에너지를 적게 소모하였고 기존의 DNN-to-SNN 전환 방법들 보다 2.3배에서 4배 더 빠르게 수렴하는 것을 확인할 수 있었다. 두 번째 방법은 스파이킹 뉴럴 네트워크에 조금 더 효율적인 연산 능력을 부여하는데 중점을 주고 있다. 비록 스파이킹 뉴럴 네트워크가 희박한 양의 스파이크로 정보를 효율적으로 전달하며 연산 오버헤드와 에너지 소모가 적지만, 두 가지 매우 중요한 문제들이 존재한다: (a) 지연속도: 좋은 성능을 내기 위해 필요한 타임스탭, (b) 시냅틱 연산수: 추론 시 생성된 총 스파이크의 수. 이러한 문제들을 적절히 해결하지 못한다면 스파이킹 뉴럴 네트워크의 큰 장점이라고 할 수 있는 에너지와 전력 효율성이 크게 저하될 수 있다. 이를 해결하기 위해 본 논문에서는 한계 전압 균형 방법론을 새로 제안한다. 제안하는 방법론은 베이시안 최적화 알고리즘을 사용하여 가장 최적의 한계전압 값을 찾는다. 또한 베이시안 최적화 알고리즘을 지연속도나 시냅틱 연산수 등의 스파이킹 뉴럴 네트워크의 특성을 고려할 수 있게 디자인한다. 더 나아가, 두 단계의 한계 전압을 제안하여 높은 에너지 효율을 가지며 더 빠르고 더 정확한 객체 인식 모델을 가능하게 한다. 실험 결과에 따르면 제안하는 방법들을 통해 state-of-the-art 객체 인식률을 달성하였고 기존의 방법들보다 PASCAL VOC에서는 2배, MS COCO에서는 1.85배 빠르게 수렴하는 것을 확인하였다. 또한 시냅틱 연산수도 PASCAL VOC에서는 40.33%, MS COCO에서는 45.31%를 줄일 수 있었다.One of the primary reasons behind the recent success of deep neural networks (DNNs) lies in the development of high-performance parallel computing systems and the availability of enormous amounts of data for training a complex model. Nonetheless, solving such advanced machine learning problems in real world applications requires a more sophisticated model with a vast number of parameters and training data, which leads to substantial amounts of computational overhead and power consumption. Given these circumstances, spiking neural networks (SNNs) have attracted growing interest as the third generation of neural networks due to their event-driven and low-powered nature. SNNs were introduced to mimic how information is encoded and processed in the human brain by employing spiking neurons as computation units. SNNs utilize temporal aspects in information transmission as in biological neural systems, thus providing sparse yet powerful computing ability. SNNs have been successfully applied in several applications, but these applications only include relatively simple tasks such as image classification, and are limited to shallow neural networks and datasets. One of the primary reasons for the limited application scope is the lack of scalable training algorithms attained from non-differential spiking neurons. In this dissertation, we investigate deep SNNs in a much more challenging regression problem (i.e., object detection), and propose a first object detection model in deep SNNs which is able to achieve comparable results to those of DNNs in non-trivial datasets. Furthermore, we introduce novel approaches to improve performance of the object detection model in terms of accuracy, latency and energy efficiency. This dissertation contains mainly two approaches: (a) object detection model in deep SNNs, and (b) improving performance of object detection model in deep SNNs. Consequently, the two approaches enable fast and accurate object detection in deep SNNs. The first approach is an object detection model in deep SNNs. We present a spiked-based object detection model, called Spiking-YOLO. To the best of our knowledge, Spiking-YOLO is the first spiked-based object detection model that is able to achieve comparable results to those of DNNs on a non-trivial dataset, namely PASCAL VOC and MS COCO. In doing so, we introduce two novel methods: a channel-wise weight normalization and a signed neuron with imbalanced threshold, both of which provide fast and accurate information transmission in deep SNNs. Our experiments show that Spiking-YOLO achieves remarkable results that are comparable (up to 98%) to those of Tiny YOLO (DNNs) on PASCAL VOC and MS COCO. Furthermore, Spiking-YOLO on a neuromorphic chip consumes approximately 280 times less energy than Tiny YOLO, and converges 2.3 to 4 times faster than previous DNN-to-SNN conversion methods. The second approach aims to provide a more effective form of computational capabilities in SNNs. Even though, SNNs enable sparse yet efficient information transmission through spike trains, leading to exceptional computational and energy efficiency, the critical challenges in SNNs to date are two-fold: (a) latency: the number of time steps required to achieve competitive results and (b) synaptic operations: the total number of spikes generated during inference. Without addressing these challenges properly, the potential impact of SNNs may be diminished in terms of energy and power efficiency. We present a threshold voltage balancing method for object detection in SNNs, which utilizes Bayesian optimization to find optimal threshold voltages in SNNs. We specifically design Bayesian optimization to consider important characteristics of SNNs, such as latency and number of synaptic operations. Furthermore, we introduce two-phase threshold voltages to provide faster and more accurate object detection, while providing high energy efficiency. According to experimental results, the proposed methods achieve the state-of-the-art object detection accuracy in SNNs, and converge 2x and 1.85x faster than conventional methods on PASCAL VOC and MS COCO, respectively. Moreover, the total number of synaptic operations is reduced by 40.33% and 45.31% on PASCAL VOC and MS COCO, respectively.Abstract i List of Figures ix List of Tables x 1 Introduction 1 2 Background 10 2.1 Object detection 10 2.2 Spiking Neural Networks 16 2.3 DNN-to-SNN conversion 18 2.4 Hyper-parameter optimization 21 3 Object detection model in deep SNNs 25 3.1 Introduction 25 3.2 Channel-wise weight normalization 27 3.2.1 Conventional weight normalization methods 27 3.2.2 Analysis of limitations in layer-wise weight normalization 29 3.2.3 Proposed weight normalization method 30 3.2.4 Analysis of the improved firing rate 38 3.3 Signed neuron with imbalanced threshold 39 3.3.1 Limitation of leaky-ReLU implementation in SNNs 39 3.3.2 The notion of imbalanced threshold 41 3.4 Evaluation 43 3.4.1 Spiking-YOLO detection results 43 3.4.2 Spiking-YOLO energy efficiency 57 4 Improving performance and efficiency of deep SNNs 60 4.1 Introduction 60 4.2 Threshold voltage balancing through Bayesian optimization 62 4.2.1 Motivation 62 4.2.2 Overall process and setup 67 4.2.3 Design of Bayesian optimization for SNNs 69 4.3 Fast and accurate object detection with two-phase threshold voltages 74 4.3.1 Motivation 74 4.3.2 Phase-1 threshold voltages: fast object detection 76 4.3.3 Phase-2 threshold voltages: accurate detection 76 4.4 Evaluation 79 4.4.1 Experimental setup 79 4.4.2 Experimental results 79 5 Conclusion 85 5.1 Dissertation summary 86 5.2 Discussion 88 5.2.1 Overview of the proposed methods and their usages 88 5.3 Challenges in SNNs 90 5.4 Future Work 92 5.4.1 Extension to various applications and DNN models 92 5.4.2 Further improve efficiency of SNNs 93 5.4.3 Optimization of deep SNNs 94 Bibliography 95 Abstract (In Korean) 110Docto

Application and Theory of Multimedia Signal Processing Using Machine Learning or Advanced Methods

This Special Issue is a book composed by collecting documents published through peer review on the research of various advanced technologies related to applications and theories of signal processing for multimedia systems using ML or advanced methods. Multimedia signals include image, video, audio, character recognition and optimization of communication channels for networks. The specific contents included in this book are data hiding, encryption, object detection, image classification, and character recognition. Academics and colleagues who are interested in these topics will find it interesting to read