한정된 자원 환경을 위한 인공신경망 설계 및 가속

학위논문(박사) -- 서울대학교대학원 : 공과대학 전기·정보공학부, 2022. 8. 윤성로.Deep learning methods have become very successful in various applications due to the availability of exponentially growing data and powerful computing resources. Due to remarkable performance, neural model has been applied to various edge devices such as mobile and embedded system. These edge devices usually suffer from constrained resources including computation and energy. To overcome this challenge, low-latency model design, model compression and acceleration are widely researched on both hardware and software side. In this dissertation, we introduce two methods with regard to low-latency model design on algorithm side. Designing compact and low-latency model is important to reduce required resources. For this reason, in aspect of algorithm, we introduced two model design methodology with neural architecture search (NAS) to find compact model: cell-based NAS and graph variational auto-encoder based NAS. Our cell-based NAS approach is based on Differentiable ARchiTecture Search (DARTS) which is well-known differentiable NAS method. Despite the popularity of DARTS, it has been reported that DARTS often shows unrobustness and sub-optimality. Through extensive theoretical analysis and empirical observations, we reveal that this issue occurs as a result of the existence of unnormalized operations. Based on our finding, we propose a novel variancestationary differentiable architecture search (VS-DARTS). VS-DARTS makes the architecture parameters a more reliable metric for deriving a desirable architecture without increasing the search cost. In addition, we derive comparable architecture using VS-DARTS with soft constrained latency objectives. Another approach to find low-latency model is using graph generative models, which has been recently focused because of their efficiency. We proposed novel graph variational auto-encoder (VAE)which shows dramatically improvements on cell-based search space. After our graph VAE extracted architectural information from the neural architecture, we conducted novel multi-objective NAS using extracted information for hardware-constrained environments. We showed that the proposed multi-objective NAS can derive various models close to Pareto optimal between latency and accuracy. Also, we evaluated our proposed method on various hardware platforms. In summary, this dissertation proposes two methods for improving performance of NAS, which can use compact and low-latency neural model for computing resource-constrained environments. The proposed methods were evaluated with various experimental studies.딥러닝은 빅데이터와 강력한 병렬 프로세서가 사용 가능해지면서 다양한 분야에서 성공적인 모습을 보여주고 있으며, 모바일과 임베디드 시스템 같은 엣지 디바이스에도 다양하게 적용되고 있다. 그러나, 엣지 디바이스는 일반적으로 컴퓨팅 및 전력 자원이 부족한 환경이다. 이를 해결하기 위해서, 저-레이턴시 모델 디자인, 모델 압축 기법과 뉴럴 모델 가속 등이 알고리즘과 하드웨어 양쪽 측면에서 활발하게 연구되고 있다. 본 논문에서는 알고리즘 측면에서 저-레이턴시 모델 디자인을위한 두 가지 새로운 방법론을 제시한다. 필요한 컴퓨팅 자원을 줄이기 위해서는 먼저 컴팩트하고 레이턴시가 작은 모델을 디자인하는 것이 중요하므로, 알고리즘 측면에서 우리는 두가지 신경구조탐색을 이용한 모델 디자인 방법론을 제시하였다: 셀 기반 신경구조탐색과 그래프 배리에이셔널 오토인코더를 이용한 신경구조탐색이다. 본 학위논문의 셀 기반 신경구조탐색은 널리 알려진 differentiable NAS 방법론인 DARTS (Differentiable ARchiTecture Search)에 기반한다. DARTS는 여러 연구에서 베이스라인 방법론으로 널리 사용되고 있음에도 불구하고, 종종 학습 불안정성과 최적화가 부족하다는 점이 이미 보고된 바 있으나 그 근본적인 원인에대해서는 일부 밝혀지지 않았었다. 우리는 이론적 분석과 관찰을 통해서 그 근본적인 문제가 각 오퍼레이션의 정규화되지 않은 출력에 기인하는 것을 밝히고, 이 문제점을 해결할 수 있는 방법론인 VS-DARTS (variance stationary DARTS)를 제안하였다. VS-DARTS는 구조 변수(architectural parameter)의 신뢰성을 높여서 탐색 비용을 늘이지 않고 성능을 높였다. 또한, 우리는 VS-DARTS에 레이턴시에 대한연성 제약(soft constraint)을 적용함으로써 기존 셀 기반 방법론에 비견되는 성능의 구조를 탐색하였다. 또 다른 저 레이턴시 모델을 찾는 접근 방법으로는 그래프 생성모델(graph generative model)을 적용하였다. 우리는 새로운 그래프 배리에이셔널 오토인코더 (graph variational auto-encoder) 방법론을 제안하여 셀 기반 탐색 공간에 대한 오토인코더 성능을 크게 향상시켰다. 이후, 제안한 배리에이셔널 오토인코더를 이용하여 뉴럴 구조의 임베딩 정보를 추출하고, 추출된 구조 임베딩 정보를 새로이 제안한 다중목적탐색(multi-objective search)에 이용하여 레이턴시-정확도에서 파레토 최적(Pareto optimal)에 가까운 구조들을 찾을 수 있음을 보였다. 또한 제안한 신경구조탐색 방법론을 다양한 하드웨어 플랫폼 상에서 검증하였다. 요약하면, 본 학위논문에서는 보다 컴팩트하고 레이턴시가 작은 모델을 찾는데 사용할 수 있는 신경구조탐색 방법론을 두가지 제안하였고, 이를 이용하여 컴퓨팅 자원이 부족한 환경을 위한 인공 신경망의 자동 설계 방법에 관하여 기술하였으며, 다양한 실험을 통해서 검증하였다.1 Introduction 1 2 Background 7 2.1 Neural Architecture Search 7 2.1.1 Previous Works on Differentiable NAS 7 2.1.2 Preliminaries on DARTS 8 2.2 Graph Variational Auto-Encoder 10 2.2.1 Graph Representation Learning . 10 2.2.2 Variational Auto-encoder for Graph. 10 2.2.3 Neural Architecture Search (NAS) with Generative Models 11 2.2.4 Preliminaries on VAE for DAGs 12 3 Neural architecture search for resource-constrained environment 15 3.1 Introduction 15 3.2 Issue of DARTS Architecture Parameter 17 3.3 Lack of Reliability of β 18 3.4 Variance-stationary DARTS (VS-DARTS) 21 3.4.1 Node Normalization 21 3.4.2 Remedying Gradient Imbalance 22 3.4.3 β-Continuous Relaxation 23 3.5 Experimental Results 28 3.5.1 Settings 28 3.5.2 Results in DARTS Search Space 30 3.5.3 Results in RobustDARTS Search Space 33 3.5.4 Ablation Study 34 3.6 Summary 35 4 Platform-aware Neural Architecture Search with Graph Variational AutoEncoder 37 4.1 Introduction 37 4.2 Proposed Methods on Graph Variational Auto-Encoder 39 4.2.1 Fail Case Study 39 4.2.2 Proposed Update Function 41 4.2.3 Observer Node 43 4.3 Proposed Predictor-based Multi-objective NAS 43 4.3.1 Training Graph VAE (Step 1) 43 4.3.2 Search Process (Step 2) 46 4.3.3 Return the set of the searched architectures (line 21-22) 47 4.4 Experimental Results 47 4.4.1 Settings 47 4.4.2 VAE Performance Comparison 48 4.4.3 Pre-predictor 50 4.4.4 Search Performance Comparison 50 4.5 Discussions 51 4.5.1 Node Index Order in DAG VAE 51 4.5.2 Model size reduction while keep the reconstructive performance 51 4.5.3 Convergence Acceleration 52 4.6 Summary 52 5 Conclusion 57 Bibliography 58 Abstract (In Korean) 72박

Adapting Neural Architecture Search for Efficient Deep Learning Models

This thesis presents a comprehensive investigation into Neural Architecture Search (NAS), an instrumental strategy in the formulation of proficient deep learning models. The study scrutinizes two distinct paradigms of architecture search: channel number search and operation search. In the context of channel number search, we introduce the bilaterally coupled supernet, dubbed BCNet, alongside CafeNet, endowed with a flexible weight-sharing strategy. Regarding operation search, the research puts forward K-shot NAS, featuring a K-shot supernet and reparameterization strategies. Additionally, with the objective of eliciting optimal solutions from an expansive search space, we propose the integration of Monte-Carlo Tree Search, an approach conceived to augment search efficiency and performance. Moreover, this research devises a cyclical weight-sharing strategy explicitly for the Vision Transformer architecture, and further customizes the transformer supernet training strategy. By delving into a plethora of architectures and methodologies, this thesis aspires to lay a robust foundation for future research endeavors in this domain

Characterization of self-organization processes in complex networks

Programa Doutoral em Física (MAP-fis)A estrutura de interações sociais numa população é muitas vezes modelada através de uma rede complexa que representa os indivíduos e respetivas relações sociais. Estas estruturas são conhecidas por afetarem de forma fundamental os processos dinâmicos que suportam. A caracterização desse efeito é, no entanto, uma tarefa complicada pois o tratamento matemático destes sistemas requer o estudo de um espaço de estados de grande dimensão, limitando a aplicabilidade de abordagens analíticas e numéricas. Esta tese teve como objetivo desenvolver métodos, inspirados na Física Estatística dos Sistemas Fora do Equilíbrio, com o fim de caracterizar processos dinâmicos em redes complexas. Nesta tese demonstramos que a estrutura de uma população naturalmente induz a emergência de padrões de correlações entre indivíduos que partilham traços semelhantes, um fenómeno também identificado em estudos empíricos. Estes padrões de correlações são independentes do tipo de processo dinâmico considerado, do tipo de informação que se propaga sendo observados numa classe alargada de redes complexas. Mostramos também que propriedades como o clustering e a densidade de ligações da rede têm um papel fundamental nos padrões de correlações emergentes. Outra questão fundamental diz respeito à relação entre as dinâmicas local e a global em redes sociais. De facto, as redes sociais afetam de forma tão fundamental os processos dinâmicos que suportam que em muitas situações o comportamento coletivo observado não tem qualquer relação aparente com a dinâmica local na sua génese. Este é um problema comum a muitos sistemas complexos e tipicamente associado a fenómenos emergentes e de auto-organização. Neste trabalho esta questão é explorada no contexto do problema da Cooperação e no âmbito da Teoria de Jogos Evolutiva. Para esse fim introduzimos uma quantidade que é estimada numericamente e a que damos o nome de Average Gradient of Selection (AGOS). Esta quantidade, relaciona de forma efetiva as dinâmicas local e global, possibilitando a descrição do processo de auto-organização em populações estruturadas. Através do AGOS mostramos que quando as interações entre indivíduos são descritas através do Dilema do Prisioneiro, uma metáfora popular no estudo da cooperação, a dinâmica coletiva emergente é sensível à forma da rede de interações entre os indivíduos. Em particular, demonstramos que quando a rede é homogénea (heterogénea) no que respeita à distribuição de grau o Dilema do Prisioneiro é transformado numa dinâmica coletiva de coexistência (coordenação). Mostramos ainda que esta transformação depende da pressão de seleção (associada ao grau de determinismo no processo de decisão dos indivíduos) e de taxa de mutações (a adoção espontânea de um novo comportamento por parte de um individuo) consideradas. A relação entre estas duas varáveis pode também resultar em alterações de regimes dinâmicos cujo o resultado pode, em casos particulares, resultar no desfecho drástico para a evolução da cooperação. Finalmente, fazemos uso do AGOS para caracterizar a dinâmica evolutiva da cooperação no caso em que a estrutura co-evolve. Demonstramos que na presença de uma estrutura social a dinâmica global é semelhante à de um jogo de coordenação entre N-pessoas, cujas características dependem de forma sensível das escalas de tempo relativas entre a evolução de comportamentos e a evolução da estrutura. Uma vez mais, a dinâmica global emergente contrasta com o Dilema do Prisioneiro que caracteriza as interações locais entre os indivíduos. Acreditamos que o AGOS, que pode ser facilmente aplicado no estudo de outros processos dinâmicos, proporciona uma contribuição significativa para o melhor entendimento de Sistemas Complexos, em particular aqueles em que as interações entre os elementos constituintes são bem definidos através uma rede complexa.The structure of social interactions in a population is often modeled by means of a complex network representing individuals and their social ties. These structures are known to fundamentally impact the processes they support. However, the characterization of how structure impacts a dynamical process is by no means an easy task. Indeed, the large configuration space spanned tends to limit the systematic applicability of numerical methods, while analytical treatments have failed to provide a good description of the system dynamics. The aim of this thesis was to develop methods inspired in the Statistical Physics of Systems far from equilibrium to characterize dynamical processes on complex networks. In this thesis we show how the structure of a population naturally induces the emergence of correlations between individuals that share similar traits, which is in accordance empirical evidence. These, so called, ’peer-influence” correlation patterns are independent of the type of dynamical process under consideration, the type of information being spread while being ubiquitous among a wide variety of network topologies. We have also find evidence that central to the ’peer-influence” patterns are topological features such as the clustering and the sparsity of the underlying network of interactions. Another fundamental problem concerns the relationship between local and global dynamics in social networks. Indeed, social networks affect in such a fundamental way the dynamics of the population they support that the collective, population-wide behavior that one observes often bears no relation to the individual processes it stems from. This is in fact a common problem among many Complex Systems typically associated with self-organization and emerging phenomena. Here we study this issue in the context of the problem of Cooperation and in the realm of Evolutionary Game Theory. To that end we introduce a numerically estimated mean-field quantity that we call the Average Gradient of Selection (AGOS). This quantity is able to effectively connect the local and global dynamics, providing a way to track the self-organization of cooperators and defectors in networked populations. With the AGOS we show that when individuals engage in a Prisoner’s Dilemma, a popular cooperation metaphor, the emerging collective dynamics depends on the shape of the underlying network of interactions. In particular, we show that degree homogeneous (heterogeneous) networks the Prisoner’s Dilemma is transformed into a collective coexistence (coordination) dynamics, contrasting with the defector dominance of the local dynamics. We further show that the extent to which these emergent phenomena are observed in structured populations is conditional on the selection pressure (the uncertainty associated with the decision making) and the rate of mutations (the spontaneously adoption of new behaviors by individuals) under consideration. Interestingly, the interplay between selection pressure and mutation rates can lead to drastic regime shifts in the evolution of cooperation. Finally, we make use of the AGOS to characterize the evolutionary dynamics of cooperation in the case of a co-evolving social structure. We demonstrate that in an adaptive social structure the population-wide dynamics resembles that of a N-person coordination game, whose characteristics depend sensitively on the relative time-scales between behavioral and network co-evolution. Once more, the resulting collective dynamics contrasts with the two-person Prisoner’s Dilemma that characterizes how individuals interact locally. We argue that the AGOS, which can be readily applied to other dynamical contexts and processes, provides a significant contribution to a better understanding of Complex Systems involving populations in which who interacts with whom is well-defined by a complex network

The Habitable Exoplanet Observatory (HabEx) Mission Concept Study Final Report

The Habitable Exoplanet Observatory, or HabEx, has been designed to be the Great Observatory of the 2030s. For the first time in human history, technologies have matured sufficiently to enable an affordable space-based telescope mission capable of discovering and characterizing Earthlike planets orbiting nearby bright sunlike stars in order to search for signs of habitability and biosignatures. Such a mission can also be equipped with instrumentation that will enable broad and exciting general astrophysics and planetary science not possible from current or planned facilities. HabEx is a space telescope with unique imaging and multi-object spectroscopic capabilities at wavelengths ranging from ultraviolet (UV) to near-IR. These capabilities allow for a broad suite of compelling science that cuts across the entire NASA astrophysics portfolio. HabEx has three primary science goals: (1) Seek out nearby worlds and explore their habitability; (2) Map out nearby planetary systems and understand the diversity of the worlds they contain; (3) Enable new explorations of astrophysical systems from our own solar system to external galaxies by extending our reach in the UV through near-IR. This Great Observatory science will be selected through a competed GO program, and will account for about 50% of the HabEx primary mission. The preferred HabEx architecture is a 4m, monolithic, off-axis telescope that is diffraction-limited at 0.4 microns and is in an L2 orbit. HabEx employs two starlight suppression systems: a coronagraph and a starshade, each with their own dedicated instrument

The Habitable Exoplanet Observatory (HabEx) Mission Concept Study Final Report

The Habitable Exoplanet Observatory, or HabEx, has been designed to be the Great Observatory of the 2030s. For the first time in human history, technologies have matured sufficiently to enable an affordable space-based telescope mission capable of discovering and characterizing Earthlike planets orbiting nearby bright sunlike stars in order to search for signs of habitability and biosignatures. Such a mission can also be equipped with instrumentation that will enable broad and exciting general astrophysics and planetary science not possible from current or planned facilities. HabEx is a space telescope with unique imaging and multi-object spectroscopic capabilities at wavelengths ranging from ultraviolet (UV) to near-IR. These capabilities allow for a broad suite of compelling science that cuts across the entire NASA astrophysics portfolio. HabEx has three primary science goals: (1) Seek out nearby worlds and explore their habitability; (2) Map out nearby planetary systems and understand the diversity of the worlds they contain; (3) Enable new explorations of astrophysical systems from our own solar system to external galaxies by extending our reach in the UV through near-IR. This Great Observatory science will be selected through a competed GO program, and will account for about 50% of the HabEx primary mission. The preferred HabEx architecture is a 4m, monolithic, off-axis telescope that is diffraction-limited at 0.4 microns and is in an L2 orbit. HabEx employs two starlight suppression systems: a coronagraph and a starshade, each with their own dedicated instrument.Comment: Full report: 498 pages. Executive Summary: 14 pages. More information about HabEx can be found here: https://www.jpl.nasa.gov/habex