High Dimensional Markov Chain Monte Carlo with Multiple GPUs

학위논문 (석사)-- 서울대학교 대학원 : 자연과학대학 통계학과, 2018. 2. 이재용.인공신경망과 같은 많은 계산을 요하는 모형이 다양한 분야에서 효과적임이 드러남에 따라 형렬 연산을 병렬처리 하기 위해 그래픽카드(GPU) 상 에서 계산하는 것이 일반화되고 있으며 이를 위해 계산을 여러 스레드로 나누는 방법을 찾는 것이 중요해지고 있다. 본 논문은 분할 된 샘플 공간에서 브리지 샘플링과 해밀토니안 몬테카를로를 결합하여 여러 GPU에 분산될 수 있는 새로운 MCMC 알고리즘을 제시한다. 이 접근법은 베이지안 뉴럴 네트워크 (Bayesian Neural Network)와 같은 타겟 분포에 대한 MCMC 샘플링을 빠르게 할 수 있다. 또한 다중 모달 (Multimodality)이 존재할 때 이 알고리즘은 낮은 확률 영역에서도 샘플링을 효율적으로 잘 할수 있는것으로 나타났다. 마지막으로 이 논문은 Adam Optimizer, 해밀토니안 몬테카를로와 같은 다른 학습 방법의 변수 분포와 본 알고리즘의 변수분포를 비교함으로써, 제한된 표본 공간이 일반화 오차에 미치는 영향에 대한 추가 연구가 수행 될 수 있음을 제시한다.Allocating computation over multiple threads to reduce running time has become a key to training big models such as deep neural networks because a Graphics Processing Unit (GPU), which is parallel in nature, can speed up intensive matrix operations. We present a new MCMC algorithm that can be distributed over multiple GPUs by combining bridge sampling with Hamiltonian Monte Carlo on partitioned sample spaces. We empirically show that this approach can expedite MCMC sampling for any unnormalized target distribution such as Bayesian Neural Network in a high dimensional setting. Furthermore, in the presence of multimodality, this algorithm is expected to be more efficient in mixing MCMC chains when proper partitions are chosen. Finally, by comparing the parameter distributions of different learning method, we suggest that further studies could be conducted on the effect of a constrained sample space on the generalization error.Abstract 3 Chapter 1 Introduction 9 1.1 Related Works 10 1.2 Contribution 12 Chapter 2 Hamiltonian Monte Carlo 13 2.1 Momentum proposal 15 2.2 Leap frog update 15 2.3 Metropolis Accept-Reject 16 Chapter 3 Bridged Hamiltonian Monte Carlo 18 3.1 Sampling from Partitioned Sample Space 18 3.1.1 Constrained HMC 19 3.2 Combining Samples from different Sample Space 20 3.2.1 Bridge Sampling 22 3.3 Practical Issues in Implementing Bridged Hamiltonian Monte Carlo 24 3.3.1 Numerical Overflow or Underflow 24 3.3.2 Partitioning Scheme 26 Chapter 4 Experiments 28 4.1 Bivariate Normal Mixture Model 28 4.2 Moon Data Classification 31 4.3 MNIST Data Classification 36 4.4 Result 37 Chapter 5 Discussion 42 초록 48Maste

On the distribution of throughput of transfer lines

Ankara : Department of Industrial Engineering and the Institute of Engineering and Sciences of Bilkent University, 1998.Thesis (Master's) -- Bilkent University, 1998.Includes bibliographical references leaves 86-107A transfer line corresponds to a manufacturing system consisting of a number of work stations in series integrated into one system by a common transfer mechanism and a control system. There is a vast literature on the transfer lines. However, little has been done on the transient analysis of these systems by making use of the higher order moments of their performance measures due to the difficulty in determining the evolution of the stochastic processes under consideration. This thesis examines the transient behavior of relatively short transfer lines and derives the distribution of the performance measures of interest. The proposed method based on the analytical derivation of the distribution of throughput is also applied to the systems with two-part types. An experiment is designed in order to compare the results of this study with the state-space representations and the simulation. They are also interpreted from the point of view of the line behavior and design issue. Furthermore, extensions are briefly discussed and directions for future research are suggested.Deler, BaharM.S