B-스플라인 과완비 체계를 이용한 비모수 베이즈 회귀 모형 연구

학위논문(박사) -- 서울대학교대학원 : 자연과학대학 통계학과, 2021.8. 이재용.본 학위 논문에서는 함수의 변화하는 부드러움을 추정하기 위해 LARK 모형을 확장한 “레비 적응 B-스플라인 회귀 모형” (LABS) 을 제안한다. 즉, 제안한 모형은 B-스플라인 기저들이 생성 커널로 갖는 LARK 모형이다. 제안한 모형은 B-스플라인 기저의 차수를 조정하면서 불연속하거나 최고점 등을 지닌 함수의 부드러움에 체계적으로 적응한다. 모의 실험들과 실제 자료 분석을 통해서 제안한 모형이 불연속점, 최고점, 곡선 부분을 모두 잘 추정하고 있음을 입증하고, 거의 모든 실험에서 최고의 성능을 발휘한다. 또한, B-스플라인 차수에 따라 LABS 모형의 평균 함수가 특정 베소프 공간에 존재하고, LABS 모형의 사전분포가 해당 베소프 공간에 상당히 넓은 받침을 갖는다는 것을 밝힌다. 추가적으로, 텐서곱 B-스플라인 기저를 도입하여 다차원 자료를 분석할 수 있는 LABS 모형을 개발한다. 제안한 모형을 “다차원 레비 적응 B-스플라인 회귀 모형” (MLABS) 이라고 명명한다. MLABS 모형은 회귀 및 분류 문제들에서 최신 모형들과 필적할만한 성능을 갖추고 있다. 특히, MLABS 모형이 저차원 회귀 문제들에서 최신 비모수 회귀 모형들보다 안정적이고 정확한 예측 능력을 지니고 있음을 실험들을 통해 보인다.In this dissertation, we propose the Lévy Adaptive B-Spline regression (LABS) model, an extension of the LARK models, to estimate functions with varying degrees of smoothness. LABS model is a LARK with B-spline bases as generating kernels. By changing the degrees of the B-spline basis, LABS can systematically adapt the smoothness of functions, i.e., jump discontinuities, sharp peaks, etc. Results of simulation studies and real data examples support that this model catches not only smooth areas but also jumps and sharp peaks of functions. The LABS model has the best performance in almost all examples. We also provide theoretical results that the mean function for the LABS model belongs to the specific Besov spaces based on the degrees of the B-spline basis and that the prior of the model has the full support on the Besov spaces. Furthermore, we develop a multivariate version of the LABS model by introducing tensor product of B-spline bases named Multivariate Lévy Adaptive B-Spline regression (MLABS). MLABS model has comparable performance on both regression and classification problems. Especially, empirical results demonstrate that MLABS has more stable and accurate predictive abilities than state-of-the-art nonparametric regression models in relatively low-dimensional data.1 Introduction 1 1.1 Nonparametric regression model 1 1.2 Literature Review 2 1.2.1 Literature review of nonparametric function estimation 2 1.2.2 Literature review of multivariate nonparametric regression 5 1.3 Outline 7 2 Bayesian nonparametric function estimation using overcomplete systems with B-spline bases 9 2.1 Introduction 9 2.2 Lévy adaptive regression kernels 11 2.3 Lévy adaptive B-spline regression 14 2.3.1 B-spline basis 15 2.3.2 Model specification 17 2.3.3 Support of LABS model 19 2.4 Algorithm 22 2.5 Simulation studies 25 2.5.1 Simulation 1 : DJ test functions 27 2.5.2 Simulation 2 : Smooth functions with jumps and peaks 30 2.6 Real data applications 35 2.6.1 Example 1: Minimum legal drinking age 35 2.6.2 Example 2: Bitcoin prices on Bitstamp 37 2.6.3 Example 3: Fine particulate matter in Seoul 39 2.7 Discussion 42 3 Bayesian multivariate nonparametric regression using overcomplete systems with tensor products of B-spline bases 43 3.1 Introduction 43 3.2 Multivariate Lévy adaptive B-spline regression 44 3.2.1 Model specifications 45 3.2.2 Comparisons between basis fucntions of MLABS and MARS 47 3.2.3 Posterior inference 50 3.2.4 Binomial regressions for MLABS 53 3.3 Simulation studies 55 3.3.1 Surface examples 58 3.3.2 Friedman's examples 60 3.4 Real data applications 63 3.4.1 Regression examples 64 3.4.2 Classification examples 66 3.5 Discussion 67 4 Concluding Remarks 70 A Appendix 72 A.1 Appendix for Chapter 2 72 A.1.1 Proof of Theorem 2.3.1 72 A.1.2 Proof of Theorem 2.3.2 75 A.1.3 Proof of Theorem 2.3.3 75 A.1.4 Full simulation results for Simulation 1 79 A.1.5 Derivation of the full conditionals for LABS 83 Bibliography 87 Abstract in Korean 95박

Bayesian nonparametric multivariate convex regression

In many applications, such as economics, operations research and reinforcement learning, one often needs to estimate a multivariate regression function f subject to a convexity constraint. For example, in sequential decision processes the value of a state under optimal subsequent decisions may be known to be convex or concave. We propose a new Bayesian nonparametric multivariate approach based on characterizing the unknown regression function as the max of a random collection of unknown hyperplanes. This specification induces a prior with large support in a Kullback-Leibler sense on the space of convex functions, while also leading to strong posterior consistency. Although we assume that f is defined over R^p, we show that this model has a convergence rate of log(n)^{-1} n^{-1/(d+2)} under the empirical L2 norm when f actually maps a d dimensional linear subspace to R. We design an efficient reversible jump MCMC algorithm for posterior computation and demonstrate the methods through application to value function approximation

Biometrika

We consider shape restricted nonparametric regression on a closed set [Formula: see text], where it is reasonable to assume the function has no more than | local extrema interior to [Formula: see text]. Following a Bayesian approach we develop a nonparametric prior over a novel class of local extremum splines. This approach is shown to be consistent when modeling any continuously differentiable function within the class considered, and is used to develop methods for testing hypotheses on the shape of the curve. Sampling algorithms are developed, and the method is applied in simulation studies and data examples where the shape of the curve is of interest.CC999999/Intramural CDC HHS/United States2018-12-01T00:00:00Z29422695PMC5798493vault:2622

Bayesian methods in bioinformatics

This work is directed towards developing flexible Bayesian statistical methods in the semi- and nonparamteric regression modeling framework with special focus on analyzing data from biological and genetic experiments. This dissertation attempts to solve two such problems in this area. In the first part, we study penalized regression splines (P-splines), which are low-order basis splines with a penalty to avoid under- smoothing. Such P-splines are typically not spatially adaptive, and hence can have trouble when functions are varying rapidly. We model the penalty parameter inherent in the P-spline method as a heteroscedastic regression function. We develop a full Bayesian hierarchical structure to do this and use Markov Chain Monte Carlo tech- niques for drawing random samples from the posterior for inference. We show that the approach achieves very competitive performance as compared to other methods. The second part focuses on modeling DNA microarray data. Microarray technology enables us to monitor the expression levels of thousands of genes simultaneously and hence to obtain a better picture of the interactions between the genes. In order to understand the biological structure underlying these gene interactions, we present a hierarchical nonparametric Bayesian model based on Multivariate Adaptive Regres-sion Splines (MARS) to capture the functional relationship between genes and also between genes and disease status. The novelty of the approach lies in the attempt to capture the complex nonlinear dependencies between the genes which could otherwise be missed by linear approaches. The Bayesian model is flexible enough to identify significant genes of interest as well as model the functional relationships between the genes. The effectiveness of the proposed methodology is illustrated on leukemia and breast cancer datasets

Penalized spline models and applications

Penalized spline regression models are a popular statistical tool for curve fitting problems due to their flexibility and computational efficiency. In particular, penalized cubic spline functions have received a great deal of attention. Cubic splines have good numerical properties and have proven extremely useful in a variety of applications. Typically, splines are represented as linear combinations of basis functions. However, such representations can lack numerical stability or be difficult to manipulate analytically. The current thesis proposes a different parametrization for cubic spline functions that is intuitive and simple to implement. Moreover, integral based penalty functionals have simple interpretable expressions in terms of the components of the parametrization. Also, the curvature of the function is not constrained to be continuous everywhere on its domain, which adds flexibility to the fitting process. We consider not only models where smoothness is imposed by means of a single penalty functional, but also a generalization where a combination of different measures of roughness is built in order to specify the adequate limit of shrinkage for the problem at hand. The proposed methodology is illustrated in two distinct regression settings

Locally adaptive smoothing with Markov random fields and shrinkage priors

We present a locally adaptive nonparametric curve fitting method that operates within a fully Bayesian framework. This method uses shrinkage priors to induce sparsity in order-k differences in the latent trend function, providing a combination of local adaptation and global control. Using a scale mixture of normals representation of shrinkage priors, we make explicit connections between our method and kth order Gaussian Markov random field smoothing. We call the resulting processes shrinkage prior Markov random fields (SPMRFs). We use Hamiltonian Monte Carlo to approximate the posterior distribution of model parameters because this method provides superior performance in the presence of the high dimensionality and strong parameter correlations exhibited by our models. We compare the performance of three prior formulations using simulated data and find the horseshoe prior provides the best compromise between bias and precision. We apply SPMRF models to two benchmark data examples frequently used to test nonparametric methods. We find that this method is flexible enough to accommodate a variety of data generating models and offers the adaptive properties and computational tractability to make it a useful addition to the Bayesian nonparametric toolbox.Comment: 38 pages, to appear in Bayesian Analysi