Efficient Algorithms for Mumford-Shah and Potts Problems

In this work, we consider Mumford-Shah and Potts models and their higher order generalizations. Mumford-Shah and Potts models are among the most well-known variational approaches to edge-preserving smoothing and partitioning of images. Though their formulations are intuitive, their application is not straightforward as it corresponds to solving challenging, particularly non-convex, minimization problems. The main focus of this thesis is the development of new algorithmic approaches to Mumford-Shah and Potts models, which is to this day an active field of research. We start by considering the situation for univariate data. We find that switching to higher order models can overcome known shortcomings of the classical first order models when applied to data with steep slopes. Though the existing approaches to the first order models could be applied in principle, they are slow or become numerically unstable for higher orders. Therefore, we develop a new algorithm for univariate Mumford-Shah and Potts models of any order and show that it solves the models in a stable way in O(n^2). Furthermore, we develop algorithms for the inverse Potts model. The inverse Potts model can be seen as an approach to jointly reconstructing and partitioning images that are only available indirectly on the basis of measured data. Further, we give a convergence analysis for the proposed algorithms. In particular, we prove the convergence to a local minimum of the underlying NP-hard minimization problem. We apply the proposed algorithms to numerical data to illustrate their benefits. Next, we apply the multi-channel Potts prior to the reconstruction problem in multi-spectral computed tomography (CT). To this end, we propose a new superiorization approach, which perturbs the iterates of the conjugate gradient method towards better results with respect to the Potts prior. In numerical experiments, we illustrate the benefits of the proposed approach by comparing it to the existing Potts model approach from the literature as well as to the existing total variation type methods. Hereafter, we consider the second order Mumford-Shah model for edge-preserving smoothing of images which –similarly to the univariate case– improves upon the classical Mumford-Shah model for images with linear color gradients. Based on reformulations in terms of Taylor jets, i.e. specific fields of polynomials, we derive discrete second order Mumford-Shah models for which we develop an efficient algorithm using an ADMM scheme. We illustrate the potential of the proposed method by comparing it with existing methods for the second order Mumford-Shah model. Further, we illustrate its benefits in connection with edge detection. Finally, we consider the affine-linear Potts model for the image partitioning problem. As many images possess linear trends within homogeneous regions, the classical Potts model frequently leads to oversegmentation. The affine-linear Potts model accounts for that problem by allowing for linear trends within segments. We lift the corresponding minimization problem to the jet space and develop again an ADMM approach. In numerical experiments, we show that the proposed algorithm achieves lower energy values as well as faster runtimes than the method of comparison, which is based on the iterative application of the graph cut algorithm (with α-expansion moves)

Development and Properties of Kernel-based Methods for the Interpretation and Presentation of Forensic Evidence

The inference of the source of forensic evidence is related to model selection. Many forms of evidence can only be represented by complex, high-dimensional random vectors and cannot be assigned a likelihood structure. A common approach to circumvent this is to measure the similarity between pairs of objects composing the evidence. Such methods are ad-hoc and unstable approaches to the judicial inference process. While these methods address the dimensionality issue they also engender dependencies between scores when 2 scores have 1 object in common that are not taken into account in these models. The model developed in this research captures the dependencies between pairwise scores from a hierarchical sample and models them in the kernel space using a linear model. Our model is flexible to accommodate any kernel satisfying basic conditions and as a result is applicable to any type of complex high-dimensional data. An important result of this work is the asymptotic multivariate normality of the scores as the data dimension increases. As a result, we can: 1) model very high-dimensional data when other methods fail; 2) determine the source of multiple samples from a single trace in one calculation. Our model can be used to address high-dimension model selection problems in different situations and we show how to use it to assign Bayes factors to forensic evidence. We will provide examples of real-life problems using data from very small particles and dust analyzed by SEM/EDX, and colors of fibers quantified by microspectrophotometry

푸리에 계수의 비모수적 추정을 이용한 베이지안 회귀분석과 그 응용

학위논문 (박사) -- 서울대학교 대학원 : 자연과학대학 통계학과, 2021. 2. 임채영.We illustrate a nonparmetric modeling of Fourier coefficients, known as a spectral density, under a Bayesian framework to forecast a stationary one-dimensional random process and to predict a stationary two-dimensional random field on a regular grid. We switch from the time/space domain to the frequency domain, and introduce a Gaussian process prior to the log-spectral density. First, we propose Bayesian modeling of spectral density for spatial regression on a regular lattice grid. An interpolation technique to convert an estimated spectral density to a covariance matrix is also proposed to avoid matrix inversion for the spatial prediction. Simulation study shows that our approach is robust in that it does not require a parametric form and/or isotropic assumption of a covariance function. Also, our approach gives better prediction results over conventional spatial prediction under most parametric covariance models that we considered. We also compare our approach with other existing spatial prediction approaches using two datasets of Korean ozone concentration. Our approach performs reasonably good in terms of mean absolute error and root mean squared error. Second, we propose Bayesian modeling of spectral density for time series regression with heteroscedastic autocovariance. Heteroskedastic autocovariance is modeled as time varying marginal variance multiplied by stationary autocorrelation. Bayesian Markov-Chain-Monte-Carlo(MCMC) is used to estimate coefficients of the B-spline basis representation of the log marginal variance function as well as a log spectral density at Fourier frequencies so that we can estimate time varying autocovariance function. Simulation results show that the proposed approach successfully detected the temporal pattern of the autocovariance structure. Even though we need to estimate spectral density at all Fourier frequencies during Bayesian procedure, our approach does not lose much efficiency on computation compared to estimating only a few parameters in a parametric model such as $ARMA$ or $GARCH$. We applied the proposed method to forecast foreign exchange rate data and it shows good prediction accuracy in a sense of overall low root mean squared errors.본 박사학위논문에서는 스펙트럴 밀도라 불리우는 일종의 푸리에(Fourier) 계수를 베이지안(Bayesian) 마코프-체인-몬테-카를로(MCMC) 관점에서 비모수적으로 모형화하는 통계적방법론을 제안하는데, 이는 등간격의 격자점에서 정의된, 정상성(stationarity)을 가진 1 차원 또는 2차원 확률과정을 예측하는 역할을 수행한다. 핵심 원리는 시간 또는 공간 영역에서 정의된 자기공분산함수를 푸리에변환을 통해 주파수 영역에서의 스펙트럴 밀도함수로 전환하는 것, 그리고 사후분석(posterior analysis)을 위해서 그 스펙트럼 밀도의 로그변환된 함수에 가우시안(Gaussian)과정 사전분포를 부여하는 것이다. 먼저 공간 자료 예측 문제에 적용할 수 있는 모형을 제안한다. 스펙트럴 밀도함수를 공분산 함수로 변환할 때 본 논문에서 제안하는 보간 기법은 전통적인 공간예측 모형에서 필요로 했던 역행렬 계산을 생략함으로서 계산 부담을 줄여준다. 본 모형은 어떠한 알려진 형태의 함수나 등방성 등의 가정을 필요로하지 않으면서도 기존에 대표적인 공간예측모형들과 비교했을 때 비슷하거나 혹은 더 나은 예측력을 가져다 준다는 것이 시뮬레이션 연구를 통해 입증되었다. 또한 이 모형을 MODIS, AURA와 같은 공신력을 가진 위성자료를 이용하여 한국 지역의 오존농도를 예측하는 문제에 적용했을 때에도 비교적 좋은 예측력을 갖는다는 것이 입증되었다. 다음으로 시계열 자료 예측 문제에 적용할 수 있는 모형을 제안한다. 여기서는 특히 정상성(stationarity) 가정이 일부 완화되어 자기공분산의 한계치(marginal auto-covariance)가 시간에 따라 변하는 이분산성(heteroscedasticity) 확률과정을 생각한다. 이 때 자기공분산은 시간에 따라 변하는 한계분산함수와 정상성을 가진 자기상관함수 사이의 곱으로 표현된다. 자기상관함수의 추정은 기존의 아이디어를 따르고, 한계분산함수의 추정에 있어서는 B-spline 기저함수를 이용한 비모수 추정법을 도입한다. 새로 도입된 과정 역시 하나의 베이지안 마코프-체인-몬테-카를로(MCMC) 안에서 구현된다. 시뮬레이션 연구를 통해 제안한 방법이 등분산성 혹은 이분산성을 지닌 시계열자료의 시간에 따른 패턴을 잘 잡아내는 것이 밝혀졌다. 기존에 잘 알려진 방법인 $ARMA$나 $GARCH$와 같은 모수적 방법론보다 훨씬 많은 수의 모수를 추정해야 함에도 계산 효율은 크게 떨어지지 않는 모습을 보여주고 있다. 본 모형을 대표적인 외국환율 자료 분석에 응용했을 때, 많은 경우 평균제곱오차의 관점에서 전반적으로 예측지점별 오차가 비교적 적게 나오는 것으로 확인되었다.Contents Abstract i 1 Introduction 1 2 Bayesian spatial regression using non-parametric modeling of Fourier coefficients 12 2.1 Preliminaries . . . . . . . . . . . . . . . . . . . . . 12 2.1.1 Spectral representation theorem . . . . . . 13 2.1.2 Whittle Likelihood Approximation . . . . . 15 2.1.3 Fast Fourier Transform algorithm . . . . . . 18 2.2 Proposed model . . . . . . . . . . . . . . . . . . . . 19 2.2.1 Periodogram . . . . . . . . . . . . . . . . . 19 2.2.2 Gaussian Mixture Approximation . . . . . . 20 2.2.3 Proposed Gibbs sampler . . . . . . . . . . . 21 2.2.4 Prediction procedures . . . . . . . . . . . . 22 2.3 Proposed model for observations on an incomplete grid . . . . . . . . . . . . . . . . . . . . . . . . . . 24 2.4 Simulation Study . . . . . . . . . . . . . . . . . . . 25 2.5 Real Data Analysis . . . . . . . . . . . . . . . . . . 33 iii 3 Bayesian time series regression using non-parametric modeling of Fourier coefficients 42 3.1 Preliminaries . . . . . . . . . . . . . . . . . . . . . 42 3.2 Proposed Method . . . . . . . . . . . . . . . . . . . 44 3.3 Simulation Study . . . . . . . . . . . . . . . . . . . 46 3.4 Real Data Analysis . . . . . . . . . . . . . . . . . . 52 4 Concluding remarks 59 A Conditional posterior distributions 71 Bibliography 75 B Proofs of the main results 75 Abstract (in Korean) 79Docto

Analysis of the Stochastic Stability and Asymptotically Stationary Statistics for a Class of Nonlinear Attitude Estimation Algorithms

Attitude estimation algorithms are critical components of satellite control systems, aircraft autopilots, and other applications. Attitude estimation systems perform their task by fusing attitude and gyroscope measurements; however, such measurements are typically corrupted by random noise and gyroscopes may have significant bias. Variations of the extended Kalman filter are commonly used, but this technique relies on instantaneous linearization of the underlying nonlinear dynamics and global stability cannot be guaranteed. Nonlinear attitude observers with guaranteed global stability have been derived and experimentally demonstrated, but only for the deterministic setting where no stochastic effects are present. The first part of this thesis extends a deterministic nonlinear attitude estimator by introducing additional dynamics that allow learning variations of gyro bias as a function of operating temperature, a common source of bias variation in rate gyro readings. The remainder of the thesis formally addresses the problem of stochastic stability and asymptotic performance for this family of estimators when the measurements contain random noise. Analysis tools from stochastic differential equation theory and stochastic Lyapunov analysis are used together to demonstrate convergence of the filter states to a stationary distribution, and to bound the associated steady-state statistics as a function of filter gains and sensor parameters. In many cases these bounds are conservative, but solutions have been found for the associated stationary Fokker-Planck PDEs for two cases. When only the gyro measurement contains noise, the attitude estimation errors are shown to converge to a bipolar Bingham distribution. When the gyro measurement is further assumed to have constant bias, the estimation errors are shown to converge to a joint bipolar Bingham and multivariate Gaussian distribution. Knowledge of the stationary distributions allow for exact computation of steady-state statistics. Further, the analysis suggests a method for modeling a continuous quaternion noise process with specified statistics on SO(3); this model is used for analyzing estimator performance when both the gyro and the attitude measurements contain noise. Bounds and exact predictions for the different noise models are validated using a high fidelity numerical integration method for nonlinear stochastic differential equations

Enhancing Classification and Regression Tree-Based Models by means of Mathematical Optimization

This PhD dissertation bridges the disciplines of Operations Research and Machine Learning by developing novel Mathematical Optimization formulations and numerical solution approaches to build classification and regression tree-based models. Contrary to classic classification and regression trees, built in a greedy heuristic manner, formulating the design of the tree model as an optimization problem allows us to easily include, either as hard or soft constraints, desirable global structural properties. In this PhD dissertation, we illustrate this flexibility to model: sparsity, as a proxy for interpretability, by controlling the number of non-zero coefficients, the number of predictor variables and, in the case of functional ones, the proportion of the domain used for prediction; an important social criterion, the fairness of the model, which aims to avoid predictions that discriminate against race, or other sensitive features; and the cost-sensitivity for groups at risk, by ensuring an acceptable accuracy performance for them. Moreover, we provide in a natural way the impact that continuous predictor variables have on each individual prediction, thus enhancing the local explainability of tree models. All the approaches proposed in this thesis are formulated through Continuous Optimization problems that are scalable with respect to the size of the training sample, are studied theoretically, are tested in real data sets and are competitive in terms of prediction accuracy against benchmarks. This, together with the good properties summarized above, is illustrated through the different chapters of this thesis. This PhD dissertation is organized as follows. The state of the art in the field of (optimal) decision trees is fully discussed in Chapter 1, while the next four chapters state our methodology. Chapter 2 introduces in detail the general framework that threads the chapters in this thesis: a randomized tree with oblique cuts. Particularly, we present our proposal to deal with classification problems, which naturally provides probabilistic output on class membership tailored to each individual, in contrast to the most popular existing approaches, where all individuals in the same leaf node are assigned the same probability. Preferences on classification rates in critical classes are successfully handled through cost-sensitive constraints. Chapter 3 extends the methodology for classification in Chapter 2 to additionally handle sparsity. This is modeled by means of regularizations with polyhedral norms added to the objective function. The sparsest tree case is theoretically studied. Our ability to easily trade in some of our classification accuracy for a gain in sparsity is shown. In Chapter 4, the findings obtained in Chapters 2 and 3 are adapted to construct sparse trees for regression. Theoretical properties of the solutions are explored. The scalability of our approach with respect to the size of the training sample, as well as local explanations on the continuous predictor variables, are illustrated. Moreover, we show how this methodology can avoid the discrimination of sensitive groups through fairness constraints. Chapter 5 extends the methodology for regression in Chapter 4 to consider functional predictor variables instead. Simultaneously, the detection of a reduced number of intervals that are critical for prediction is performed. The sparsity in the proportion of the domain of the functional predictor variables to be used is also modeled through a regularization term added to the objective function. The obtained trade off between accuracy and sparsity is illustrated. Finally, Chapter 6 closes the thesis with general conclusions and future lines of research.Esta tesis combina las disciplinas de Investigación Operativa y Aprendizaje Automático a través del desarrollo de formulaciones de Optimización Matemática y algoritmos de resolución numérica para construir modelos basados en árboles de clasificación y regresión. A diferencia de los árboles de clasificación y regresión clásicos, generados de manera heurística y voraz, construir un árbol a través de un problema de optimización nos permite incluir fácilmente propiedades estructurales globales deseables. En esta tesis, ilustramos esta flexibilidad para modelar los siguientes aspectos: sparsity, como sinónimo de interpretabilidad, controlando el número de coeficientes no nulos, el número de variables predictoras y, si son funcionales, la proporción de dominio usado en la predicción; un criterio social importante, la equidad del modelo, evitando predicciones que discriminen a algunos individuos por su etnia u otras características sensibles; y la sensibilidad al coste de grupos de riesgo, asegurando un rendimiento aceptable para ellos. Además, con este enfoque se obtiene de manera natural el impacto que las variables predictoras continuas tienen en la predicción de cada individuo, mejorando así la explicabilidad local de los modelos de clasificación y regresión basados en árboles. Todos los enfoques propuestos en esta tesis se formulan a través de problemas de Optimización Continua que son escalables con respecto al tamaño de la muestra de entrenamiento, se estudian desde el punto de vista teórico, se evalúan en conjuntos de datos reales y son competitivos frente a los procedimientos habituales. Esto, junto a las buenas propiedades resumidas en el párrafo anterior, se ilustra a lo largo de los diferentes capítulos de esta tesis. La tesis se estructura de la siguiente manera. El estado del arte sobre árboles de decisión (óptimos) se discute ampliamente en el Capítulo 1, mientras que los cuatro capítulos siguientes exponen nuestra metodología. El Capítulo 2 introduce de forma detallada el marco general que hila los capítulos de esta tesis: un árbol aleatorizado con cortes oblicuos. En particular, presentamos nuestra propuesta para tratar problemas de clasificación, la cual construye la probabilidad de pertenencia a cada clase ajustada a cada individuo, a diferencia de las técnicas más populares existentes, en las que a todos los individuos en el mismo nodo hoja se les asigna la misma probabilidad. Se tratan con éxito preferencias en las tasas de clasificación en clases críticas mediante restricciones de sensibilidad al coste. El Capítulo 3 extiende la metodología de clasificación del Capítulo 2 para tratar adicionalmente sparsity. Esto se modela mediante regularizaciones con normas poliédricas que se añaden a la función objetivo. Se estudian propiedades teóricas del árbol más sparse, y se demuestra nuestra habilidad para sacrificar un poco de precisión en la clasificación por una ganancia en sparsity. En el Capítulo 4, los resultados obtenidos en los Capítulos 2 y 3 se adaptan para construir árboles sparse para regresión. Se exploran propiedades teóricas de las soluciones. Los experimentos numéricos demuestran la escalabilidad de nuestro enfoque con respecto al tamaño de la muestra de entrenamiento, y se ilustra cómo se generan las explicaciones locales en las variables predictoras continuas. Además, mostramos cómo esta metodología puede reducir la discriminación de grupos sensibles a través de las denominadas restricciones de justicia. El Capítulo 5 extiende la metodología de regresión del Capítulo 4 para considerar variables predictoras funcionales. De manera simultánea, la detección de un número reducido de intervalos que son críticos para la predicción es abordada. La sparsity en la proporción de dominio de las variables predictoras funcionales a usar se modela también a través de un término de regularización añadido a la función objetivo. De esta forma, se ilustra el equilibrio obtenido entre la precisión de predicción y la sparsity en este marco. Por último, el Capítulo 6 cierra la tesis con conclusiones generales y líneas futuras de investigación

Calculation of the flow field in supersonic mixed-compression inlets at angle of attack using the three-dimensional method of characteristics with discrete shock wave fitting

The influence of molecular transport is included in the computation by treating viscous and thermal diffusion terms in the governing partial differential equations as correction terms in the method of characteristics scheme. The development of a production type computer program is reported which is capable of calculating the flow field in a variety of axisymmetric mixed-compression aircraft inlets. The results agreed well with those produced by the two-dimensional method characteristics when axisymmetric flow fields are computed. For three-dimensional flow fields, the results agree well with experimental data except in regions of high viscous interaction and boundary layer removal

Toward quantitative limited-angle ultrasound reflection tomography to inform abdominal HIFU treatment planning

High-Intensity Focused Ultrasound (HIFU) is a treatment modality for solid cancers of the liver and pancreas which is non-invasive and free from many of the side-effects of radiotherapy and chemotherapy. The safety and efficacy of abdominal HIFU treatment is dependent on the ability to bring the therapeutic sound waves to a small focal ”lesion” of known and controllable location within the patient anatomy. To achieve this, pre-treatment planning typically includes a numerical simulation of the therapeutic ultrasound beam, in which anatomical compartment locations are derived from computed tomography or magnetic resonance images. In such planning simulations, acoustic properties such as density and speed-of-sound are assumed for the relevant tissues which are rarely, if ever, determined specifically for the patient. These properties are known to vary between patients and disease states of tissues, and to influence the intensity and location of the HIFU lesion. The subject of this thesis is the problem of non-invasive patient-specific measurement of acoustic tissue properties. The appropriate method, also, of establishing spatial correspondence between physical ultrasound transducers and modeled (imaged) anatomy via multimodal image reg-istration is also investigated; this is of relevance both to acoustic tissue property estimation and to the guidance of HIFU delivery itself. First, the principle of a method is demonstrated with which acoustic properties can be recovered for several tissues simultaneously using reflection ultrasound, given accurate knowledge of the physical locations of tissue compartments. Second, the method is developed to allow for some inaccuracy in this knowledge commensurate with the inaccuracy typical in abdominal multimodal image registration. Third, several current multimodal image registration techniques, and two novel modifications, are compared for accuracy and robustness. In conclusion, relevant acoustic tissue properties can, in principle, be estimated using reflected ultrasound data that could be acquired using diagnostic imaging transducers in a clinical setting

Statistical learning of random probability measures

The study of random probability measures is a lively research topic that has attracted interest from different fields in recent years. In this thesis, we consider random probability measures in the context of Bayesian nonparametrics, where the law of a random probability measure is used as prior distribution, and in the context of distributional data analysis, where the goal is to perform inference given avsample from the law of a random probability measure. The contributions contained in this thesis can be subdivided according to three different topics: (i) the use of almost surely discrete repulsive random measures (i.e., whose support points are well separated) for Bayesian model-based clustering, (ii) the proposal of new laws for collections of random probability measures for Bayesian density estimation of partially exchangeable data subdivided into different groups, and (iii) the study of principal component analysis and regression models for probability distributions seen as elements of the 2-Wasserstein space. Specifically, for point (i) above we propose an efficient Markov chain Monte Carlo algorithm for posterior inference, which sidesteps the need of split-merge reversible jump moves typically associated with poor performance, we propose a model for clustering high-dimensional data by introducing a novel class of anisotropic determinantal point processes, and study the distributional properties of the repulsive measures, shedding light on important theoretical results which enable more principled prior elicitation and more efficient posterior simulation algorithms. For point (ii) above, we consider several models suitable for clustering homogeneous populations, inducing spatial dependence across groups of data, extracting the characteristic traits common to all the data-groups, and propose a novel vector autoregressive model to study of growth curves of Singaporean kids. Finally, for point (iii), we propose a novel class of projected statistical methods for distributional data analysis for measures on the real line and on the unit-circle