BOCK : Bayesian Optimization with Cylindrical Kernels

A major challenge in Bayesian Optimization is the boundary issue (Swersky, 2017) where an algorithm spends too many evaluations near the boundary of its search space. In this paper, we propose BOCK, Bayesian Optimization with Cylindrical Kernels, whose basic idea is to transform the ball geometry of the search space using a cylindrical transformation. Because of the transformed geometry, the Gaussian Process-based surrogate model spends less budget searching near the boundary, while concentrating its efforts relatively more near the center of the search region, where we expect the solution to be located. We evaluate BOCK extensively, showing that it is not only more accurate and efficient, but it also scales successfully to problems with a dimensionality as high as 500. We show that the better accuracy and scalability of BOCK even allows optimizing modestly sized neural network layers, as well as neural network hyperparameters.Comment: 10 pages, 5 figures, 5 tables, 1 algorith

Combinatorial Bayesian Optimization using the Graph Cartesian Product

This paper focuses on Bayesian Optimization (BO) for objectives on combinatorial search spaces, including ordinal and categorical variables. Despite the abundance of potential applications of Combinatorial BO, including chipset configuration search and neural architecture search, only a handful of methods have been proposed. We introduce COMBO, a new Gaussian Process (GP) BO. COMBO quantifies "smoothness" of functions on combinatorial search spaces by utilizing a combinatorial graph. The vertex set of the combinatorial graph consists of all possible joint assignments of the variables, while edges are constructed using the graph Cartesian product of the sub-graphs that represent the individual variables. On this combinatorial graph, we propose an ARD diffusion kernel with which the GP is able to model high-order interactions between variables leading to better performance. Moreover, using the Horseshoe prior for the scale parameter in the ARD diffusion kernel results in an effective variable selection procedure, making COMBO suitable for high dimensional problems. Computationally, in COMBO the graph Cartesian product allows the Graph Fourier Transform calculation to scale linearly instead of exponentially. We validate COMBO in a wide array of realistic benchmarks, including weighted maximum satisfiability problems and neural architecture search. COMBO outperforms consistently the latest state-of-the-art while maintaining computational and statistical efficiency.Comment: Accepted to NeurIPS 2019, code: https://github.com/QUVA-Lab/COMB

Major medical causes by breed and life stage for dogs presented at veterinary clinics in the Republic of Korea: a survey of electronic medical records

Background Age and breed are considered the greatest risk factors for disease prevalence and mortality in companion dogs. Understanding the prevalence of diseases, in relation to age and breed, would support appropriate guidance for future health care strategies and provide useful information for the early diagnosis of diseases. The purpose of this study was to investigate the major medical causes for dogs visiting primary-care veterinary clinics in the Republic of Korea, stratified by age and breed. Methods A total of 15,531 medical records of canine patients were analyzed from 11 veterinary clinics who shared data from January 1, 2016 to December 31, 2016. An electronic medical record (EMR) system was used for data collection, which included the animal identification number, age, breed, gender, neuter status, clinical information, and diagnosis. EMR data were classified using the International Classification of Disease system from the World Health Organization; presenting signs or diagnoses were identified according to breed and life stage. Results Within the age groups, preventive medicine (16.7% confidence intervals (CI) [15.9–17.5]) was the most common cause for clinic visits for the 10 year), the prevalences of heart disease, kidney disease, Cushing’s disease, and mammary tumors were higher than in the other age groups. Small and toy breed dogs comprised 67.7% of all dogs in this analysis. For all breeds, otitis externa, dermatitis or eczema, vomiting, and diarrhea were common medical problems. Discussion This study identified the most common medical disorders and differences in prevalences of diseases, according to age and breeds. The information from EMRs for dogs visiting primary-care veterinary clinics can provide background knowledge that is required to enable a better understanding of disease patterns and occurrence by age and breeds. The information from this study could enable the creation of strategies for preventing diseases and enable the identification of health problems for more effective disease management in companion dogs

Radial and Directional Posteriors for Bayesian Deep Learning

We propose a new variational family for Bayesian neural networks. We decompose the variational posterior into two components, where the radial component captures the strength of each neuron in terms of its magnitude; while the directional component captures the statistical dependencies among the weight parameters. The dependencies learned via the directional density provide better modeling performance compared to the widely-used Gaussian mean-field-type variational family. In addition, the strength of input and output neurons learned via our posterior provides a structured way to compress neural networks. Indeed, experiments show that our variational family improves predictive performance and yields compressed networks simultaneously

Mixed Variable Bayesian Optimization with Frequency Modulated Kernels

The sample efficiency of Bayesian optimization(BO) is often boosted by Gaussian Process(GP) surrogate models. However, on mixed variable spaces, surrogate models other than GPs are prevalent, mainly due to the lack of kernels which can model complex dependencies across different types of variables. In this paper, we propose the frequency modulated (FM) kernel flexibly modeling dependencies among different types of variables, so that BO can enjoy the further improved sample efficiency. The FM kernel uses distances on continuous variables to modulate the graph Fourier spectrum derived from discrete variables. However, the frequency modulation does not always define a kernel with the similarity measure behavior which returns higher values for pairs of more similar points. Therefore, we specify and prove conditions for FM kernels to be positive definite and to exhibit the similarity measure behavior. In experiments, we demonstrate the improved sample efficiency of GP BO using FM kernels (BO-FM).On synthetic problems and hyperparameter optimization problems, BO-FM outperforms competitors consistently. Also, the importance of the frequency modulation principle is empirically demonstrated on the same problems. On joint optimization of neural architectures and SGD hyperparameters, BO-FM outperforms competitors including Regularized evolution(RE) and BOHB. Remarkably, BO-FM performs better even than RE and BOHB using three times as many evaluations.Comment: Proceedings of the Thirty-Seventh Conference on Uncertainty in Artificial Intelligence, PMLR 161:950-960, 202

Batch Bayesian Optimization on Permutations using Acquisition Weighted Kernels

In this work we propose a batch Bayesian optimization method for combinatorial problems on permutations, which is well suited for expensive cost functions on permutations. We introduce LAW, a new efficient batch acquisition method based on the determinantal point process, using an acquisition weighted kernel. Relying on multiple parallel evaluations, LAW accelerates the search for the optimal permutation. We provide a regret analysis for our method to gain insight in its theoretical properties. We then apply the framework to permutation problems, which have so far received little attention in the Bayesian Optimization literature, despite their practical importance. We call this method LAW2ORDER. We evaluate the method on several standard combinatorial problems involving permutations such as quadratic assignment, flowshop scheduling and the traveling salesman, as well as on a structure learning task

Nitric Oxide Emission Reduction in Reheating Furnaces through Burner and Furnace Air-Staged Combustions

In this study, a series of experiments were conducted on a testing facility and a real-scale furnace, for analyzing the nitric oxide (NO) emission reduction. The effects of the temperature, oxygen concentration, and amount of secondary combustion air were investigated in a single-burner combustion system. Additionally, the NO-reduction rate before and after combustion modifications in both the burner and furnace air-staged combustion were evaluated for a real-scale reheating furnace. The air-to-fuel equivalence ratio (λ) of individual combustion zones for the furnace was optimized for NO reduction without any incomplete combustion. The results indicated that the NO emission for controlling the λ of a single-zone decreased linearly with a decrease in the λ values in the individual firing tests (top-heat, bottom-heat, and bottom-soak zones). Moreover, the multi-zone control of the λ values for individual combustion zones was optimized at 1.13 (top-preheat), 1.0 (bottom-preheat), 1.0 (top-heat), 0.97 (bottom-heat), 1.0 (top-soak), and 0.97 (bottom-soak). In this firing condition, the modifications reduced the NO emissions by approximately 23%, as indicated by a comparison of the data obtained before and after the modifications. Thus, the combined application of burner and furnace air-staged combustions facilitated NO-emission reduction