Practical Bayesian Optimization of Machine Learning Algorithms

Machine learning algorithms frequently require careful tuning of model hyperparameters, regularization terms, and optimization parameters. Unfortunately, this tuning is often a "black art" that requires expert experience, unwritten rules of thumb, or sometimes brute-force search. Much more appealing is the idea of developing automatic approaches which can optimize the performance of a given learning algorithm to the task at hand. In this work, we consider the automatic tuning problem within the framework of Bayesian optimization, in which a learning algorithm's generalization performance is modeled as a sample from a Gaussian process (GP). The tractable posterior distribution induced by the GP leads to efficient use of the information gathered by previous experiments, enabling optimal choices about what parameters to try next. Here we show how the effects of the Gaussian process prior and the associated inference procedure can have a large impact on the success or failure of Bayesian optimization. We show that thoughtful choices can lead to results that exceed expert-level performance in tuning machine learning algorithms. We also describe new algorithms that take into account the variable cost (duration) of learning experiments and that can leverage the presence of multiple cores for parallel experimentation. We show that these proposed algorithms improve on previous automatic procedures and can reach or surpass human expert-level optimization on a diverse set of contemporary algorithms including latent Dirichlet allocation, structured SVMs and convolutional neural networks

Bayesian Optimization with Unknown Constraints

Recent work on Bayesian optimization has shown its effectiveness in global optimization of difficult black-box objective functions. Many real-world optimization problems of interest also have constraints which are unknown a priori. In this paper, we study Bayesian optimization for constrained problems in the general case that noise may be present in the constraint functions, and the objective and constraints may be evaluated independently. We provide motivating practical examples, and present a general framework to solve such problems. We demonstrate the effectiveness of our approach on optimizing the performance of online latent Dirichlet allocation subject to topic sparsity constraints, tuning a neural network given test-time memory constraints, and optimizing Hamiltonian Monte Carlo to achieve maximal effectiveness in a fixed time, subject to passing standard convergence diagnostics.Comment: 14 pages, 3 figure

Observing the Spontaneous Breakdown of Unitarity

During the past decade, the experimental development of being able to create ever larger and heavier quantum superpositions has brought the discussion of the connection between microscopic quantum mechanics and macroscopic classical physics back to the forefront of physical research. Under equilibrium conditions this connection is in fact well understood in terms of the mechanism of spontaneous symmetry breaking, while the emergence of classical dynamics can be described within an ensemble averaged description in terms of decoherence. The remaining realm of individual-state quantum dynamics in the thermodynamic limit was addressed in a recent paper proposing that the unitarity of quantum mechanical time evolution in macroscopic objects may be susceptible to a spontaneous breakdown. Here we will discuss the implications of this theory of spontaneous unitarity breaking for the modern experiments involving truly macroscopic Schrodinger cat states.Comment: 4 pages, no figure

An organic geochemical approach to problems of glacial-interglacial climatic variability

Submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy at the Massachusetts Institute of Technology and the Woods Hole Oceanographic Institution May 1988The concentration and carbon isotopic composition (δ13C) of sedimentary organic carbon (Corg), N/C ratios, and terrigenous and marine δ13C-Corg end-members form a basis from which to address problems of Late Quaternary glacial-interglacial climatic variability in a 208.7 m hydraulic piston core (DSDP 619) from the Pigmy Basin in the northern Gulf of Mexico. Paired analyses of δ13C-Corg and N/C are consistent with the hypothesis that the sedimentary organic carbon in the Pigmy Basin is a climatically-determined mixture of C3-photosynthetic terrigenous and marine organic matter, confirming the model of Sackett (1964). A high resolution (~1.4-2.7 ky/sample) δ13C-Corg record shows that sedimentary organic carbon in interglacial oxygen isotope (sub)stages 1 and 5a-b are enriched in 13C (average ± 10 values are -24.2 ± 1.2% 0 and -23.0 ± 0.8% relative to PDB, respectively) while glacial isotope stage values 2 are relatively depleted (-25.6 ± 0.5%). Concentrations of terrigenous and marine sedimentary organic carbon are calculated using δ13C-Corg and Corg measurements, and terrigenous and marine δ13C-Corg end-members. The net accumulation rate of terrigenous organic carbon is 3.7±3.1 times higher in isotope stages 2-4 than in (sub)stages 1 and 5a-b, recording higher erosion rates of terrigenous organic material in glacial periods than interglacial periods. The concentration and net accumulation rates of marine and terrigenous Corg suggest that the nutrient-bearing plume of the Mississippi River may have advanced and retreated across the Pigmy Basin as sea level fell and rose in response to glacial-interglacial sea level change. A study of selected organic biomarker compounds which could serve as tracers of terrigenous and marine sedimentary organic matter sources was performed by comparison with contemporaneous sedimentary organic carbon isotopic composition (δ13C-Corg). Organic carbon-normalized concentrations of total long chain (C37-C39) unsaturated alkenones and individual C27-C29 desmethyl sterols were determined to be useful proportional indicators of preserved marine and terrigenous organic carbon, respectively. The a1kenones, whose source is marine phytoplankton of the class Prymnesiophyceae, generally occurred in higher concentrations in interglacial isotope stages 1 and 5a-b than in the intervening stages, including glacial stages 2 and 4. Sterols (C27-C29) of a dominantly terrigenous origin had lower concentrations during interglacial stages than in glacial stages. The sedimentary records of both terrigenous and marine organic carbon-normalized biomarker compound concentrations appear to be systematically altered by the remineralization of sedimentary organic carbon, as indicated by a simple, first-order organic carbon decay model. The sedimentary deposition of some terrigenous 4-desmethy1stero1s may be affected by differential hydraulic particle sorting as they are transported from river deltas across the continental shelf and slope to the hemipelagic Pigmy Basin. The marine phytoplanktonic alkenones which originate in the surface ocean and sink through the water column would not be subject to comparable particle sorting. The lack of any 4-desmethyl- or 4-α-methy1sterol which was linearly related to the proportion of marine sedimentary organic matter (as scaled by δ13C-Corg) indicated that either (1) sedimentary diagenesis had obscured the biomarker/Corg versus δ13C-Corg record, or (2) the selected compounds were not proportional indicators of preserved marine organic carbon input. The diagenetic alteration of the sedimentary sterol concentration records in which marine sterols were apparently more susceptible to degradation than terrigenous sterols was consistent with present-day sediment trap and recent (10-1-102y) sediment core observations. Preferential preservation of terrigenous sterols may result in a biased sedimentary record of sterol input which could be misinterpreted as indicating solely terrigenous sterol sources. The value and limitations of a simple model which characterizes the effects of sedimentary diagenesis and source input changes on the relationship between organic carbon-normalized biomarker compounds and sedimentary organic matter carbon isotopic composition are discussed. The potential occurrence of sterol double bond hydrogenations (Δ5',Δ22) in three classes of C27-29-4-desmethy1stero1s was evaluated by examining the time series of expected product/precursor relationships with sterol data from the ~2-100kybp DSDP 619 record. Only the Δ5-hydrogenations of the C29 sterols (24-ethy1cholest-5-en- 3β-01, 24-ethylcholesta-5,22-dien-3β-01) showed significant temporally-increasing trends. The 24-ethy1cho1estan-3β-01/24-ethy1cho1est- 5-en-3β-ol (C29Δ°/C29Δ5) ratio also positively correlated with paired sedimentary organic carbon isotopic composition (δ13C-Corg) values. This may be due to increased susceptibility to diagenetic transformation reactions by the organic matter accompanying finer grain-sized terrigenous sediment particles. A long-term source change of 24-ethylcholestan-3β-01 relative to 24-ethylcholest-5-en-3β-01 to explain the correlation with δ13C-Corg seems less likely since both compounds are predominantly of a terrigenous origin in the Pigmy Basin. A comparison of histograms of stanol/stenol (ΔO/Δ5) ratios for the C27-29 -4-desmethylsterols indicates the following sequence in the relative degree of tranformation: C27 > C28 > C29. The C27 - and C28-sterols appear to have attained their respective degrees of transformation before -2kybp, perhaps prior to deposition in the Pigmy Basin. However, differential rates of competing reactions of both the precursor and products may have obscured these simple transformation ratio records. The sedimentary record of a ratio (Uk37) of long chain (C37) unsaturated alkenones is a useful indicator of glacial-interglacial climatic change in the Late Quaternary northern Gulf of Mexico where a planktonic foraminiferal δ18O-CaCO3 record is complicated by meltwater and/or fluvial events (Williams and Kohl, 1986). Using laboratory temperature calibration data of the Uk37 ratio (Prahl and Wakeham, 1987), it is suggested that the minimum glacial surface mixed layer (SML) temperature was 8±1°C colder than the Holocene high SML temperature of 25.6±O.5°C in a Pigmy Basin hydraulic piston core (DSDP 619). However, this glacial-interglacial Uk37-temperature difference was significantly larger than the differences predicted by either the foraminiferal δ18O or foraminiferal assemblage temperature methods (O.8-2.0°C). A possible cause for this large difference is that the Prymnesiophyte assemblages in this area may vary in response to climatically-induced hydrographic changes. Interglacial periods may be dominated by pelagic Prymnesiophyte assemblages, while glacial periods may be dominated by neritic assemblages. Correlation of the Uk37 ratio with the sedimentary organic carbon composition (δ13C-Corg) is consistent with the predominance of preserved input of erosive terrigenous over marine organic carbon during glacial stages in the northern Gulf of Mexico when sea level was as much as 150m lower than in the present interglacial stage. Marine organic carbon burial dominated in warmer interglacial stages 1 and 5a-b

Recalibrating the compass: towards effective competition law enforcement on mixed markets

The undertaking, a cornerstone of EU competition law, has consistently been approached as a functional concept. Any entity engaged in economic activity should be considered an undertaking, thereby ensuring consistent application of competition law across competitors. However, current national enforcement practice reveals a departure from the functional approach to the undertaking on mixed markets, where public and private firms compete. Particularly, allegedly anticompetitive behaviour by public entities has evaded competition law scrutiny in the Netherlands, because competition law was found not to apply to these public entities. Drawing on a jointly interpreted string of CJEU cases in competition law and state aid law – which this article coins as the “Compass doctrine” – the Dutch competition authority and courts found that economic activity by public entities is exempt from competition law when connected to the exercise of public power. Analysis of the Compass doctrine cases reveals how a number of case-specific outcomes taken together have allowed for an undermining of the functional approach to the undertaking. It is demonstrated how the sum of the Compass doctrine is larger than its individual parts, which seems to have been unforeseen by the CJEU. This article demonstrates how the Compass doctrine has two adverse consequences: (1) because it undermines the functional approach to the undertaking as the subject of competition law, it impedes effective enforcement; (2) the Compass doctrine enables public firms to behave anticompetitively on mixed markets. The CJEU never anticipated the advent of commercial behaviour by public entities, who with the Compass doctrine in hand can infringe competition law with impunity. Experiences in the Netherlands to this effect should be regarded as a canary in the coal mine for mixed markets across the EU. Therefore, it is incumbent on the CJEU to revisit the Compass doctrine in future cases, which may follow from preliminary references. This article recommends the CJEU to (re)emphasize that once an entity is engaged in economic activity, it can no longer escape competition law scrutiny by being connected to the exercise of public authority. To protect the level playing field on mixed markets, all economic activity should explicitly be subject to EU competition law

Effects of industrial development on rural welfare

Writers and exponents of the decentralization-of-industry theory seem to think that it is the hope of rural people. It is not the purpose or intention of this study to prove that industrial development is desirable or objectionable, but to find out its effects on rural welfare as revealed by data collected in four survey areas. Precaution was taken to be fair in all statements and not to be biased or let preconceived ideas about the problem influence the conclusions drawn

Analysis and Modeling of Polysilicon Critical Dimensions

The objective of the project was to evaluate polysilicon CD etch bias as it was affected by etch time, measurement feature within the exposure field, and the product being measured. Characterization of the trends has generated a summarizing model that uses etch time, measurement feature, and product type as significant factors in determining polysilicon CD etch bias. Etch time was found to be a continuous factor, while measurement site and product type were categorical. Measurement Site A was found to be independent of measurement Site B, independent of Site C, etc., as well as Product 1 was found independent of Product 2. When tested with JMP Statistical software the generated model produced an R-sq Adj. =0.96. A comparison of modeled values with collected data for several different combinations of conditions showed a maximum difference of 6% on a normalized scale