Neural Network Exploration Using Optimal Experiment Design

We consider the question "How should one act when the only goal is to learn as much as possible?" Building on the theoretical results of Fedorov [1972] and MacKay [1992], we apply techniques from Optimal Experiment Design (OED) to guide the query/action selection of a neural network learner. We demonstrate that these techniques allow the learner to minimize its generalization error by exploring its domain efficiently and completely. We conclude that, while not a panacea, OED-based query/action has much to offer, especially in domains where its high computational costs can be tolerated

Application of response surface methodology to stiffened panel optimization

In a multilevel optimization frame, the use of surrogate models to approximate optimization constraints allows great time saving. Among available metamodelling techniques we chose to use Neural Networks to perform regression of static mechanical criteria, namely buckling and collapse reserve factors of a stiffened panel, which are constraints of our subsystem optimization problem. Due to the highly non linear behaviour of these functions with respect to loading and design variables, we encountered some difficulties to obtain an approximation of sufficient quality on the whole design space. In particular, variations of the approximated function can be very different according to the value of loading variables. We show how a prior knowledge of the influence of the variables allows us to build an efficient Mixture of Expert model, leading to a good approximation of constraints. Optimization benchmark processes are computed to measure time saving, effects on optimum feasibility and objective value due to the use of the surrogate models as constraints. Finally we see that, while efficient, this mixture of expert model could be still improved by some additional learning techniques

Active Learning with Statistical Models

For many types of machine learning algorithms, one can compute the statistically `optimal' way to select training data. In this paper, we review how optimal data selection techniques have been used with feedforward neural networks. We then show how the same principles may be used to select data for two alternative, statistically-based learning architectures: mixtures of Gaussians and locally weighted regression. While the techniques for neural networks are computationally expensive and approximate, the techniques for mixtures of Gaussians and locally weighted regression are both efficient and accurate. Empirically, we observe that the optimality criterion sharply decreases the number of training examples the learner needs in order to achieve good performance.Comment: See http://www.jair.org/ for any accompanying file

Distilled Sensing: Adaptive Sampling for Sparse Detection and Estimation

Adaptive sampling results in dramatic improvements in the recovery of sparse signals in white Gaussian noise. A sequential adaptive sampling-and-refinement procedure called Distilled Sensing (DS) is proposed and analyzed. DS is a form of multi-stage experimental design and testing. Because of the adaptive nature of the data collection, DS can detect and localize far weaker signals than possible from non-adaptive measurements. In particular, reliable detection and localization (support estimation) using non-adaptive samples is possible only if the signal amplitudes grow logarithmically with the problem dimension. Here it is shown that using adaptive sampling, reliable detection is possible provided the amplitude exceeds a constant, and localization is possible when the amplitude exceeds any arbitrarily slowly growing function of the dimension.Comment: 23 pages, 2 figures. Revision includes minor clarifications, along with more illustrative experimental results (cf. Figure 2

Machine Learning Driven Design Of Experiments For Predictive Models In Production Systems

Machine learning (ML) describes the ability of algorithms to structure and interpret data independently or to learn correlations. The use of ML is steadily increasing in companies of all sizes. However, insufficient market readiness of many ML solutions inhibits their application, especially in production systems. Predictive models apply ML to understand the complex behavior of a system through regression from operational data. This enables determining the relationship between factors and target variables. Accurate predictions of these models for production systems are essential for their application, as even minor variations can significantly affect the process. This accuracy depends on the available data to train the ML model. Production data usually shows a high epistemic uncertainty, leading to inaccurate predictions unfit for real-world applications. This paper presents ML-driven, data-centric Design of Experiments (DoE) to create a process-specific dataset with low epistemic uncertainty. This leads to improved accuracy of the predictive models, ultimately making them feasible for production systems. Our approach focuses on determining epistemic uncertainty in historical data of a production system to find data points of high value to the ML model in the factor space. To identify an efficient set of experiments, we cluster these data points weighted by feature importance. We evaluate the model by running these experiments and using the collected data for further training of a prediction model. Our approach achieves a significantly higher increase in accuracy compared to continuing the training of the prediction model with the same amount of regular operating data

InfoMax Bayesian learning of the Furuta pendulum

We have studied the InfoMax (D-optimality) learning for the two-link Furuta pendulum. We compared InfoMax and random learning methods. The InfoMax learning method won by a large margin, it visited a larger domain and provided better approximation during the same time interval. The advantages and the limitations of the InfoMax solution are treated