Data-driven Discovery of Multiple-Physics Electromagnetic Partial Differential Equations

The subject of data-driven discovery for equations has developed rapidly in recent years, especially in the field of finding equations of unknown forms, which provides new ideas for the study of complex systems. When there are unknown noise sources and other uncertain factors in the system, it is quite difficult to directly derive the system governing equation, because the equation is complicated and the calculation cost is large. But if we try to find the equation directly from the data, it will be helpful to improve these problems. For the data in nonlinear multi-physics electromagnetic system, the deep learning method can be used to find the equation, which can obtain the governing equation form accurately and has high time efficiency and parameter precision. This thesis studies the algorithm of data-driven discovery equations in electromagnetic multiple physics problems and realizes the inversion of Maxwell's multiple physics equations. Firstly, three methods of data-driven equation discovery are introduced, including symbol regression, sparse regression and neural network. Secondly, an algorithm based on sparse regression and convolutional neural network is proposed for multiple physics equations of Maxwell equations. This algorithm uses Euler method to approximate time differentiation and convolution kernel to compute space differentiation. At the same time, in the training process, the pareto analysis method was used to remove the redundancy. Then, the model algorithm is applied to the multi-physics coupling simulation data of electromagnetic plasma, and the homogeneous and non-homogeneous equations of electromagnetic propagation are realized by using less time and space observation field samples, which has certain anti-noise performance. For the problem of propagation in uniform medium, the influence of spatial and temporal sampling method on the inversion precision of equation coefficients is studied. Under the condition of inhomogeneous media propagation, this thesis finds the changing law of inhomogeneous coefficient by changing the weight scale of neural network, aiming at the problem that the equation coefficient varies with the spatial scale. By using the properties of trigonometric series and some prior knowledge, the expression of the coefficient of inhomogeneous terms is approximated, and satisfactory results are obtained. Finally, the thesis summarizes the proposed method and its main conclusion. In both homogeneous and inhomogeneous media, the model has good performance. Meanwhile, the author discusses the possible improvement methods for other problems and the idea that the structure of the model can be adjusted in a small range in the future and applied to the high-dimensional space and the problems with high-order spatial differentiation in the governing equations

Support Vector Machine Implementations for Classification & Clustering

BACKGROUND: We describe Support Vector Machine (SVM) applications to classification and clustering of channel current data. SVMs are variational-calculus based methods that are constrained to have structural risk minimization (SRM), i.e., they provide noise tolerant solutions for pattern recognition. The SVM approach encapsulates a significant amount of model-fitting information in the choice of its kernel. In work thus far, novel, information-theoretic, kernels have been successfully employed for notably better performance over standard kernels. Currently there are two approaches for implementing multiclass SVMs. One is called external multi-class that arranges several binary classifiers as a decision tree such that they perform a single-class decision making function, with each leaf corresponding to a unique class. The second approach, namely internal-multiclass, involves solving a single optimization problem corresponding to the entire data set (with multiple hyperplanes). RESULTS: Each SVM approach encapsulates a significant amount of model-fitting information in its choice of kernel. In work thus far, novel, information-theoretic, kernels were successfully employed for notably better performance over standard kernels. Two SVM approaches to multiclass discrimination are described: (1) internal multiclass (with a single optimization), and (2) external multiclass (using an optimized decision tree). We describe benefits of the internal-SVM approach, along with further refinements to the internal-multiclass SVM algorithms that offer significant improvement in training time without sacrificing accuracy. In situations where the data isn't clearly separable, making for poor discrimination, signal clustering is used to provide robust and useful information – to this end, novel, SVM-based clustering methods are also described. As with the classification, there are Internal and External SVM Clustering algorithms, both of which are briefly described

Bayesian Approaches For Image Restoration

Ph.DDOCTOR OF PHILOSOPH

Model combination by decomposition and aggregation

Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Nuclear Engineering, 2004.Includes bibliographical references (p. 265-282).This thesis focuses on a general problem in statistical modeling, namely model combination. It proposes a novel feature-based model combination method to improve model accuracy and reduce model uncertainty. In this method, a set of candidate models are first decomposed into a group of components or features and then components are selected and aggregated into a composite model based on data. However, in implementing this new method, some central challenges have to be addressed, which include candidate model choice, component selection, data noise modeling, model uncertainty reduction and model locality. In order to solve these problems, some new methods are put forward. In choosing candidate models, some criteria are proposed including accuracy, diversity, independence as well as completeness and then corresponding quantitative measures are designed to quantify these criteria, and finally an overall preference score is generated for each model in the pool. Principal component analysis (PCA) and independent component analysis (ICA) are applied to decompose candidate models into components and multiple linear regression is employed to aggregate components into a composite model.(cont.) In order to reduce model structure uncertainty, a new concept of fuzzy variable selection is introduced to carry out component selection, which is able to combine the interpretability of classical variable selection and the stability of shrinkage estimators. In dealing with parameter estimation uncertainty, exponential power distribution is proposed to model unknown non-Gaussian noise and parametric weighted least-squares method is devise to estimate parameters in the context of non-Gaussian noise. These two methods are combined to work together to reduce model uncertainty, including both model structure uncertainty and parameter uncertainty. To handle model locality, i.e. candidate models do not work equally well over different regions, the adaptive fuzzy mixture of local ICA models is developed. Basically, it splits the entire input space into domains, build local ICA models within each sub-region and then combine them into a mixture model. Many different experiments are carried out to demonstrate the performance of this novel method. Our simulation study and comparison show that this new method meets our goals and outperforms existing methods in most situations.by Mingyang Xu.Ph.D

Feed forward neural networks and genetic algorithms for automated financial time series modelling

This thesis presents an automated system for financial time series modelling. Formal and applied methods are investigated for combining feed-forward Neural Networks and Genetic Algorithms (GAs) into a single adaptive/learning system for automated time series forecasting. Four important research contributions arise from this investigation: i) novel forms of GAs are introduced which are designed to counter the representational bias associated with the conventional Holland GA, ii) an experimental methodology for validating neural network architecture design strategies is introduced, iii) a new method for network pruning is introduced, and iv) an automated method for inferring network complexity for a given learning task is devised. These methods provide a general-purpose applied methodology for developing neural network applications and are tested in the construction of an automated system for financial time series modelling. Traditional economic theory has held that financial price series are random. The lack of a priori models on which to base a computational solution for financial modelling provides one of the hardest tests of adaptive system technology. It is shown that the system developed in this thesis isolates a deterministic signal within a Gilt Futures prices series, to a confidences level of over 99%, yielding a prediction accuracy of over 60% on a single run of 1000 out-of-sample experiments. An important research issue in the use of feed-forward neural networks is the problems associated with parameterisation so as to ensure good generalisation. This thesis conducts a detailed examination of this issue. A novel demonstration of a network's ability to act as a universal functional approximator for finite data sets is given. This supplies an explicit formula for setting a network's architecture and weights in order to map a finite data set to arbitrary precision. It is shown that a network's ability to generalise is extremely sensitive to many parameter choices and that unless careful safeguards are included in the experimental procedure over-fitting can occur. This thesis concentrates on developing automated techniques so as to tackle these problems. Techniques for using GAs to parameterise neural networks are examined. It is shown that the relationship between the fitness function, the GA operators and the choice of encoding are all instrumental in determining the likely success of the GA search. To address this issue a new style of GA is introduced which uses multiple encodings in the course of a run. These are shown to out-perform the Holland GA on a range of standard test functions. Despite this innovation it is argued that the direct use of GAs to neural network parameterisation runs the risk of compounding the network sensitivity issue. Moreover, in the absence of a precise formulation of generalisation a less direct use of GAs to network parameterisation is examined. Specifically a technique, artficia1 network generation (ANG), is introduced in which a GA is used to artificially generate test learning problems for neural networks that have known network solutions. ANG provides a means for directly testing i) a neural net architecture, ii) a neural net training process, and iii) a neural net validation procedure, against generalisation. ANG is used to provide statistical evidence in favour of Occam's Razor as a neural network design principle. A new method for pruning and inferring network complexity for a given learning problem is introduced. Network Regression Pruning (NRP) is a network pruning method that attempts to derive an optimal network architecture by starting from what is considered an overly large network. NRP differs radically from conventional pruning methods in that it attempts to hold a trained network's mapping fixed as pruning proceeds. NRP is shown to be extremely successful at isolating optimal network architectures on a range of test problems generated using ANG. Finally, NRP and techniques validated using ANG are combined to implement an Automated Neural network Time series Analysis System (ANTAS). ANTAS is applied to the gilt futures price series The Long Gilt Futures Contract (LGFC)

Probabilistic Neural Networks for Special Tasks in Electromagnetics

Tato práce pojednává o technikách behaviorálního modelování pro speciální úlohy v elektromagnetismu, které je možno formulovat jako problém aproximace, klasifikace, odhadu hustoty pravděpodobnosti nebo kombinatorické optimalizace. Zkoumané methody se dotýkají dvou základních problémů ze strojového učení a combinatorické optimalizace: ”bias vs. variance dilema” a NP výpočetní komplexity. Boltzmanův stroj je v práci navržen ke zjednodušování komplexních impedančních sítí. Bayesovský přístup ke strojovému učení je upraven pro regularizaci Parzenova okna se snahou o vytvoření obecného kritéria pro regularizaci pravděpodobnostní a regresní neuronové sítě.The thesis deals with behavioural modelling techniques capable solving special tasks in electromagnetics which can be formulated as approximation, classification, probability estimation, and combinatorial optimization problems. Concept of the work lies in applying a probabilistic approach to behavioural modelling. Examined methods address two general problems in machine learning and combinatorial optimization: ”bias vs. variance dilemma” and NP computational complexity. The Boltzmann machine is employed to simplify a complex impedance network. The Parzen window is regularized using the Bayesian strategy for obtaining a model selection criterion for probabilistic and general regression neural networks.

Modelling discrepancy in Bayesian calibration of reservoir models

Simulation models of physical systems such as oil field reservoirs are subject to numerous uncertainties such as observation errors and inaccurate initial and boundary conditions. However, after accounting for these uncertainties, it is usually observed that the mismatch between the simulator output and the observations remains and the model is still inadequate. This incapability of computer models to reproduce the real-life processes is referred to as model inadequacy. This thesis presents a comprehensive framework for modelling discrepancy in the Bayesian calibration and probabilistic forecasting of reservoir models. The framework efficiently implements data-driven approaches to handle uncertainty caused by ignoring the modelling discrepancy in reservoir predictions using two major hierarchical strategies, parametric and non-parametric hierarchical models. The central focus of this thesis is on an appropriate way of modelling discrepancy and the importance of the model selection in controlling overfitting rather than different solutions to different noise models. The thesis employs a model selection code to obtain the best candidate solutions to the form of non-parametric error models. This enables us to, first, interpolate the error in history period and, second, propagate it towards unseen data (i.e. error generalisation). The error models constructed by inferring parameters of selected models can predict the response variable (e.g. oil rate) at any point in input space (e.g. time) with corresponding generalisation uncertainty. In the real field applications, the error models reliably track down the uncertainty regardless of the type of the sampling method and achieve a better model prediction score compared to the models that ignore discrepancy. All the case studies confirm the enhancement of field variables prediction when the discrepancy is modelled. As for the model parameters, hierarchical error models render less global bias concerning the reference case. However, in the considered case studies, the evidence for better prediction of each of the model parameters by error modelling is inconclusive

New Information in Naturalistic Data Is Also Signalled by Pitch Movement: An Analysis from Monolingual English/Spanish and Bilingual Spanish Speakers

New Information in Naturalistic Data Is Also Signalled by Pitch Movement:  An Analysis from Monolingual English/Spanish and Bilingual Spanish SpeakersIn communication, speakers and listeners need ways to highlight certain information and relegate other information to the background. They also need to keep track of what information they (think they) have already communicated to the listener, and of the listeners' (supposed) knowledge of topics and referents. This knowledge and its layout in the utterance is commonly referred to as information structure, i.e., the degree to which propositions and referents are given or new. All languages have 'chosen' different ways to encode such information structure, for instance by modifying the pitch or intensity of the vocal signal or the order of words in a sentence. In this study, we assess whether the use of pitch to signal new information holds in typologically different languages such as English and Spanish by analyzing three population group monolingual California English speakers, bilingual speakers of English and Spanish from California (Chicano Spanish), and monolingual Mexican Spanish speakers from Mexico City. Our study goes beyond previous work in several respects. First, most current work is based on sentences just read or elicited in response to highly standardized and often somewhat artificial stimuli whose generalizability to more naturalistic settings may be questionable. We opted instead to use semidirected interviews whose more naturalistic setting provides data with a higher degree of authenticity. Second, in order to deal with the resulting higher degree of noise in the data as well as the inherent multifactoriality of the data, we are using state-of-the-art statistical methods to explore our data, namely generalized linear mixed-effects modeling, to accommodate speaker- and lexically-specific variability. Despite the noisy data, we find that contour tones including H+L or L+H sequences signal new information, and that items encoding new information also exhibit proportionally longer stressed vowels, than those encoding given information. We also find cross-dialectal variation between monolingual Mexican Spanish speakers on the one hand and monolingual English speakers and Chicanos on the other: Mexican Spanish speakers modify pitch contours less than monolingual English speakers, whereas the English patterns affect even the Spanish pronunciation of early bilinguals. Our findings, therefore, corroborate Gussenhoven's theory (2002) that some aspects of intonation are shared cross-linguistically (longer vowel length & higher pitch for new info), whereas others are encoded language-specifically and vary even across dialects (pitch excursion & the packaging of information structure)