Estimation of Noisy Cost Functions by Conventional and Adjusted Simulated Annealing Techniques

L'algorithme de recuit simulé est largement utilisé dans la communauté d'optimisation pour résoudre divers types de problèmes, discrets et continus. L'objectif de cette thèse est d'analyser le recuit simulé dans des environnements déterministes et stochastiques pour des problèmes discrets. Les objectifs précis sont de classer des problèmes clés, d'offrir des suggestions et des recommandations à suivre en utilisant l'algorithme de recuit simulé et de recuit simulé sous bruit. Plus spécifiquement, des problèmes apparaissent en optimisation en présence de bruit, et sur la manière de le contrôler. Nous proposons la méthode de recuit simulé bruité (NSA: Noisy Simulated Annealing), basée sur la modification de l'algorithme de Metropolis-Hastings présentée par Ceperlay and Dewing, qui surpasse les techniques de recuit simulé analogues, délivrant des solutions numériques similaires, à coût réduit. Nous considérons les principales approches qui traitent le bruit dans le cadre du recuit simulé afin d'en extraire leurs attributs distinctifs et de produire une comparaison plus pertinente. Nous évaluons ensuite les performances numériques de l'approche sur des instances du problème du voyageur de commerce. Les résultats obtenus montrent un clair avantage pour le recuit simulé bruité, en présence de bruit.The Simulated Annealing (SA) algorithm is extensively used in the optimization community for solving various kinds of problems, discrete and continuous. This thesis aims to analyze SA in both deterministic and stochastic environments for discrete problems. Precise objectives are to classify key problems, offer suggestions and recommendations to be undertaken by using SA and Simulated Annealing Under Noise (SAUN). More specifically, problems appear in optimization due to the existence of noise when evaluating the objective function, and how to control this noise. We propose a method, called Noisy Simulated Annealing (NSA), based on the Metropolis-Hasting algorithm modification presented by Ceperlay and Dewing, that outperforms analogous SA techniques, delivering similar numerical solutions, at a reduced cost. We consider the main approaches in the SA setting that handle noise in order to extract their distinctive attributes and make the comparison more relevant. We next assess the numerical performance of the approach on traveling salesman problem instances. The outcomes of our tests show a clear advantage for NSA when solving different problems to get high-quality solutions in presence of noise

Parametric Uncertainty in Numerical Weather Prediction Models

Numerical Weather Prediction (NWP) models form the basis of weather forecasting. The accuracy of model forecasts can be enhanced by providing a more accurate initial state for the model, and by improving the model representation of relevant atmospheric processes. Modelling of subgrid-scale physical processes causes additional uncertainty in the forecasts since, for example, the rates at which parts of the physical processes occur are not exactly known. The efficiency of these sub-processes in the models is controlled via so called closure parameters. This thesis is motivated by a practical need to estimate the values of these closure parameters objectively, and to assess the uncertainties related to them. In this thesis the Ensemble Prediction and Parameter Estimation System (EPPES) is utilised to determine the optimal closure parameter values, and to learn about their uncertainties. Closure parameters related to convective processes, formation of convective rain and stratiform clouds are studied in two atmospheric General Circulation Models (GCM): the Integrated Forecasting System (IFS) of the European Centre for Medium-Range Weather Forecasts (ECMWF) and the ECMWF model HAMburg version (ECHAM5). The parameter estimation is conducted by launching ensembles of medium range forecasts with initial time parameter variations. The fit of each ensemble member to analyses is then evaluated with respect to a target criterion, and the likelihoods of the forecasts are discerned. The target criterion is first set to be 500 hPa level geopotential height Mean Squared Error (MSE) at forecast days three and ten. After the proof of concept experimentations, the use of total energy norm as the target criterion is explored. EPPES estimation with both likelihoods results in parameter values converging to more optimal values during a three-month sampling period. The improved forecast accuracy of the models with the new parameter values are verified through headline skill scores (Root Mean Square Error (RMSE) and Anomaly Correlation Coefficient (ACC)) of 500 hPa geopotential height and a scorecard consisting of multiple model fields. The sampling process also provides information about parameter uncertainties. Three uses for the uncertainty data are highlighted: (i) parametrization deficiencies can be identified from large parameter uncertainties, (ii) parameter correlations can indicate a need for the coupling of parameters, and (iii) adding parameter variations into an ensemble prediction system (EPS) can be used to increase the ensemble spread. The relationship between medium range forecasts and model climatology is explored, too. Closure parameter modification induced cloud cover changes at forecast day three carry over to the very long range forecasts as well. This link could be used to improve model climatology by enhancing the computationally cheaper medium range forecast skill of the model.Jokapäiväisten sääennusteiden pohjana ovat numeeristen sääennustemallien tuottamat ennusteet ilmakehän tulevasta tilasta. Mallien ennustetarkkuutta voidaan parantaa tarkentamalla mallille syötettävää ilmakehän alkutilaa tai mallintamalla ilmakehän ilmiöt realistisemmin. Hilaväliä pienempien ilmiöiden kuvaaminen malleissa tuottaa ennusteisiin oman epävarmuutensa, mm. koska näihin ilmiöihin liittyvien prosessien tehokkuutta ei tiedetä tarkasti. Malleissa näiden aliprosessien nopeutta säädelläään ns. sulkuparametrien kautta. Tämän väitöskirjan tavoitteena on sulkuparametrien arvojen objektiivinen valinta sekä niihin liittyvien epävarmuuksien selvittäminen. Tässä väitöskirjassa parametrien optimaalisten arvojen ja niiden epävarmuuksien estimointi suoritetaan EPPES (Ensemble Prediction and Parameter Estimation System) -algoritmilla. Konvektioon, konvektiiviseen sateeseen ja kerrospilvien muodostumiseen liittyviä parametrejä tutkitaan kahdella globaalilla ilmakehämallilla: Euroopan keskipitkien sääennusteiden keskuksen (ECMWF) IFS (Integrated Forecasting System) -sääennustemallilla ja ECHAM5 (ECMWF model HAMburg version) -ilmastomallilla. Parametrien estimointia varten niiden arvoja muunnellaan keskipitkien sääennusteiden ryväsennustejärjestelmässä. Jokaisen ryppään jäsenen ennustetta verrataan analyysikenttään ja ennusteen osuvuus mitataan ennalta määrätyllä kohdefunktiolla. Kohdefunktiona käytetään ensimmäiseksi 500 hPa painepinnan geopotentiaalikorkeuden MS (Mean Squared) -virhettä kolmen ja 10 päivän sääennusteissa ja EPPES-algoritmin todetaan toimivan halutulla tavalla. Tämän jälkeen kohdefunktioksi vaihdetaan ilmakehän kokonaisenergianormi. Kolmen kuukauden otannoissa kummatkin käytetyt kohdefunktiot johtavat parametrien konvergoitumiseen optimoituihin arvoihin. Uusien parametriarvojen todetaan parantavan enuusteita käyttäen validointimenetelminä 500 hPa painepinnan geopotentiaalikorkeudella RMSE (Root Mean Squared Error) ja ACC (Anomaly Correlation Coefficient) arvoja sekä laajoja mallien vertailutaulukoita. Estimoinnin aikana saadaan myös lisää tieto parametreihin liittyvistä epävarmuuksista. Kolme käyttötarkoitusta nostetaan esiin: (i) suuret epävarmuudet parametreissä viittaavat puutteisiin parametrisaatioissa, (ii) voimakkaat parametrien korrelaatiot ilmaisevat tarpeesta parametrien yhdistämiseksi ja (iii) parametrivariaatioiden lisääminen ryväsennustejärjestelmään kasvattaa järjestelmän ryväshajontaa. Viimeiseksi selvitetään yhteyttä keskipitkien ennusteiden ja mallin klimatologian välillä. Parametrien vaihtamisen aiheuttamien pilvisyyden muutosten rakenne kolmen päivän ennusteissa on havaittavissa myös mallin pitkissä vuosittaisennusteissa. Näin ollen mallin klimatologiaa voisi parantaa myös tarkentamalla mallin ennustuskykyä laskennallisesti halvemmissa keskipitkissä sääennusteissa

The impact of modifying attentional bias on vulnerability to pain

The preferential deployment of attention to noxious versus benign information in the internal and external environment - “attentional bias” - is thought to confer vulnerability to pain. The current thesis tested this putative mechanism by modifying the bias using the visual-probe task (attentional bias modification; ABM) and examining effects of this experimental manipulation on attentional bias and critical pain outcomes. Drawing on recent evidence that the impact of pain on attentional bias varies across its temporal components, this thesis additionally tested the component stages of attentional bias implicated in pain experience by manipulating the duration for which visual-probe stimuli were presented. Study 1 confirmed that both rapid and slower attentional orienting was biased in individuals with persistent musculoskeletal pain. Results from Studies 2 and 3 indicated that acute experimentally-induced pain modified the faster bias and that participants whose fast bias was modified had reduced vulnerability to cold pressor pain, in comparison with control participants. This suggested that mechanisms of initial orienting were more active in the acute pain experience. Studies 4 and 5 revealed that concurrently retraining fast and slower bias was optimal for persistent musculoskeletal pain. Results of a systematic review and meta-analysis indicated a small overall statistical effect of ABM on pain severity. Critically, however, whereas ABM had been effective at reducing acute pain severity, this was not the case for persistent pain. Overall, these findings suggest that the faster bias influenced vulnerability to acute pain, indicating a potential therapeutic target for future research. However, retraining the earlier stage of attention alone did not influence persistent pain outcomes, where there appeared to be greater involvement of the slower bias. It was concluded that not only could attentional bias influence critical pain outcomes, but that the optimal timings may vary across temporal pain classifications

Mathematical modelling and numerical simulation of physical cloud processes in a wide range of spatiotemporal scales

The mathematical modelling and numerical simulation of clouds and climate include numerous phenomena that are tough nuts to crack as they cover a wide range of spatiotemporal scales. In many ways, time is a vital factor, for instance, predicting the significance of a millisecond phenomenon for the future century is a major undertaking. Additionally, the computational time required by numerical models is a challenge. Luckily, we have a fine set of tools in our mathematical backpack. Here, we explore how a detailed cloud model can be improved to simulate the interactions with ice crystals. A new ice microphysics module is validated against a set of similar cloud models. Further on, the cloud model is shown to be an improvement over the previous generation of cloud models as it incorporates detailed aerosol-cloud interactions, which in our study is shown to impact cloud lifetime through ice nuclei recycling and marine ice nuclei import via updrafts. Additionally, the cloud model, which has a fine resolution in the order of meters, is harnessed to develop three different emulators to represent selected cloud processes in an improved detailed way. Emulators can be called also parametrisation or a machine learning model. Further on, created parameterisations are implemented within a global climate model, which has a much coarser resolution in the order of 10–100 kilometres. The implementation enables more precise climate simulations by having a more detailed subgrid scale description of cloud processes. As an adventurous side quest, we elaborate on how the proof-of-concept emulator could be embellished by showing an optimised way of creating the design of the simulation experiment in our applied case and we compare our results with the proof-of-concept method used in the study where the emulators were created

Equilibration of deep neural networks and carrier chirality in Rashba systems

This thesis reports results of studies conducted on the equilibration of two systems and consists of two parts: the first part deals with the optimisation of deep neural networks, whereas the second part with the decay of non-equilibrium states in strongly Rashba-coupled systems at low temperature. Deep learning is a conceptually simple, highly effective, and widely used tool, yet there remains insufficient understanding for why it works. The optimisation of deep neural networks with common algorithms such as stochastic gradient descent performs unexpectedly well given the complexity of the underlying high-dimensional non-convex minimisation problem. The first part of this thesis therefore looks at the optimisation procedure from the perspective of statistical physics. This allows us to interpret the loss function landscape of deep neural networks as the counterpart of the potential energy landscape in molecular systems and the optimisation of the network as its equilibration dynamics. Using landscape exploration tools developed in theoretical chemistry, we resolve the structure of the loss function landscape, from which we can draw conclusions for the relaxational dynamics of typical optimisers and, consequently, for deep learning. The second part investigates how a non-equilibrium charge-carrier chirality distribution in a clean, strongly Rashba-coupled system at low temperatures decays over time. We first motivate this analysis based on experimental studies of transport properties in Rashba materials at low temperatures and subject to external magnetic fields. We investigate whether chirality imbalances could serve as the source for those experimental observations and develop a framework that models the behaviour of such a system. We then proceed with a more general theoretical study of the equilibration mechanisms of chirality in low-temperature strongly Rashba-coupled systems and compute the relaxation timescales of those mechanisms.This thesis is the outcome of doctoral studies conducted at the University of Cambridge with the financial support of the Engineering and Physical Sciences Research Council of the UK

Aperfeiçoamento do algoritmo de otimização híbrido Pincus-Nelder e Mead para detecção de dano em estruturas a partir de dados vibracionais

Dissertação (mestrado) - Universidade Federal de Santa Catarina, Centro Tecnológico, Programa de Pós-Graduação em Engenharia Civil, Florianópolis, 2014.Neste estudo, uma proposta de aperfeiçoamento do algoritmo deotimização híbrido estocástico-determinístico Pincus - Nelder e Mead(P-NMA) para detecção de dano em estruturas a partir de dadosvibracionais é apresentada. A formulação da função objetivo para oproblema de minimização é estabelecido pela diferença entre parâmetrosdinâmicos experimentais da estrutura na condição com dano e aquelescalculados utilizando um modelo de elementos finitos (MEF) querepresenta tal condição. Uma estratégia que permite acelerar aconvergência do algoritmo híbrido P-NMA Original para o problema deidentificação de dano é desenvolvida, sendo o algoritmo resultante aquidesignado por P-NMA Modificado. Para se poder ter uma base deescolha de parâmetros envolvidos na parte estocástica do algoritmo,funções testes de otimização global foram utilizadas. Em seguida, cincoexemplos numéricos de identificação de dano, todos retirados daliteratura, nomeadamente, uma viga simplesmente apoiada, uma viga embalanço, duas treliças planas e um pórtico plano com vários cenários dedano são apresentados. Os parâmetros dinâmicos utilizados nestetrabalho (frequências naturais e formas modais) são obtidos através dasolução do problema de autovalores. Para simular as características dainformação obtida por meio de um ensaio dinâmico real, foi consideradaa existência de apenas poucos modos de baixa frequência e também foianalisada a influência do ruído nas medições, que inclui igualmente aimprecisão na coleta de dados. Para testar a precisão e eficiência doalgoritmo resultante do aperfeiçoamento/modificações implementadas(o algoritmo P-NMA Modificado), uma comparação dos resultadosobtidos por meio deste novo algoritmo proposto e do algoritmo P-NMAOriginal é realizada. Além disso, os resultados do último exemplonumérico (pórtico plano) são comparados com aqueles existentes naliteratura, na qual o algoritmo P-NMA Original e o algoritmo genético(AG) foram utilizados. Em todos os casos analisados, as modificaçõesimplementadas funcionaram bem e os resultados foram promissores, oalgoritmo híbrido P-NMA Modificado foi mais preciso e apresentoumenor custo computacional comparativamente ao algoritmo híbrido PNMAOriginal e AG.Abstract : In this study, an improvement in the hybrid stochastic-deterministicoptimization algorithm Pincus-Nelder and Mead (P-NMA) for vibrationbaseddamage detection in structures is proposed. The formulation of theobjective function for the minimization problem is established by thedifference between experimental dynamic parameters of the damagedstructure and those calculated using a finite element model (FEM) thatrepresents such condition. A strategy which allows a quick convergenceof the Original hybrid algorithm P-NMA for the damage identificationproblem is developed, and the resulting algorithm is denominated hereas Modified P-NMA. In order to have a basis for the choice ofparameters involved in the stochastic part of the hybrid algorithm, testfunctions for global optimization were used. Thereafter, five numericalexamples, all taken from literature, namely, a simply supported beam, acantilever beam, two plane trusses and a portal plane frame with severaldamage scenarios are presented. The dynamic parameters used in thiswork (natural frequencies and mode shapes) are obtained by solving theeigenvalue problem. To simulate the characteristics of informationobtained in a real dynamic test, only a few modes of low frequency wereconsidered and the influence of noise in the measurements, whichinclude the errors during data collection was also analyzed. To test theaccuracy and efficiency of the resultant algorithm from theimprovement/ implemented modifications (the Modified hybrid PNMA),a comparison of the results obtained from this new proposedalgorithm and the Original P-NMA algorithm is performed. In addition,the results of the last numerical example (portal plane frame) arecompared with those available in the literature, where the Original PNMAalgorithm and the genetic algorithm (GA) were used. In allanalyzed cases, the implemented improvements were satisfactory andthe results were promising, the Modified hybrid P-NMA algorithm wasmore accurate and had lower computational cost compared to theOriginal hybrid P-NMA algorithm and GA

Permutation Entropy and Bubble Entropy: Possible interactions and synergies between order and sorting relations

[EN] Despite its widely demonstrated usefulness, there is still room for improvement in the basic Permutation Entropy (PE) algorithm, as several subsequent studies have proposed in the recent years. For example, some improved PE variants try to address possible PE weaknesses, such as its only focus on ordinal information, and not on amplitude, or the possible detrimental impact of equal values in subsequences due to motif ambiguity. Other evolved PE methods try to reduce the influence of input parameters. A good representative of this last point is the Bubble Entropy (BE) method. BE is based on sorting relations instead of ordinal patterns, and its promising capabilities have not been extensively assessed yet. The objective of the present study was to comparatively assess the classification performance of this new method, and study and exploit the possible synergies between PE and BE. The claimed superior performance of BE over PE was first evaluated by conducting a series of time series classification tests over a varied and diverse experimental set. The results of this assessment apparently suggested that there is a complementary relationship between PE and BE, instead of a superior/inferior relationship. A second set of experiments using PE and BE simultaneously as the input features of a clustering algorithm, demonstrated that with a proper algorithm configuration, classification accuracy and robustness can benefit from both measures.Cuesta Frau, D.; Vargas-Rojo, B. (2020). Permutation Entropy and Bubble Entropy: Possible interactions and synergies between order and sorting relations. Mathematical Biosciences and Engineering. 17(2):1637-1658. https://doi.org/10.3934/mbe.2020086S163716581721. C. Bandt and B. Pompe, Permutation entropy: A natural complexity measure for time series, Phys. Rev. Lett., 88 (2002), 174102.2. M. Zanin, L. Zunino, O. A. Rosso and D. Papo, Permutation entropy and its main biomedical and econophysics applications: A review, Entropy, 14 (2012), 1553-1577.14. F. Siokis, Credit market jitters in the course of the financial crisis: A permutation entropy approach in measuring informational efficiency in financial assets, Phys. A Statist. Mechan. Appl., 499 (2018).15. A. F. Bariviera, L. Zunino, M. B. Guercio, L. Martinez and O. Rosso, Efficiency and credit ratings: A permutation-information-theory analysis, J. Statist. Mechan. Theory Exper., 2013 (2013), P08007.16. A. F. Bariviera, M. B. Guercio, L. Martinez and O. Rosso, A permutation information theory tour through different interest rate maturities: the libor case, Philos. Transact. Royal Soc. A Math. Phys. Eng. Sci., 373 (2015).20. B. Fadlallah, B. Chen, A. Keil and J. Príncipe, Weighted-permutation entropy: A complexity measure for time series incorporating amplitude information, Phys. Rev. E, 87 (2013), 022911.Deng, B., Cai, L., Li, S., Wang, R., Yu, H., Chen, Y., & Wang, J. (2016). Multivariate multi-scale weighted permutation entropy analysis of EEG complexity for Alzheimer’s disease. Cognitive Neurodynamics, 11(3), 217-231. doi:10.1007/s11571-016-9418-924. D. Cuesta-Frau, Permutation entropy: Influence of amplitude information on time series classification performance, Math. Biosci. Eng., 5 (2019), 1-16.25. F. Traversaro, M. Risk, O. Rosso and F. Redelico, An empirical evaluation of alternative methods of estimation for Permutation Entropy in time series with tied values, arXiv e-prints, arXiv:1707.01517 (2017).26. D. Cuesta-Frau, M. Varela-Entrecanales, A. Molina-Picó and B. Vargas, Patterns with equal values in permutation entropy: Do they really matter for biosignal classification?, Complexity, 2018 (2018), 1-15.29. D. Cuesta-Frau, A. Molina-Picó, B. Vargas and P. González, Permutation entropy: Enhancing discriminating power by using relative frequencies vector of ordinal patterns instead of their shannon entropy, Entropy, 21 (2019).30. H. Azami and J. Escudero, Amplitude-aware permutation entropy: Illustration in spike detection and signal segmentation, Comput. Meth. Program. Biomed., 128 (2016), 40-51.32. G. Manis, M. Aktaruzzaman and R. Sassi, Bubble entropy: An entropy almost free of parameters, IEEE Transact. Biomed. Eng., 64 (2017), 2711-2718.34. L. Zunino, F. Olivares, F. Scholkmann and O. A. Rosso, Permutation entropy based time series analysis: Equalities in the input signal can lead to false conclusions, Phys. Lett. A, 381 (2017), 1883-1892.38. D. E. Lake, J. S. Richman, M. P. Griffin and J. R. Moorman, Sample entropy analysis of neonatal heart rate variability, Am. J. Physiology-Regulatory Integrat. Comparat. Physiol., 283 (2002), R789-R797, PMID: 12185014.41. I. Unal, Defining an Optimal Cut-Point Value in ROC Analysis: An Alternative Approach, Comput. Math. Methods Med., 2017 (2017), 14.47. A. K. Jain, M. N. Murty and P. J. Flynn, Data clustering: A review, ACM Comput. Surv., 31 (1999), 264-323.51. J. Sander, M. Ester, H.-P. Kriegel and X. Xu, Density-based clustering in spatial databases: The algorithm gdbscan and its applications, Data Min. Knowl. Discov., 2 (1998), 169-194.52. J. Wu, Advances in K-means Clustering: A Data Mining Thinking, Springer Publishing Company, Incorporated, 2012.53. S. Panda, S. Sahu, P. Jena and S. Chattopadhyay, Comparing fuzzy-c means and k-means clustering techniques: A comprehensive study, in Advances in Computer Science, Engineering & Applications (eds. D. C. Wyld, J. Zizka and D. Nagamalai), Springer Berlin Heidelberg, Berlin, Heidelberg, 2012, 451-460.54. A. L. Goldberger, L. A. N. Amaral, L. Glass, J. M. Hausdorff, P. C. Ivanov, R. G. Mark, et al., PhysioBank, PhysioToolkit, and PhysioNet: Components of a new research resource for complex physiologic signals, Circulation, 101 (2000), 215-220.58. R. G. Andrzejak, K. Lehnertz, F. Mormann, C. Rieke, P. David and C. E. Elger, Indications of nonlinear deterministic and finite-dimensional structures in time series of brain electrical activity: Dependence on recording region and brain state, Phys. Rev. E, 64 (2001), 061907.60. N. Iyengar, C. K. Peng, R. Morin, A. L. Goldberger and L. A. Lipsitz, Age-related alterations in the fractal scaling of cardiac interbeat interval dynamics, Am. J. Physiology-Regulatory Integrat. Comparat. Physiol., 271 (1996), R1078-R1084, PMID: 8898003

A review of modelling and verification approaches for computational biology

This paper reviews most frequently used computational modelling approaches and formal verification techniques in computational biology. The paper also compares a number of model checking tools and software suits used in analysing biological systems and biochemical networks and verifiying a wide range of biological properties