Efficient Optimization of Echo State Networks for Time Series Datasets

Echo State Networks (ESNs) are recurrent neural networks that only train their output layer, thereby precluding the need to backpropagate gradients through time, which leads to significant computational gains. Nevertheless, a common issue in ESNs is determining its hyperparameters, which are crucial in instantiating a well performing reservoir, but are often set manually or using heuristics. In this work we optimize the ESN hyperparameters using Bayesian optimization which, given a limited budget of function evaluations, outperforms a grid search strategy. In the context of large volumes of time series data, such as light curves in the field of astronomy, we can further reduce the optimization cost of ESNs. In particular, we wish to avoid tuning hyperparameters per individual time series as this is costly; instead, we want to find ESNs with hyperparameters that perform well not just on individual time series but rather on groups of similar time series without sacrificing predictive performance significantly. This naturally leads to a notion of clusters, where each cluster is represented by an ESN tuned to model a group of time series of similar temporal behavior. We demonstrate this approach both on synthetic datasets and real world light curves from the MACHO survey. We show that our approach results in a significant reduction in the number of ESN models required to model a whole dataset, while retaining predictive performance for the series in each cluster

Reduced Order Modelling of a Reynolds Number 10⁶ Jet Flow Using Machine Learning Approaches

The extraction of the most dynamically important coherent flow structures using reduced order models (ROM) is a challenging task in various fluid dynamics applications. In particular, for high-speed round jet flows, the axisymmetric pressure mode of interest is known to be responsible for sound radiation at small angles to the jet axis and dominant contribution to the jet noise peak. In this work the axisymmetric pressure mode of the Navier-Stokes solution of a high speed jet flow at low frequency is reconstructed from simulation data using popular Machine Learning (ML) methods, whose output can later be exploited for data-driven design of effective turbulent acoustic source models. The data used as input for the ML techniques are derived from the Large Eddy Simulation database obtained by application of the high-resolution CABARET method accelerated on GPU cards for flow solutions to NASA Small Hot Jet Acoustic Rig (SHJAR) jets. The SHJAR simulation database is fed to Spectral Proper Orthogonal (SPOD), and the resulting time coefficients of the turbulent pressure fluctuations are the targets of the three machine learning methods put to test in this work. The first Machine Learning method used is the Feed-forward Neural Networks technique, which was successfully implemented for a turbulent flow over a plunging aerofoil in the literature. The second method is based on the application of Genetic Programming, which is a symbolic regression method well-known in optimisation research, but it has not been applied for turbulent flow reconstruction before. The third method, commonly known as Echo State Networks (ESNs), is a time series prediction and reconstruction method from the field of Reservoir Computing. A report on the attempts to apply these methods for approximation and extrapolation of the turbulent flow signals are discussed