This paper presents an automatic procedure for the optimization of the hyperparameters of a delay rational model approximating the frequency-domain behavior of high-speed interconnects. The proposed model is built via a kernel-based regression, such as the Least-Square Support Vector Machine (LS-SVM), by considering an ad-hoc kernel with two hyperparameters related to the propagation delays introduced by the system. Such hyperparameters, along with the Tikhonov regularizer used by the LS-SVM regression, are carefully tuned via an automatic approach based on a k-fold cross-validation and Bayesian optimization. The feasibility of the effectiveness of the proposed modeling approach are investigated on a high-speed link

Canavero, Flavio G.

Treviso, Felipe

Trinchero, Riccardo

English

PORTO@iris (Publications Open Repository TOrino - Politecnico di Torino)

16 July 2022POLITECNICO DI TORINORepository ISTITUZIONALEBayesian Optimization of Hyperparameters in Kernel-Based Delay Rational Models / Treviso, Felipe; Trinchero,Riccardo; Canavero, Flavio G.. - In: IEEE ELECTROMAGNETIC COMPATIBILITY MAGAZINE. - ISSN 2162-2264. -ELETTRONICO. - 10:2(2021), pp. 90-93. [10.1109/MEMC.2021.9477255]OriginalBayesian Optimization of Hyperparameters in Kernel-Based Delay Rational ModelsIEEE postprint/Author's Accepted ManuscriptPublisher:PublishedDOI:10.1109/MEMC.2021.9477255Terms of use:openAccessPublisher copyright©2021 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in anycurrent or future media, including reprinting/republishing this material for advertising or promotional purposes, creatingnew collecting works, for resale or lists, or reuse of any copyrighted component of this work in other works.(Article begins on next page)This article is made available under terms and conditions as specified in the  corresponding bibliographic description inthe repositoryAvailability:This version is available at: 11583/2911831 since: 2021-07-08T16:21:29ZIEEEBayesian Optimization of Hyperparameters inKernel-Based Delay Rational ModelsFelipe TrevisoDept. of Electronics and Telecom.Politecnico di TorinoTorino, Italyfelipe.treviso@polito.itRiccardo TrincheroDept. of Electronics and Telecom.Politecnico di TorinoTorino, Italyriccardo.trinchero@polito.itFlavio G. CanaveroDept. of Electronics and Telecom.Politecnico di TorinoTorino, Italyflavio.canavero@polito.itAbstract—This paper presents an automatic procedure forthe optimization of the hyperparameters of a delay rationalmodel approximating the frequency-domain behavior of high-speed interconnects. The proposed model is built via a kernel-based regression, such as the Least-Square Support VectorMachine (LS-SVM), by considering an ad-hoc kernel with twohyperparameters related to the propagation delays introducedby the system. Such hyperparameters, along with the Tikhonovregularizer used by the LS-SVM regression, are carefully tunedvia an automatic approach based on a k-fold cross-validation andBayesian optimization. The feasibility of the effectiveness of theproposed modeling approach are investigated on a high-speedlink.Index Terms—High-speed link, delay rational model, MachineLearning regression, cross-validation.I. INTRODUCTIONIn high-speed links, the availability of accurate and fastinterconnect models is essential for signal and power in-tegrity (SPI) and electromagnetic interference (EMI) predic-tions through simulations. The simulations of the channel canbe performed in an early phase of the system design, andthus represent an useful tool for the design optimization andthe assessment of its performance. Due to the inherent lineartime invariant nature of the link interconnects, their models areusually based on the so-called rational models. However, ac-curate rational models of long interconnects usually require alarge number of poles. Such large number of poles unavoidablyhas an impact on the simulation time required by a transientsimulation involving systems composed by such interconnects.An alternative to reduce the model complexity and speed upthose analyses is the use of delay rational models (DRM) [1]–[3], which account explicitly for the delay present in suchstructures.A promising method for the estimation of a DRM is touse the least-squares support vector machine (LS-SVM) [4].This kernel-based technique allows the manipulation of infinitedimensional spaces, such that the model ensures that theexact delays of the system are accounted. By doing so, itis possible to accurately identify the delays of the originaltransfer function [5], [6], which is the hardest step to builda DRM. However, some parameters of the model and kernelshould be tuned in order to obtain a better performance. Thetuning of these so-called hyperparameters is a critical step inthe estimation of this approximated model, and the optimalapproximation can only be reached through an optimizationof such hyperparameters.Due to the complex structure of the model, the above op-timization is a non-convex problem, which may achieve localminima if performed with standard optimization techniques.This paper presents an automatic approach for the optimizationof the model hyperparameters and thus of its accuracy, basedon the combination of a k-fold cross-validation (CV) [7]scheme and a Bayesian optimization [8]. The strength andthe robustness the proposed approach are investigated on ahigh-speed link.II. KERNEL-BASED DELAY RATIONAL MODELDelay rational models are a natural choice to represent dis-tributed systems [1], [2]. Those models approximate sampledpairs of data from a transfer function {(sk, H(sk))}Kk=1 as alinear combination of basis functions containing poles pj ∈ Cand delays τi ∈ R asH(sk) ≈ H̃(sk) = r0 +nτ∑i=1np∑j=1rijsk − pje−skτi , (1)where the residues rij ∈ C are the coefficients of this linearcombination and r0 is a constant bias term.Kernel-based techniques are a class of machine learning(ML) regression models which can be applied to fit the dataof a transfer function. For that regression, they use as inputs ∈ C and output H̃(s) ∈ C. The dual space formulation ofthis model reads:H̃(s) =K∑k=1αkk(s, sk) + b, (2)where b ∈ C is a bias constant, αk ∈ C represent the Kmodel coefficients and k(s, sk) is the so-called kernel function.This is a non-parametric model where the number of estimatedcoefficients is always equal to K + 1 (the number of trainingsamples plus one), independent of the number and shape ofthe basis functions that such kernel reproduces. On the otherhand, the model in (1) requires the estimation of npnτ + 1coefficients, a number that changes according to the numberof basis functions accounted for by the model.Considering the LS-SVM framework, the dual formulationmodel in (2) can be suitably estimated by solving a system oflinear equations [9]. The main requirement for this represen-tation is the definition of the kernel function, which can bedone through [10]:k(s, sk) = 〈ϕ(s),ϕ(sk)〉 . (3)where ϕ(s) are the basis functions used in the model.In order that the above model corresponds to a DRM, thekernel function equates to:k(s, sk) = kp(s, sk)kτ (s, sk), (4)withkp(s, sk) =np∑j=1|p′j |(s− pj)(s∗k − p∗j) , (5)andkτ (s, sk)=(e−τM (s∗+sk)−e−τm(s∗+sk))−(s∗+sk) , s∗ + sk 6= 0τM − τm, s∗ + sk = 0. (6)The above kernel depends on the definition of the np polespj = p′j + jp′′j , and of an interval from a minimum delayτm to a maximum τM where the system’s propagation delaysshould be located. Once defined, such kernel represents a spaceformed by bases in the following form:ϕj(s; pj , τ) =|p′j |1/2s− pje−sτ , (7)which is equivalent to the basis of the DRM in (1). Inthe specific case of (4), the poles are a discrete set (i.e.,{p1, . . . , pnp}), while kτ (s, sk) accounts for all possible delayterms between τm and τM . Indeed, the use of such kernelprovides a feature space with an infinite number of dimensions,i.e., an infinite number of basis.Taking into account the kernel definition, the kernel-basedmodel can be equivalently represented in its primal spaceformulation asH̃(s) =np∑j=1∫ τMτmw′j(τ)ϕj(s; pj , τ)dτ + b, (8)wherew′j(τ) =K∑k=1αk|p′j |1/2s∗k − p∗je−s∗kτ (9)are constant coefficients defined for every combination of pjand τ considered in the model.After estimating the model, the weight w′j(τ) providesinformation on the values of pj and specially, τ , that havea larger influence in the model. Indeed, we can define thefollowing total weight W (τ):W (τ) =√√√√ np∑j=1‖w(τ, pj)‖2, (10)which sums the contributions of all the poles in the finalmodel, leading to a τ -dependency only. The analysis of (10)allows the accurate identification of the dominant propagationdelays of the system [5]. Those delays are the τ values wherethe peaks of W (τ) occur. Those few identified delays can beused to estimate compact delay rational models with a smallnumber of poles [1].The above format of model was shown to be accuratewithout the use of optimal poles [5]. However, similar tothe standard formulation of the DRM, the considered delayinterval has a large impact in the performance of the model,and therefore must be carefully tuned.III. TUNING OF THE MODEL HYPERPARAMETTERS VIACROSS VALIDATION AND BAYESIAN OPTIMIZATIONThe accuracy of the modeling scheme presented in theprevious section unavoidably depends on the tuning of thekernel hyperparameters τm and τM , which provide informa-tion on the minimum and maximum candidate delays of thesystem, and of the Tikhonov regularizer γ used by the LS-SVM regression. As an example, if a small value for thehyperparmetr τM is used, the model may be unable to followthe dynamic phase variations produced by the system. On theother hand, if a more conservative strategy is considered, asan example by using the largest delay interval allowed by thefrequency sampling (i.e., τm = 0 s and τM = 1/∆f ), the delayidentification procedure can be rather cumbersome, since thevalues of W (τ) must be analyzed in a large interval. Theregularizer γ provides a trade-off between the model error onthe training samples and the 2-norm of the model coefficients‖w′j(τ)‖22, thus preventing over-fitting.Techniques such as k-fold CV [7] and Bayesian optimiza-tion [8] can be seen as promising candidates to overcome theabove issues, since they are widely used within ML techniquesto select, without any manual tuning, the best configurationof the model hyperparmeters. In the k-fold CV [7], thetraining set is split into k smaller sets, called folds. Then,for each of the k folds, the model is trained using k − 1folds as training data and by using a given configuration ofthe hyperparameters λ = {γ, τm, τM}, while the remainingfold is kept as a validation set (i.e., to evaluate the modelaccuracy on data which were not used during the training).The above scheme is iterated for all the k-folds. Then, for eachanalyzed combination of the hyperparameters λ, the overallmodel performance is assessed by the k-fold CV error CV (λ)error,which is the average of the values computed during the kiterations, according toCV (λ)error =1kk∑n=1CV (λ)error,n, (11)where CV (λ)error,n is the mean squared error of the model on then-th test fold.The common choice for k is usually 5 or 10, as these valueshave been shown empirically to yield test error rate estimatesthat suffer neither from excessively high bias nor from veryhigh variance. An illustration of the k-fold CV for the case20 20 20 20 2020 20 20 20 20Step 1Step 2Step 5 20 20 20 20 20test fold train foldlegendfold 1 fold 2 fold 3 fold 4 fold 5⋮ ⋮ ⋮ ⋮ ⋮ ⋮Fig. 1. Example: a k-fold cross-validation with k = 5 and K = 100 trainingsamples.with K = 100 training samples and 5 folds (i.e., k = 5) isshown in Fig. 1.The optimum value of the hyperparameters λ? ={γ?, τ?m, τ?M} is selected as the one that minimizes the cor-responding overall CV error CV (λ)error, i.e.,λ? = arg minλCV (λ)error. (12)Together with the CV, a Bayesian optimization algorithm isused to solve the minimization problem in (12). This algorithmspecifically selects the next evaluated configuration of thehyperparameters λ in order to maximize the reward towardsfinding the global optimum of a non-convex function [8].By using this sampling scheme, the search converges to theoptimum solution in fewer iterations than with standard searchschemes, while keeping the capacity to avoid local minima.An open-source implementation of this optimization schemeis available in the scikit-optimize Python library [11].IV. APPLICATION EXAMPLEThe proposed methodology is exemplified by consideringthe transfer function H(s) = Vout(s)/E(s) for s = jω =j2πf of the high-speed link circuit in Fig. 2. The link consistsof three transmission lines based on microstrips, together withlumped elements that represent the parasitic effects of the link,in order to approximate the structure of a realistic interconnect.This structure has been implemented and simulated in HSPICEin a bandwidth from 0 to 20 GHz. Simulation results consistsof K = 1001 frequency points with a frequency spacing ∆f =20 MHz. A subset containing 101 samples, randomly selectedamong the available data, has been used as a validation dataset,whilst the remaining 900 samples are used as training set toconstruct the LS-SVM model.A DRM based on the modeling approach presented inSec. II is constructed to approximate H(s). The set of polesp is defined by drawing its real and imaginary parts randomlyand independently from a normal distribution N (0, 4π× 109)with zero mean and standard deviation of 4π × 109 rad/s.In the cases where the real part of the pole was positive,corresponding to an unstable pole, it was forced to be negativeby flipping its sign.During the model training, the optimum set of hyperparam-eters γ, τm and τM is selected via the CV-based Bayesianoptimization scheme presented in Sec. III using 5 folds and50 iterations. The parameters search space is restricted toE(s)75Ω 0.5nH0.1pF25Ω1nH0.2pF0.5nH0.1pF5Ω0.1pF0.5nH0.5pF 50ΩVout(s)εr = 4.1σ = 58 MS/m100 µm150 µm20 µmlength=10 cmlength=5 cmlength=8 cmFig. 2. Schematic of the circuit modeled in the application example.the intervals γ ∈ [103, 1025] and τm, τM ∈ [0, 50] ns. Theconstraint τM > τm is not enforced, however it is expected asthe logical outcome of the optimization. After the 50 cycles,the obtained optimized parameters are γ? = 6.56 × 1021,τ?m = 5.68 × 10−8 and τ?M = 7.39 ns. Figures 3 and 4 showthat these parameters provide a very accurate model whenapplying it to the validation data, where the model outputalmost perfectly matches the original points of H(s).0 5 10 15 20f [GHz]0.100.150.200.250.300.35‖H(s)‖H(s)Validation dataModel outputFig. 3. Magnitude comparison of the LS-SVM model output and the originaltransfer function H(s).Additionally, the proposed LS-SVM model of H(s) canbe used to identify the dominating propagation delays of thesystem of Fig. 2. These delays should be searched only withinthe optimized delay interval τ?m to τ?M , making it possibleto perform a more fine discretization without incurring intoan unreasonable computational time. The delays are identifiedfrom W (τ), which is computed according to (10). All thiscomputational procedure took only 940.7 s, of which 937.2 swere used for the estimation of the optimized model and 3.5 sfor the delay identification. The plot of W (τ) is shown inFig. 5. In this figure, the black curve provided by the optimizedmodel is compared with the blue one obtained by means of abasic tuning of the parameters, where τm is set to its minimum0 5 10 15 20f [GHz]−3−2−101236H(s)H(s)Validation dataModel outputFig. 4. Phase comparison of the LS-SVM model output and the originaltransfer function H(s).and τM and γ are set to their maximum possible values. It isobserved that the peaks of the black curve are much moreidentifiable than in the blue curve, while it is also less noisy.Moreover, τM in the blue curve goes up to 50 ns, which resultsin a worse resolution of the τ -axis if a fixed number of pointsis considered in the discretization. The peaks of such plotcorrespond to the propagation delays produced by the originaltransfer function. For example, the first marked peak occursat 0.76 ns, while the two transmission lines in the main signalpath have a total length of 13 cm. Those values representa propagation speed of 1.71 × 108 m/s, which is compatiblewith the real speed in such structures. The additional peaksare also clear in the plot, together with smaller peaks that canbe identified if necessary. Such obtained delays are useful tothe estimation of compact and accurate DRMs.V. CONCLUSIONKernel-based techniques are a very flexible tool to dealwith systems with delays. They provide a way to model thedata using a very large or infinite number of basis. This factmakes it possible to ensure that the true propagation delays ofthe distributed system are accounted in the estimated model.However, the added flexibility comes at the cost of tuningsome extra parameters. Those parameters were optimized viaa Bayesian method together with a 5-fold cross validation.Such optimization provided an accurate kernel-based modelafter few iterations. The optimized model provides an easyway to identify the system dominant propagation delays, wherethe delays should be searched in a small interval given by theoptimized parameters, making it a simpler task than if thewhole possible interval would be considered.0 10 20 30 40 50τ [ns]01234W(τ)×10140 2 4 6W (τ) - Basic tuning of pars.W (τ) - Optimized parametersMost relevant peaksFig. 5. Plot of W (τ) obtained from the LS-SVM model of H(s) indicatingthe dominating propagation delays of the system of Fig. 2. The detail amplifiesit in the interval from τ?m to τ?M .REFERENCES[1] A. Chinea, P. Triverio, S. Grivet-Talocia, “Delay-Based Macromodellingof Long Interconnects From Frequency-Domain Terminal Responses”,IEEE Trans. on Advanced Packaging, Vol. 33, No. 1, Feb. 2010.[2] A. Chinea et al., “Signal Integrity Verification of Multichip LinksUsing Passive Channel Macromodels”, IEEE Trans. on Components,Packaging and Manufact. Tech., vol. 1, no. 6, pp. 920-933, Jun. 2011.[3] E. R. Samuel, L. Knockaert and T. Dhaene, “Model Order Reduction ofTime-Delay Systems Using a Laguerre Expansion Technique,” in IEEETransactions on Circuits and Systems I: Regular Papers, vol. 61, no. 6,pp. 1815-1823, June 2014.[4] J.A.K. Suykens, T. Van Gestel, J. De Brabanter, B. De Moor, J.Vandewalle, Least Squares Support Vector Machines, World Scientific,Singapore, 2002 (ISBN 981-238-151-1).[5] F. Treviso, R. Trinchero and F. G. Canavero, “Machine Learning Appliedto the Blind Identification of Multiple Delays in Distributed Systems,”2020 XXXIIIrd General Assembly and Scientific Symposium of theInternational Union of Radio Science, Rome, Italy, 2020.[6] F. Treviso, R. Trinchero and F. G. Canavero, “Multiple Delay Identifi-cation in Long Interconnects via LS-SVM Regression,” in IEEE Access,vol. 9, pp. 39028-39042, 2021, doi: 10.1109/ACCESS.2021.3063713.[7] B. Ghojogh and M. Crowley, “The theory behind overfitting, cross vali-dation, regularization, bagging, and boosting: Tutorial”, arXiv preprintarXiv:1905.12787, 2019.[8] E. Brochu, M. Cora,and N.de Freitas, “A tutorial on Bayesian opti-mization of expensive cost functions, with application to active usermodeling and hierarchical reinforcement learning”, arXiv preprintarXiv:1012.2599, 2010.[9] R. Trinchero, et al., “Machine Learning and Uncertainty Quantificationfor Surrogate Models of Integrated Devices With a Large Number ofParameters,” in IEEE Access, vol. 7, pp. 4056-4066, 2019.[10] N. Cristianini, J. Shawne-Taylor, An Introduction to Support VectorMachines and Other Kernel Based Learning Methods, Cambridge Uni-versity Press, UK, 2000 (ISBN 0-521-78019-5).[11] Head, Tim, Kumar, Manoj, Nahrstaedt, Holger, Louppe, Gilles, andShcherbatyi, Iaroslav, “scikit-optimize/scikit-optimize”. Zenodo, 04-Sep-2020.

Bayesian Optimization of Hyperparameters in Kernel-Based Delay Rational Models

https://iris.polito.it/bitstream/11583/2911831/2/2021069997.pdf

Bayesian Optimization of Hyperparameters in Kernel-Based Delay Rational Models

Abstract

Similar works

Full text

Available Versions

PORTO@iris (Publications Open Repository TOrino - Politecnico di Torino)