Baum, Andreas

Vermue, Laurent

Partial Least Squares (PLS) regression is a statistical method for supervised multivariate analysis. It relates two data blocks X and Y to each other with the aim of establishing a prediction model. When deployed in production, this model can be used to predict an outcome y from a newly measured feature vector x. PLS is popular in chemometrics, process control and other analytic fields, due to its striking advantages, namely the ability to analyze small sample sizes and the ability to handle high-dimensional data with cross-correlated features (where Ordinary Least Squares regression typically fails). In addition, and in contrast to many other machine learning approaches, PLS models can be interpreted using its latent variable structure just like principal components can be interpreted for a PCA analysis

Online Research Database In Technology

  General rights Copyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights.   Users may download and print one copy of any publication from the public portal for the purpose of private study or research.  You may not further distribute the material or use it for any profit-making activity or commercial gain  You may freely distribute the URL identifying the publication in the public portal  If you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediately and investigate your claim.         Downloaded from orbit.dtu.dk on: Mar 30, 2019Multiblock PLS: Block dependent prediction modeling for PythonBaum, Andreas; Vermue, LaurentPublished in:The Journal of Open Source SoftwareLink to article, DOI:10.21105/joss.01190Publication date:2019Document VersionPublisher's PDF, also known as Version of recordLink back to DTU OrbitCitation (APA):Baum, A., & Vermue, L. (2019). Multiblock PLS: Block dependent prediction modeling for Python. The Journal ofOpen Source Software, 4(34), [1190]. DOI: 10.21105/joss.01190Multiblock PLS: Block dependent prediction modelingfor PythonAndreas Baum1 and Laurent Vermue11 Department of Applied Mathematics and Computer Science, Technical University of Denmark,Richard Petersens Plads 324, DK-2800 Kgs. Lyngby, DenmarkDOI: 10.21105/joss.01190Software• Review• Repository• ArchiveSubmitted: 06 January 2019Published: 10 February 2019LicenseAuthors of papers retain copy-right and release the work un-der a Creative Commons Attri-bution 4.0 International License(CC-BY).IntroductionPartial Least Squares (PLS) regression is a statistical method for supervised multivariateanalysis. It relates two data blocks X and Y to each other with the aim of establishinga prediction model. When deployed in production, this model can be used to predict anoutcome y from a newly measured feature vector x. PLS is popular in chemometrics,process control and other analytic fields, due to its striking advantages, namely the abil-ity to analyze small sample sizes and the ability to handle high-dimensional data withcross-correlated features (where Ordinary Least Squares regression typically fails). Inaddition, and in contrast to many other machine learning approaches, PLS models canbe interpreted using its latent variable structure just like principal components can beinterpreted for a PCA analysis.Multivariate data is often structured in blocks, e.g. X1, X2, … , Xi. This could meanthat one has obtained data from two different analytic methodologies for a similar set ofsamples, which may indicate two totally independent feature spaces. In such cases it isoften important to understand how each data block contributes to the prediction of Y.Examples for data measured in blocks could be the following.1. It can be of interest to relate patient clinical records to data obtained through differ-ent high-throughput omics measurements. These data could typically be structuredin blocks referring to genomics, transciptomics, proteomics, metabolomics etc.2. Spectroscopic methods are useful to predict and assure food quality parameters.When measuring food samples by several different spectroscopic methods, e.g. byapplying near infrared and UV-vis spectroscopy, it is meaningful to combine thedata blocks to obtain reliable prediction models.3. A process utilizing fermentation technology is typically carried out in several se-quential production phases, i.e. seed and main fermentation phase. If sensor datais available for all production phases it is meaningful to include these as individ-ual data blocks when establishing prediction models for quality control parameters,such as product yield.Several Data Fusion approaches were proposed to establish combined prediction modelsfrom such multiblock data (Li, Wu, & Ngom, 2016). One of the proposed methodsis Multiblock-PLS (MB-PLS) (Westerhuis, Kourti, & MacGregor, 1998). It is closelyrelated to PLS regression, but instead of obtaining an interpretative model for the entire(concatenated) data matrix X one obtains model parameters for each individual datablock Xi. Furthermore, it provides a relative importance measure, i.e. expressing howBaum et al., (2019). Multiblock PLS: Block dependent prediction modeling for Python. Journal of Open Source Software, 4(34), 1190.https://doi.org/10.21105/joss.011901much each block Xi contributes to the prediction of Y. Subsequently, this informationcan be used to recognize block specific patterns in the data.At the current stage software packages for MB-PLS exist for Matlab (http://www.models.life.ku.dk/MBToolbox) and R (Bougeard & Dray, 2018). In the following sections wegive a brief introduction to the statistical method and its implementation. The pack-age is distributed under the BSD-3-Clause license and made available at https://github.com/DTUComputeStatisticsAndDataAnalysis/MBPLS together with several introduc-tory Jupyter notebook examples. It is also available as pip installable Python pack-age (https://pypi.org/project/mbpls/) and comes with a Read-the-Docs documentation(https://mbpls.readthedocs.io).MethodsThe MB-PLS package can be utilized for PLS and MB-PLS regression. The statisticalbackground is briefly introduced in the following. More detailed information is given in thembpls help of the Python package (https://mbpls.readthedocs.io/en/latest/mbpls.html).PLSPLS was introduced by S. Wold, Ruhe, Wold, & III (1984) and aims at finding a suitablesubspace projection w which maximizes co-variance between a so called score vector tand a response vector y that will yield a least squares solution. The formal PLS criterionfor univariate responses y is given in eq. 1.argmaxwcov(t,y)∣∣∣∣∣∣min I∑i=1J∑j=1(xij − tiwj)2 ∧ ‖w‖ = 1 (1)This procedure is typically repeated to findK latent variables (LV). In each latent variablestep, the score vector tk is subsequently projected onto its respective matrix Xk to findthe loading vector pk (eq. 2). Once found, Xk is deflated by the explained variance (eq.3) and the next latent variable k + 1 can be calculated using Xk+1.pk = Xktk (2)Xk+1 = Xk − tkpkT (3)Algorithms to perform PLS regression include the Nonlinear Iterative PArtial LeastSquares (NIPALS) (S. Wold et al., 1984), UNIversal PArtial Least Squares (UNIPALS)(Dunn III, Scott, & Glen, 1989), Kernel UNIPALS (Lindgren, Geladi, & Wold, 1993;Rännar, Geladi, Lindgren, & Wold, 1995; Rännar, Lindgren, Geladi, & Wold, 1994) andSIMPLS algorithm (de Jong, 1993). While NIPALS represents an iterative approach,the other algorithms are based on Singular Value Decomposition (SVD). All the abovementioned algorithms are implemented in the MB-PLS package. Benchmark results andcomparisons to other Software packages are provided below.MB-PLSMB-PLS can be understood as an extension of PLS to incorporate several data blocksX1, ...,Xi, which all share a common sample dimension. The prediction accuracy does notdeviate from normal PLS, if all data blocks were concatenated into a single block, but theBaum et al., (2019). Multiblock PLS: Block dependent prediction modeling for Python. Journal of Open Source Software, 4(34), 1190.https://doi.org/10.21105/joss.011902X1,kp1,kt1,kMB-PLSYvkukX2,kp2,kt2,kBIP = 43% BIP = 57% 67% expl. var.k = 1Figure 1: The figure illustrates the extraction of a single LV (k = 1). MB-PLS offers extra exploratoryfeatures for each block, i.e. block scores, block loadings and block importances (BIP).advantage of MB-PLS is to gain extra model interpretability concerning the underlyingblock structure of the data. For each LV one obtains extra block scores, block loadingsand block importances (BIP). The extraction of a single LV using MB-PLS is illustratedin figure 1. The results are read in a fashion that 67% variance in Y are explained by thefirst LV. The two blocks X1 and X2 contribute to the prediction of the 67% with theirrelative BIPs, 43% and 57%, respectively. More important blocks result in more influentialblock loadings and contribute stronger to the prediction of Y. Hence, interpretation ofpatterns among block scores with high importance are recommended.To assert that the BIP is a meaningful indicator it is necessary to standardize the dataprior to MB-PLS analysis. When standardization is employed all features in all blockshave a variance of 1. For post-hoc interpretation of the loadings an inverse transformationis carried out to ensure straight forward interpretation of the results.Software and ImplementationThe package is written in pure Python 3. In its core it builds on Numpy and Scipy forefficient data handling and fast mathematical operations of big data-sets. To achievea fast implementation all algorithms using SVD employ Scipy’s partial SVD capability,i.e. by only calculating the first singular value at each PLS iteration. Multiple matrixmultiplications use the optimized Numpy multi-array multiplication. In addition, theMB-PLS implementation can handle missing data without prior imputation based on thesparse NIPALS algorithm by H. Martens & Martens (2001). The overall code designfollows the structure and philosophy of Scikit-learn (Pedregosa et al., 2011). Therefore,objects are instantiated in a Scikit-learn manner and can be accessed with the samemethods, i.e. fit, predict, transform and score. Furthermore, Scikit-learn’s base classesand validation methods are incorporated. As a result, all objects are fully compatibleto Scikit-learn and, thus, allow the use of model selection functions, e.g. cross validationand grid search, as well as a processing pipeline. For exploratory analysis, each fittedmodel contains a custom plot method that the fitted model attributes in a meaningfulmanner using Matplotlib, which allows a straight forward evaluation of the MBPLS resultswithout requiring any additional coding.BenchmarkTo compare the four algorithms, the run-times are analyzed for different data-set sizes withtwo basic shapes, i.e. non-symmetric shapes with more samples than variables (N > P )or vice versa (N < P ) and symmetric shapes (N = P ). To simulate the multiblock andmultivariate behaviour, each data-set is split into two X-blocks with the size N × P2 andBaum et al., (2019). Multiblock PLS: Block dependent prediction modeling for Python. Journal of Open Source Software, 4(34), 1190.https://doi.org/10.21105/joss.011903Figure 2: Comparison of run-times based on different data-set sizesaccompanied by a Y-block of size N × 10. The data is randomly generated for each run,so that the obtained times exhibit worst-case behaviour, since there are no actual latentstructures. All algorithms are set to find the first 20 LVs and are run three times for eachdata-set size on a machine with two Intel ® Xeon ® X5650 @ 2.67 GHz processors and48 GB RAM.As to be seen in both both plots of figure 2 all algorithms implemented in the Pythonmbpls package substantially outperform the above mentioned R-package Ade4-MBPLS byBougeard & Dray (2018), which was run on the same machine. In general NIPALS is thefastest multiblock algorithm that is only outperformed by the SIMPLS algorithm, whichonly supports single block PLS. However, figure 2a shows how the KERNEL algorithmperforms progressively better in cases where N >> P or N << P . As to be seen inthis plot, the runtime of this algorithm is a combination of an exponential part givenby the right plot and dependent on min(N,P ) and a linear part defined by diff(N,P ).Due to the exponential part, min(N,P ) has to be considered carefully when choosing theKERNEL algorithm over NIPALS.An important feature of this Python mbpls package is its invariance to shape rotations,i.e. it obtains the same run-times for both N > P and N < P given the same ratio NPand its respective inverse, which e.g. is not the case for the R-package.AcknowledgementThe authors gratefully acknowledge the financial support through the BioPro (Innova-tionsfonden project nr. 10513) and DABAI (Innovationsfonden project nr. 10599 and10577) project.ReferencesBougeard, S., & Dray, S. (2018). Supervised multiblock analysis in r with the ade4package. Journal of Statistical Software, 86(1), 1–17. doi:10.18637/jss.v086.i01de Jong, S. (1993). SIMPLS: An alternative approach to partial least squares regres-sion. Chemometrics and intelligent laboratory systems, 18(3), 251–263. doi:10.1016/0169-7439(93)85002-XDunn III, W. J., Scott, D. R., & Glen, W. G. (1989). Principal components analysisand partial least squares regression. Tetrahedron computer methodology, 2(6), 349–376.doi:10.1016/0898-5529(89)90004-3Baum et al., (2019). Multiblock PLS: Block dependent prediction modeling for Python. Journal of Open Source Software, 4(34), 1190.https://doi.org/10.21105/joss.011904Li, Y., Wu, F.-X., & Ngom, A. (2016). A review on machine learning principles for multi-view biological data integration. Briefings in Bioinformatics, 19(2), 325–340. doi:10.1093/bib/bbw113Lindgren, F., Geladi, P., & Wold, S. (1993). The kernel algorithm for pls. Journal ofChemometrics, 7(1), 45–59. doi:10.1002/cem.1180070104Martens, H., & Martens, M. (2001). Multivariate analysis of quality: An introduction.Chichester: Wiley.Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel,M., et al. (2011). Scikit-learn: Machine learning in python. Journal of Machine LearningResearch, 12(Oct), 2825–2830.Rännar, S., Geladi, P., Lindgren, F., & Wold, S. (1995). A pls kernel algorithm for datasets with many variables and few objects. Part ii: Cross-validation, missing data andexamples. Journal of Chemometrics, 9(6), 459–470. doi:10.1002/cem.1180090604Rännar, S., Lindgren, F., Geladi, P., & Wold, S. (1994). A pls kernel algorithm for datasets with many variables and fewer objects. Part 1: Theory and algorithm. Journal ofChemometrics, 8(2), 111–125. doi:10.1002/cem.1180080204Westerhuis, J. A., Kourti, T., & MacGregor, J. F. (1998). Analysis of multiblock andhierarchical pca and pls models. Journal of Chemometrics, 12(5), 301–321. doi:10.1002/(SICI)1099-128X(199809/10)12:5<301::AID-CEM515>3.0.CO;2-SWold, S., Ruhe, A., Wold, H., & III, W. J. D. (1984). The collinearity problem in linearregression. The partial least squares (pls) approach to generalized inverses. SIAM Journalon Scientific and Statistical Computing, 5(3), 735–743. doi:10.1137/0905052Baum et al., (2019). Multiblock PLS: Block dependent prediction modeling for Python. Journal of Open Source Software, 4(34), 1190.https://doi.org/10.21105/joss.011905

Multiblock PLS: Block dependent prediction modeling for Python

http://orbit.dtu.dk/ws/files/169761830/10.21105.joss.01190.pdf

Multiblock PLS: Block dependent prediction modeling for Python

Abstract

Similar works

Full text

Available Versions

Online Research Database In Technology