Partial least squares regression

Regression analysis
Part of a series on Statistics

Models
Linear regression Simple regression Ordinary least squares Polynomial regression General linear model
Generalized linear model Discrete choice Logistic regression Multinomial logit Mixed logit Probit Multinomial probit Ordered logit Ordered probit Poisson
Multilevel model Fixed effects Random effects Mixed model
Nonlinear regression Nonparametric Semiparametric Robust Quantile Isotonic Principal components Least angle Local Segmented
Errors-in-variables
Estimation
Least squares Ordinary least squares Linear (math) Partial Total Generalized Weighted Non-linear Non-negative Iteratively reweighted Ridge regression
Least absolute deviations Bayesian Bayesian multivariate
Background
Regression model validation Mean and predicted response Errors and residuals Goodness of fit Studentized residual Gauss–Markov theorem
Statistics portal

Partial least squares regression (PLS regression) is a statistical method that bears some relation to principal components regression; instead of finding hyperplanes of maximum variance between the response and independent variables, it finds a linear regression model by projecting the predicted variables and the observable variables to a new space. Because both the X and Y data are projected to new spaces, the PLS family of methods are known as bilinear factor models. Partial least squares Discriminant Analysis (PLS-DA) is a variant used when the Y is categorical.

PLS is used to find the fundamental relations between two matrices (X and Y), i.e. a latent variable approach to modeling the covariance structures in these two spaces. A PLS model will try to find the multidimensional direction in the X space that explains the maximum multidimensional variance direction in the Y space. PLS regression is particularly suited when the matrix of predictors has more variables than observations, and when there is multicollinearity among X values. By contrast, standard regression will fail in these cases (unless it is regularized).

The PLS algorithm is employed in partial least squares path modeling,^[1]^[2] a method of modeling a "causal" network of latent variables (causes cannot be determined without experimental or quasi-experimental methods, but one typically bases a latent variable model on the prior theoretical assumption that latent variables cause manifestations in their measured indicators). This technique is argued to be a form of structural equation modeling, distinguished from the classical method by being component-based rather than covariance-based.^[3] Yet, others dispute that this is the case.^[4]

Partial least squares was introduced by the Swedish statistician Herman Wold, who then developed it with his son, Svante Wold. An alternative term for PLS (and more correct according to Svante Wold^[5]) is projection to latent structures, but the term partial least squares is still dominant in many areas. Although the original applications were in the social sciences, PLS regression is today most widely used in chemometrics and related areas. It is also used in bioinformatics, sensometrics, neuroscience and anthropology. In contrast, PLS path modeling is most often used in social sciences, econometrics, marketing and strategic management. However, within the realm of psychology, it has received criticism for being an unreliable estimation and testing tool.^[4]

Underlying model

The general underlying model of multivariate PLS is

X = T P^{\top} + E

Y = U Q^{\top} + F

where $X$ is an $n\times m$ matrix of predictors, $Y$ is an $n\times p$ matrix of responses; $T$ and $U$ are $n \times l$ matrices that are, respectively, projections of $X$ (the X score, component or factor matrix) and projections of $Y$ (the Y scores); $P$ and $Q$ are, respectively, $m \times l$ and $p \times l$ orthogonal loading matrices; and matrices $E$ and $F$ are the error terms, assumed to be independent and identically distributed random normal variables. The decompositions of $X$ and $Y$ are made so as to maximise the covariance between $T$ and $U$ .

Algorithms

A number of variants of PLS exist for estimating the factor and loading matrices $T, U, P$ and $Q$ . Most of them construct estimates of the linear regression between $X$ and $Y$ as $Y = X \tilde{B} + \tilde{B}_0$ . Some PLS algorithms are only appropriate for the case where $Y$ is a column vector, while others deal with the general case of a matrix $Y$ . Algorithms also differ on whether they estimate the factor matrix $T$ as an orthogonal, an orthonormal matrix or not.^[6]^[7]^[8]^[9]^[10]^[11] The final prediction will be the same for all these varieties of PLS, but the components will differ.

PLS1

PLS1 is a widely used algorithm appropriate for the vector $Y$ case. It estimates $T$ as an orthonormal matrix. In pseudocode it is expressed below (capital letters are matrices, lower case letters are vectors if they are superscripted and scalars if they are subscripted):

 1  function PLS1( $X, y, l$ )
 2   $X^{(0)} \gets X$ 
 3   $w^{(0)} \gets X^T y/||X^Ty||$ , an initial estimate of  $w$ .
 4   $t^{(0)} \gets X w^{(0)}$  
 5  for  $k=0$  to  $l$ 
 6       $t_k \gets {t^{(k)}}^T t^{(k)}$  (note this is a scalar)
 7       $t^{(k)} \gets t^{(k)} / t_k$ 
 8       $p^{(k)} \gets {X^{(k)}}^T t^{(k)}$ 
 9       $q_k \gets {y}^T t^{(k)}$  (note this is a scalar)
10      if  $q_k = 0$ 
11           $l \gets k$ , break the for loop
12      if  $k < l$ 
13           $X^{(k+1)} \gets X^{(k)} - t_k t^{(k)} {p^{(k)}}^T$ 
14           $w^{(k+1)} \gets {X^{(k+1)}}^T y$ 
15           $t^{(k+1)} \gets X^{(k+1)}w^{(k+1)}$ 
16  end for
17  define  $W$  to be the matrix with columns  $w^{(0)},w^{(1)},...,w^{(l-1)}$ .
    Do the same to form the  $P$  matrix and  $q$  vector.
18   $B \gets W {(P^T W)}^{-1} q$ 
19   $B_0 \gets q_0 - {P^{(0)}}^T B$ 
20  return  $B, B_0$

This form of the algorithm does not require centering of the input $X$ and $Y$ , as this is performed implicitly by the algorithm. This algorithm features 'deflation' of the matrix $X$ (subtraction of $t_k t^{(k)} {p^{(k)}}^T$ ), but deflation of the vector $y$ is not performed, as it is not necessary (it can be proved that deflating $y$ yields the same results as not deflating.). The user-supplied variable $l$ is the limit on the number of latent factors in the regression; if it equals the rank of the matrix $X$ , the algorithm will yield the least squares regression estimates for $B$ and $B_{0}$

Extensions

In 2002 a new method was published called orthogonal projections to latent structures (OPLS). In OPLS, continuous variable data is separated into predictive and uncorrelated information. This leads to improved diagnostics, as well as more easily interpreted visualization. However, these changes only improve the interpretability, not the predictivity, of the PLS models.^[12] L-PLS extends PLS regression to 3 connected data blocks.^[13] Similarly, OPLS-DA (Discriminant Analysis) may be applied when working with discrete variables, as in classification and biomarker studies.

In 2015 partial least squares was related to a procedure called the three-pass regression filter (3PRF).^[14] Supposing the number of observations and variables are large, the 3PRF (and hence PLS) is asymptotically normal for the "best" forecast implied by a linear latent factor model. In stock market data, PLS has been shown to provide accurate out-of-sample forecasts of returns and cash-flow growth.^[15]

Software implementation

Most major statistical software packages offer PLS regression.. PLS regression is implemented in SAS via the PLS procedure. The 'pls' package in R provides a range of algorithms.^[16]

References

↑ Tenenhaus, M.; Esposito Vinzi, V.; Chatelinc, Y-M.; Lauro, C. (January 2005). "PLS path modeling" (PDF). Computational Statistics & Data Analysis. 48 (1): 159–205. doi:10.1016/j.csda.2004.03.005.
↑ Vinzi, V.; Chin, W.W.; Henseler, J.; et al., eds. (2010). Handbook of Partial Least Squares. ISBN 978-3-540-32825-4.
↑ Tenenhaus, M. (2008). "Component-based structural equation modelling" (PDF).
1 2 Rönkkö, M.; McIntosh, C.; Antonakis, J. (2015). "On the adoption of partial least squares in psychological research: Caveat emptor". Personality and Individual Differences. pp. 76–84. doi:10.1016/j.paid.2015.07.019.
↑ Wold, S; Sjöström, M.; Eriksson, L. (2001). "PLS-regression: a basic tool of chemometrics". Chemometrics and Intelligent Laboratory Systems. 58 (2): 109–130. doi:10.1016/S0169-7439(01)00155-1.
↑ Lindgren, F; Geladi, P; Wold, S (1993). "The kernel algorithm for PLS". J. Chemometrics. 7: 45–59. doi:10.1002/cem.1180070104.
↑ de Jong, S.; ter Braak, C.J.F. (1994). "Comments on the PLS kernel algorithm". J. Chemometrics. 8 (2): 169–174. doi:10.1002/cem.1180080208.
↑ Dayal, B.S.; MacGregor, J.F. (1997). "Improved PLS algorithms". J. Chemometrics. 11 (1): 73–85. doi:10.1002/(SICI)1099-128X(199701)11:1<73::AID-CEM435>3.0.CO;2-#.
↑ de Jong, S. (1993). "SIMPLS: an alternative approach to partial least squares regression". Chemometrics and Intelligent Laboratory Systems. 18 (3): 251–263. doi:10.1016/0169-7439(93)85002-X.
↑ Rannar, S.; Lindgren, F.; Geladi, P.; Wold, S. (1994). "A PLS Kernel Algorithm for Data Sets with Many Variables and Fewer Objects. Part 1: Theory and Algorithm". J. Chemometrics. 8 (2): 111–125. doi:10.1002/cem.1180080204.
↑ Abdi, H. (2010). "Partial least squares regression and projection on latent structure regression (PLS-Regression)". Wiley Interdisciplinary Reviews: Computational Statistics. 2: 97–106. doi:10.1002/wics.51.
↑ Trygg, J; Wold, S (2002). "Orthogonal Projections to Latent Structures". Journal of Chemometrics. 16 (3): 119–128. doi:10.1002/cem.695.
↑ Sæbøa, S.; Almøya, T.; Flatbergb, A.; Aastveita, A.H.; Martens, H. (2008). "LPLS-regression: a method for prediction and classification under the influence of background information on predictor variables". Chemometrics and Intelligent Laboratory Systems. 91 (2): 121–132. doi:10.1016/j.chemolab.2007.10.006.
↑ Kelly, Bryan; Pruitt, Seth (2015-06-01). "The three-pass regression filter: A new approach to forecasting using many predictors". Journal of Econometrics. High Dimensional Problems in Econometrics. 186 (2): 294–316. doi:10.1016/j.jeconom.2015.02.011.
↑ Kelly, Bryan; Pruitt, Seth (2013-10-01). "Market Expectations in the Cross-Section of Present Values". The Journal of Finance. 68 (5): 1721–1756. doi:10.1111/jofi.12060. ISSN 1540-6261.
↑ "Package 'pls'" (PDF).

External links

imDEV free Excel add-in for PLS and PLS-DA
PLS in Brain Imaging
on-line PLS regression (PLSR) at Virtual Computational Chemistry Laboratory
Uncertainty estimation for PLS
A short introduction to PLS regression and its history

Authority control	GND: 4591652-4

This article is issued from Wikipedia - version of the 11/18/2016. The text is available under the Creative Commons Attribution/Share Alike but additional terms may apply for the media files.