<![CDATA[
A multivariate data analysis technique used to relate several response variables to several explanatory variables, aiming to identify the underlying factors, or linear combination of the X variables, which best model the Y dependent variables.
<P>
PARTIAL LEAST SQUARES REGRESSION
<P>
Partial Least Squares (PLS) regression is a multivariate data analysis
technique which can be used to relate several response (Y) variables to
several explanatory (X) variables. <p>
The method aims to identify the underlying factors, or linear combination
of the X variables, which best model the Y dependent variables.<p>
PLS can deal efficiently with data sets where there are very many variables
that are highly correlated and involving substantial random noise.
<p>
Partial least squares (PLS) is a method of modelling relationships between
one or more Y variables and one-to-many explanatory variables.
<p>
The PLS approach was developed and adapted principally by Svante Wold and
Harald Martens from an algorithm first proposed by H. Wold (1975).
<P>
<BR>
An application to the study of potato variety yields data :
<p>
DATA : <p>
<blockquote>
Y VARIABLES <br>
<font color="FF0000">
- variety yields from a number of centres (centres = variables)
</font>
<p>
X VARIABLES <br>
<font color="FF0000">
- variety characteristics, e.g. disease resistance.
</font>
</blockquote>
<br>
QUESTIONS TO BE ANSWERED : <p>
<blockquote>
WHAT INFLUENCES VARIETY X CENTRE VARIATION ? <br>
<font color="FF0000">
- are there variety characteristics which control how variety yields
change between environments ?
</font>
<p>
HOW EFFECTIVE IS THE PLS MODEL ? <br>
<font color="FF0000">
- what proportion of variation is explained by the fitted model ?
</font>
<p>
</blockquote>
<P>
<BR>
The example is taken from potato variety trials in the UK.
<p>
It is often difficult to interpret
variety x environment interactions in the analysis of data from trials repeated over several
locations. One approach is to calculate the principal components scores from the variety x
trials data, after the main effects of variety and trials have been removed, and then try to
relate these scores to other varietal characters.
<p>
A similar, but more direct, approach is to use PLS to provide a parsimonious linear model of
the systematic relationship between the variety x trials data and the other variety characters.
<P>
POTATO VARIETY X ENVIRONMENT DATA
<BR>
The data consists of potato variety tuber yields for 9 varieties, all sown at twelve centres
throughout the UK in the years 1983-87. So each item in the table represents the mean from
five years of trials with the variety at that centre.
<p>
In addition there was available botanical, agronomic, disease and pest susceptibility scores for
each of the varieties, all of which were rated on a scale 1-9 where 9 means the variety
possesses the characteristic to a high degree, e.g. good drought resistance, pale crisp colour,
high dry matter content.
<P>
LOCATION OF POTATO VARIETY TRIALS
<BR>
The 12 centres were distributed throughout England, Wales, Scotland and N. Ireland.
<P>
PLS METHOD AND V x E MODELS
<BR>
<a href="slide06.htm">
PLS METHOD </A>
<p>
Aims to identify weighted combinations of the X variables
<a href="glossary.htm#loading"> (loading vectors) </a>
which best model the Y variables.
<p>
<A HREF="ref.htm">
KEY PAPERS </A>
<P>
Development of theory and expository texts.
<P>
<a href="slide04.htm">
V x E MODELLING </A>
<p>
PLS fits within a broad class of V x E modelling strategies.
<P>
<BR>
There are a number of versions of the PLS method. The one described here can be used to
relate several Y variables to several X variables.
<p>
The analysis of V x E data by the fitting of linear and multiplicative models has a well
established history. PLS can be fitted within this broad class of modelling strategies.
<P>
LOADING VECTOR FOR VARIETY CHARACTERS
<BR>
Factor 1 ( ordered )
<pre>
1983-87
Dry Matter -0.52
Crisp Colour -0.48
Tuber Blight -0.34
Maturity -0.21
Gangrene -0.08
External Damage -0.06
Virus Y -0.01
Foliage Blight 0.01
Foliage Cover 0.13
Dormancy 0.15
Drought Resistance 0.16
Internal Damage 0.24
Common Scab 0.31
Leaf Roll Virus 0.34
</pre>
<P>
<BR>
Application of the PLS to the potato data suggested that there was only one factor of
relevance for prediction. It explains some 40% of the variation in the Y matrix.
<p>
The Factor 1 loadings are presented here in rank order. This places dry matter and crisp
colour at one end of the scale and leaf roll virus and common scab at the other end.
<p>
Crisp colour assessment is known to be positively correlated with dry matter content. Also,
leaf roll virus susceptibility is known to be negatively correlated with DM content of tubers.
<P>
PARTIAL LEAST SQUARES MODEL
<BR>
<img alt="equation" src="../images/image3.gif" WIDTH=240 HEIGHT=41>
<br>
<ul>
<li> Y is the yield data matrix with variety and environment effects
removed;
<li> t are <a href="glossary.htm#latent"> latent variables </a> or
scores;
<li> q are <a href="glossary.htm#loading"> loading vectors </a>.
</ul>
We also have a matrix of explanatory variables (X) which may be associated
with components of the Y matrix. <p>
<img alt="equation" src="../images/image4.gif" WIDTH=238 HEIGHT=49> <p>
<ul>
<li> t are same latent variables as above
<li> p are loading vectors.
</ul>
Note : p and q describe how varieties and explanatory variables combine
and t describes variation across environments.
<P>
<BR>
The PLS method aims to identify the underlying factors, or a linear combination of the X
variables, which best model the Y dependent variables.
<P>
STAGES IN THE PLS ALGORITHM
<BR>
The algorithm involves an iterative procedure.
<p>
First estimate a linear combinations of the X variables to give a latent vector with the
property that all of the Y variables can be predicted from this vector optimally by least
squares.
<p>
A second latent vector is derived which represents a linear combination of the X residuals
after projects on to the first latent variable. This vector has the property of optimally
predicting the Y residuals from the first stage.
<p>
This procedure is continued until the contribution of a new latent vector is negligble.
<p>
Details of the PLS algorithm have been given by Aastveit and Martens (1986), amongst
others.
<P>
IDENTIFYING OPTIMAL FACTORS
<BR>
Cross-validation (Stone, 1974) can be used to access the appropriate
number of factors to fit.
<P>
The data are split into groups. One group of observations is
ommitted and the PLS model is fitted for the remaining data.
<P>
Predictions are made for the ommitted data and the sums of
squares of predicted minus observed is calculated :
<P>
The procedure is repeated until each observation has been deleted once.
<P>
The total sums of squares of predicted minus observed is used as a
measure of the predictive usefulness of the factor.
<P>
MODELLING G x E INTERACTIONS
<BR>
PLS can be used to explore some of the mechanisms which make up the interaction terms in
the variety x centres data table.
<P>
BREAKING DOWN THE INTERACTION
<BR>
The PLS method as applied here has similarities to principal component analysis of
interactions. PLS has the advantage of directly involving the external information in the
fitting of the multi-linear model.
<p>
In effect PLS may be best understood as separate PCA of X and of Y at the same time with
component estimates providing prediction of Y from X.
<P>
ADVANTAGES/ WEAKNESSES OF PLS
<BR>
<h2> ADVANTAGES </h2>
<ul>
<li>Flexibility of algorithm
<ul>
<li>Size dimensions do not matter e.g. there can be more variables than
observations;
<li>Some versions of the algorithm can accept missing data.
</ul>
<li>Can fit data with fewer components than other modelling approaches.
</ul>
<p>
<h2> WEAKNESSES </h2>
<ul>
<li>Distributional properties of estimates not known<P>
<li>Sometimes difficult to interpret loadings.
</ul>
<P>
<BR>
PLS is a relatively flexible algorithm which produces a minimal number of factor solutions
when attempting to relate one complex matrix to another.
<p>
Although it has a well established heuristic base the distributional properties of estimators
from the PLS are not known.
<p>
It can be difficult, sometimes, to interpret loadings. This is particularly the case where
judgement is needed in the scaling of variables before applying PLS, for example, with binary
variables where there may be no natural scale.
<P>
ALTERNATIVES TO PLS
<BR>
PLS has some similarities with PCA plus regression (PCR).
Both methods project the Y data into a multi-dimensional space determined
by vectors or factors and use coordinates in this space as regressors with
the X variables as regressands. But while PCR projects into a space
estimated by the Y variables alone, PLS projects into a space determined
by both the Y and X variables.
<p>
Canonical correlation tries to find linear combinations of the X variables which correlate
maximally with the linear combination of the Y variables. However, it is sensitive to strong
correlations in either the X or Y variables and the number of variables must be less that the
number of observations.
<p>
Ridge regression (RR) is a method for stabilizing regression estimates when
there is extreme collinearity amongst the variables. Multiresponse RR
applies separate RRs to each principal component's linear combination
of the responses using separate ridge parameters for each one.
<p>
The connection between PLS, PCR and RR is well described in a paper in
by Ildiko Frank and Jerome Friedman, "A statistical view of some
chemometrics regression tools" published in Technometrics, 1993, 35(2),
109-135.
<P>
GENSTAT PLS PROCEDURE
<BR>
A GENSTAT 5 procedure, PLS, written by <a href="address.htm"> Ian Wakeling
</a> and <a href="address.htm"> Nick Bratchell</a>, fits a partial least squares
model. <p>
The following is an extract from their description of the procedure : <p>
<ul>
<li> <a href="slide17b.htm"> The PLS model and the fitting process </a>
<li> <a href="slide17c.htm"> Output from the PLS procedure </a>
<li> <a href="slide17d.htm"> The PLS procedure options </a> <p>
</ul>
The <a href="../example/gen.dt"> potato data </a> and the GENSTAT <a
href="../example/gen.cod"> instructions </a> for fitting the PLS model to
these data are available.
<P>
THE PLS MODEL AND ITS FITTING
<BR>
Procedure description by <a href="address.htm"> Ian Wakeling </a> and
<a href="address.htm"> Nick Bratchell </a> <p>
If Y and X denote matrices of dependent and independent variablesrespectively,
then the aim of PLS is to fit a bilinear model having the form T=XW, X=TP'+E
and Y=TQ'+F, where W is a matrix of coefficients whose columns define the PLS
factors as linear combinations of the independent variables.
Successive PLS factors contained in the columns of T are selected both to
minimise the residuals in E and simultaneously to have high squared
covariance with a single Y variate (PLS1) or a linear combination of multiple
Y variates (PLS2). The columns T are constrained to be mutually orthogonal.
See <a href="ref.htm#helland"> Helland (1988) </a> or <a
href="ref.htm#hoskuldsson"> Hoskuldsson (1988) </a> for a more comprehensive
description of the PLS method. <p>
The procedure allows the calculation of PLS1 and PLS2 models with cross-
validation to assist in the determination of the correct number of dimensions
to include in the model. By setting the NGROUPS option the data are randomly
divided into a number of groups; samples in each group are then modelled from
the remaining samples only. The sum of squares of differences between these
"leave out predictions" and the observed values of Y are called PRESS. Many
tests of significance for determining the correct number of dimensions are
based on comparing values of PRESS for PLS models of varying rank. Values of
PRESS are used in the procedure to perform <a href="ref.htm#osten">
Osten's (1988) </a> test of significance and may also be plotted out in a
scree diagram. In addition to the factor scores, factor loadings and residuals,
the procedure also calculates a leverage measure <a href="ref.htm#naesmart">
(Martens & Naes 1989 page 276) </a> and a single linear combination of the X
variables (ESTIMATES) which summarises the entire PLS model. <p>
To use a PLS model to make predictions from new observations on the X
variables, two methods are available. Either the user may do this manually
by using the model as specified in the estimates matrix, or the new X data
may be specified beforehand as the pointer to variates XPREDICT and the
corresponding predicions obtained as YPREDICT. <p>
Although the PLS method is often presented in terms of an iterative
algorithm (Manne 1987), the X block loadings vector for the first PLS
dimension (w1) is simply the eigenvector of X'YY'X corresponding to its
largest eigenvalue. To find the second and subsequent dimensions, X and Y
are deflated by orthogonalising with respect to the current PLS factor (t=Xw)
and the eigenanalysis repeated. The above approach was adopted by <a
href="ref.htm#rogers"> Rogers (1987) </a> in an implementation of a Genstat 4
macro. Here we adopt a very similar approach by performing a singular value
decomposition on the matrix X'Y which simultaneously obtains loading vectors
for both data blocks <a href="ref.htm#hoskuldsson"> (Hoskuldsson 1988). </a>
<P>
OUTPUT FROM THE PLS MODEL AND ITS FITTING
<pre>
Output from the PLS procedure can be selected using the following settings of
the PRINT option.
data the unscaled data values (with labels).
xloadings X-component loadings (columns of the matrix W - see above).
yloadings variable loadings for the bilinear model of the matrix of
dependent variables. Note that these are standardized to unit
length and are not the same as the columns of the matrix Q
above. To obtain Q form the matrix C, whose columns are the
standardized loadings and post-multiply by the diagonal
matrix supplied as the output parameter B.
ploadings variable loadings for the bilinear model of the matrix of
independent variables (columns of the matrix P - see above).
scores X and Y component scores. The X component scores are the
columns of the matrix T and are mutually orthogonal. The Y
component scores, usually given the symbol u, are not in
fact needed in the calculation of the PLS model unless an
iterative algorithm is used (see method section). They are
provided here for completeness, as sometimes it is useful to
plot the Y component scores against the X component scores
to give a visual indication of the degree of fit for each
PLS dimension.
leverage measure of leverage.
xerrors residual sum of squares and residual standard deviations for
all the independent variables. When NGROUPS>1 additional
statistics are calculated from the cross-validated residuals,
derived when each object is left out. The PRESS value
is equal to the sum of squares of cross-validated standard
deviations for each X variable multipled by N-1, where N is
the total number of observations. The cross-validated
standard deviations may therefore be used to measure the
predictive ability of the model for each of the variables.
yerrors residual sum of squares and residual standard deviations for
all the dependent variables (see xerrors above).
scree scree diagram of PRESS.
xpercent percentage variance explained for the X variables.
ypercent percentage variance explained for the Y variables.
predictions predicted values for any observations that were not included
in the PLS model but were supplied using the XPREDICT
parameter.
groups details of groupings used for cross-validation.
estimates estimated PLS regression coefficients.
fittedvalues fitted values from the PLS regressions.
The default settings are estimates, xpercent, ypercent, scores, xloadings,
yloadings, ploadings.
</pre>
<P>
PLS PROCEDURE OPTIONS
<pre>
The data for PLS are supplied using the X and Y parameters, as pointers to
variates containing the columns of the X and Y matrices. Other parameters
allow output to be saved in appropriate data structures.
'options'
PRINT = strings Printed output required (data, xloadings,
yloadings, ploadings, scores, leverage,
xerrors, yerrors, scree, xpercent, ypercent,
predictions, groups, estimates, fittedvalues);
default esti,xper,yper,scor,xloa,yloa,ploa
NROOTS = scalar Number of PLS dimensions to be extracted
YSCALING = string Whether to scale the Y variates to unit variance;
(yes, no); default no
XSCALING = string Whether to scale the X variates to unit variance;
(yes, no); default no
NGROUPS = scalar Number of cross-validation groups into which to divide
the data; default 1 (i.e. no cross-validation performed)
SEED = scalar A scalar indicating the seed value to use when
or dividing the data randomly into NGROUPS groups for the
factor cross-validation or a factor to indicate a specific set
of groupings to use for the cross-validation; default
takes the (scalar) value of NGROUPS
LABELS = text Sample labels for X and Y that are to be used in the
printed output; defaults to the integers 1...n where
n is the length of the variates in X and Y
PLABELS = text Sample labels for XPREDICT that are to be used in
the printed output; default uses the integers 1, 2 ...
'parameters'
Y = pointers Pointer to variates containing the dependent variables
X = pointers Pointer to variates containing the independent variables
YLOADINGS = pointers Pointer to variates used to store the Y component
loadings for each dimension extracted
XLOADINGS = pointers Pointer to variates used to store the X component
loadings for each dimension extracted
PLOADINGS = pointers Pointer to variates used to store the loadings for
the bilinear model for the X block
YSCORE = pointers Pointer to variates used to store the Y component
scores for each dimension extracted
XSCORE = pointers Pointer to variates used to store the X component
scores for each dimension extracted
B = matrices A diagonal matrix containing the regression
coefficients of YSCORE on XSCORE for each dimension
YPREDICT = pointers A pointer to variates used to store predicted Y values
for samples in the prediction set
XPREDICT = pointers A pointer to variates containing data for the
independent variables in the prediction set
ESTIMATES = matrices An NX+1 by NY matrix (where NX and NY are the numbers
of variates contained in X and Y respectively) used to
store the PLS regression coefficients for a PLS model
with NROOTS dimensions
FITTED = pointers Pointer to variates used to store the fitted values for
each Y variate
LEVERAGE = variates Variate used to store the leverage that each sample has
on the PLS model
PRESS = variates Variate used to contain the Predictive Residual Error
Sum of Squares for each dimension in the PLS model,
available only if cross-validation has been selected
RSS = variates Variate used to store the Residual Sum of Squares for
each dimension extracted
YRESIDUAL = pointers Pointer to variates used to store the residuals from the
Y block after NROOTS dimensions have been extracted,
uncorrected for any scaling applied using YSCALING
XRESIDUAL = pointers Pointer to variates used to store the residuals from the
X block after NROOTS dimensions have been extracted,
uncorrected for any scaling applied using XSCALING
XPRESIDUAL = pointers Pointer to variates used to store the residuals from the
XPREDICT block after NROOTS dimensions have been
extracted
It is usual to centre all variables prior to a PLS analysis, the procedure
will automatically do so even if the XSCALING/YSCALING options are not set.
On exit from the procedure the variates pointed to by X and Y are unchanged.
</pre>
<P>
]]>