Principal Components Analysis Example

18 Workstations

n = 18

Vector of number of packets sent per workstation

(Subscript[x, s])  =  {7718, 6958, 8551, 6924, 6298, 6120, 6184, 6527, 
          5081, 4216, 5532, 5638, 4147, 3562, 2955, 4261, 3644, 2020}

Vector of  number of packets received per workstation

(Subscript[x, r])  =  {7258, 7232, 7062, 6526, 5251, 5158, 5051, 4850, 
          4825, 4762, 4750, 4620, 4229, 3497, 3480, 3392, 3120, 2946}

\(There\ is\ a\ considerable\ correlation\ between\ \(x\_r\) 
          and\ x\_s . \ Compute\ y\_ki\ , 
        1 ≤ k ≤ \ 
          2\ \ \ \ so\ that\ successive\ y\_k\ vectors\ provide\ next\ best\ discriminatory\ power\)

Steps in Principal Component Analysis

1.  Compute the mean and standard deviation of (Subscript[x, r]) and (Subscript[x, s])

\(\(x\_r\)\&_\  = \ N[Mean[x\_r]]\)

\(\(x\_s\)\&_\  = \ N[Mean[x\_s]]\)

(Subscript[s, (Subscript[x, r])]) =  N[StandardDeviation[(Subscript[x, r])]]

(Subscript[s, (Subscript[x, s])]) =  N[StandardDeviation[(Subscript[x, s])]]

2.  Normalize the input variables to zero mean and unit standard deviation

\(x\&^\_s = \ N[\(x\_s - \(x\_s\)\&_\)\/s\_\(x\_s\)\ ]\)

\(x\&^\_r = N[\ \(x\_r - \(x\_r\)\&_\)\/s\_\(x\_r\)]\)

\(StandardDeviation[x\&^\_r]\)

3.  Compute the correlation between the variables

	First,  define covariance.

\(\(\(			\)\(Cov\  = \ 
      Function[{x, y}, 
        Total[\((x\  - \ Mean[x])\) \((y\  - \ Mean[y])\)]\/\((Length[x] - 1)\)]\)\)\)

Covariance  >  1 means as x increases,  
          y increases;  Covariance  <  1 means as x increases,  
          y decreases

		Notice that Cov[x, x]  =  Variance[x]

\(\(\(			\)\(N[Cov[x\_s, x\_s]]\)\)\)

\(\(\(			\)\(N[Variance[x\_s]]\)\)\)

Define Correlation  in terms of covariance

\(\(\(		\)\(Cor\  = \ 
      Function[{x, y}, \ 
        N[Cov[x, 
              y]/\((StandardDeviation[x]\ *\ 
                StandardDeviation[y])\)]]\)\)\)

\(Cor[x\&^\_r, x\&^\_r]\)

4.  Prepare the correlation matrix

	In general, ((Cm)^nxn)= ((Subscript[Cm, i, j])=(Subscript[Cm, j, i])= Cor[(Subscript[x, i]), (Subscript[x, j])])

Cm = (({{Cor[(Subscript[x, s]), (Subscript[x, s])], Cor[(Subscript[x, s]), (Subscript[x, r])]}, {Cor[(Subscript[x, r]), (Subscript[x, s])], Cor[(Subscript[x, r]), (Subscript[x, r])]}}))

\(\(\(
\)\(		\)\(Cm\  // MatrixForm\)\)\)

5.  Compute the eigenvalues of the correlation matrix

Solve the characteristic equation                             Det[λ IdentityMatrix[2]  -  Cm] ==  0

\(\(\(		\)\(Em\  = \ 
      Solve[Det[λ\ IdentityMatrix[2]\  - \ Cm] == \ 
          0, λ]\)\)\)

\(\(\(		\)\(Eigenvalues[Cm]\)\)\)

6.  
          Compute the eigenvectors for each of the eigenvalues of the correlation matrix

	For each eigenvalue (Subscript[λ, i]), solve the  the characteristic equation for the eigenvector (Subscript[q, i])                             Cm (Subscript[q, i]) == (Subscript[λ, i]) (Subscript[q, i])

\(\(\(		\)\(Eigenvectors[Cm]\)\)\)

7.  
        Obtain the principal factors by multiplying the eigenvectors by the normalized vectors

\(MatrixForm[{\ x\&^\_s, \ x\&^\_r}]\)

\(\(\(\ \)\(y\  = \ 
      Eigenvectors[Cm]\  . \ \ {\ x\&^\_s, \ x\&^\_r}\)\)\)

(Subscript[y, [[1]]])

(Subscript[y, [[2]]])

The two principal factors should be orthogonal,  
      namely their dot product should be 0

(Subscript[y, [[1]]])  .  
      (Subscript[y, [[2]]])

8.  Compute the sum and sum of squares of the principal factors

\(\(\(	\)\(Total[
      y\_\(\([[\)\(1\)\(]]\)\)]\)\)\)

\(\(\(	\)\(y\_\(\([[\)\(1\)\(]]\)\)\  . \ y\_\(\([[\)\(1\)\(]]\)\)\)\)\)

\(\(\(	\)\(Total[
      y\_\(\([[\)\(2\)\(]]\)\)]\)\)\)

\(\(\(	\)\(y\_\(\([[\)\(2\)\(]]\)\)\  . \ y\_\(\([[\)\(2\)\(]]\)\)\)\)\)

Variation explained by factor (Subscript[y, [[1]]])  =  
          (Subscript[y, [[1]]])  .  
              (Subscript[y, [[1]]]) / ((Subscript[y, [[1]]])  .  
                  (Subscript[y, [[1]]])  +  (Subscript[y, [[2]]])  .  
                  (Subscript[y, [[2]]]))

\(\(\(	\)\(y\_\(\([[\)\(1\)\(]]\)\)\  . \ y\_\(\([[\)\(1\)\(]]\)\)\ /\ \((y\_\(\([[\)\(1\)\(]]\)\)\  . \ 
            y\_\(\([[\)\(1\)\(]]\)\)\  + \ y\_\(\([[\)\(2\)\(]]\)\)\  . \ 
            y\_\(\([[\)\(2\)\(]]\)\))\)\)\)\)

Variation explained by factor (Subscript[y, [[2]]])  =  
          (Subscript[y, [[2]]])  .  
              (Subscript[y, [[2]]]) / ((Subscript[y, [[1]]])  .  
                  (Subscript[y, [[1]]])  +  (Subscript[y, [[2]]])  .  
                  (Subscript[y, [[2]]]))

\(\(\(	\)\(y\_\(\([[\)\(2\)\(]]\)\)\  . \ y\_\(\([[\)\(2\)\(]]\)\)\ /\ \((y\_\(\([[\)\(1\)\(]]\)\)\  . \ 
            y\_\(\([[\)\(1\)\(]]\)\)\  + \ y\_\(\([[\)\(2\)\(]]\)\)\  . \ 
            y\_\(\([[\)\(2\)\(]]\)\))\)\)\)\)

9.  
        Plot the values of the principal factors

\(\(\(	\)\(\(ListPlot[Transpose[y], \ 
      AxesLabel -> \ {"Principal factor 1", \ "Principal factor 2"}, \ PlotRange -> {\ {\(-4\), 4}, {\(-4\), 4}}, \ 
      PlotStyle -> PointSize[0.04]]\)\(
\)
    \)\)\)

Most variation associated with first principal factor;  
      it can be used to classify  workstations as low,  
      medium or high load workstations