The eigenvectors obtained through LDA is orthogonal. Usually, an SVD or eigen decomposition is used to obtain the eigenvectors and eigenvalues. Therefore, the solution will be orthogonal.

For the second question, I think the figure you posted but with upside down should be the right curve for showing the PCA eigenvalues. In real applications such as the face recognition, the eigenspaces with top k(k

Hello Sir,

Thanks for the input ! I guess the reason is that my matrix was close to singular when i used eig( Sw_inv * Sb) which gave me the 'wrong eigenvectors'. Gonna have to look through the mathworks forum to understand and figure out a way around this. Thanks again.

Haziq

You may try using PCA before using LDA to avoid the ill-posed solution, which is a widely used technique. Also, a small diagonal can be added to Sw, i.e., calculate eig((Sw+\lambda *I) *Sb) instead. Good luck!

Sorry, it should be eig((Sw+\lambda *I)_inv *Sb) in the last piece of answer.

Eigenvectors of symmetric matrices are always orthogonal. Since the scatter matrix for LDA is symmetric by construction (as the covariance matrix for PCA is as well) your eigenvectors must be orthogonal. If they are not, you should check your implementation or look for numerical instabilities.

it is usually not orthogonal. LDA applies eigen decomposition to S_b/S_w. orthogonality of eigen vectors depends on the symmetry of covariance matrix. although both S_b and S_w are symmetric, S_b/S_w is usually not, which is why LDA discriminant vectors are always non-orthogonal.

Shuhui and Xiaoxiong very nicely rationalized. The LDA is not orthogonal this is an impossibility. Your calculatuions are well executed and concise.