Dear ahamed,

You can go alternative formulation for the PCA rather than SVD,.....both diagonalization is zero,, if it were not very small amount of.  Floating -poijnt round of error.Alternative formulation matrix eigen vactor ,eigen vector sommtime row _mode,Column mode .

A

If you know the null-space L, can't you just perform the maximization over the orthogonal complement  of L? A and B are non-singular on that space. 

One could also maximize x^HAx under the constraint that xB^x=1. Then we will automatically avoid the nullspace of B.

The answer Axel gives makes sense to me, but the right answer depends on exactly what you want to achieve by getting the maximum of x^T A x / x^T B x, I think. So what made you try to find the solution of the problem, Ahmad?

Dear Axel, 

Thank you, as Sunghee and I think, your answer is very logical. 

Could you provide me an example by solving over the orthogonal complement L ? 

Thank you in advance

Yes Sunghee, 

I am trying to maximise over the orthogonal complement L. 

Thank you for your comment.

Thank you Jayanta

Dear Axel,

I should correct my previous comment to you. You can not just say that x belongs to the orthogonal complement of L. You can say that L belongs to the space excluding the null space, i.e. excluding L.

Hi again Ahmad,

yes, I think you are right, it is not as easy as I thought first. I think it is also not enough to consider the (set theoretic) complement of L if you don't make some additional assumptions.

My first concern is: don't we have to assume that the matrices are Hermitian in order to guarantee that the maximizing vectors are generalized eigenvectors?

Also, in order for the maximum to be corresponding to the maximal eigenvalue, does not B have to be positive semidefinite?

And finally, we have to assume that the null space of B is contained in the null space of A in order for the maximization to work at all? 

Hello Axel,

Yes that's true. A and B are both Hermitian and positive semi definite  (and for sure not positive definite since they have a common null space). 

Thanks again

OK, for completeness: IF they are Hermitian and we also assume that the nullspace over B is contained in the nullspace of A, we may perform the maximization over the orthogonal complement of null(B)  (in fact, any complementary space will do), not only over R^N \ null(B).

Short explanation: if x is any vector, we may write it as v+n, where Bn=0 (and then also An=0) and v and n are perpendicular, we have

(v+n)^HA(v+n)=v^HAv+n^HAv=v^HAv +(An)^Hv= v^HAv,

and also (v+n)^HB(v+n)=v^HBv. Hence, the maximization may equivalently be performed on the orthogonal complement of null B. On this space, B is positive definite, so the usual theorem about generalized eigenvectors and maximum of the function applies.

Dear Ahmad and Axel,

First, I think in vector space, the orthogonal complement of L is equivalent to the subspace excluding L in a sense (given a subspace L).

The vector space is not like a set and it's somewhat tricky to derive certain conclusions about vector space.

For the answer you were trying to seek, I can give you the following tangible solution.

Do SVD for B and assume that B = VDV^H where D is a diagonal matrix where the column vectors of V are the eigenvectors of B associated with the corresponding diagonal components of D as their eigenvalues. Since B is assumed to be positive semidefinite, the diagonal components of D are nonnegative real numbers. Without loss of generality, assume that V = [W Z] where the column vectors of W are the eigenvectors with nonzero (or equivalently, positive) eigenvalues and those of Z are the eigenvectors with zero eigenvalues. If the number of positive eigenvalues is k (having assume that B is in R^{n x n}), then W is in R^{n x k} and Z is in R^{n x (n-k)}. (By the way, the range of Z is, actually, the null space of B.)

Then any vector x in the orthogonal complement of B can be expressed as x = W y where y is in R^k. However, for the reason which will be clear in the following, we let x = W F z where z is in R^k, F is in R^{k x k}, and F is the diagonal matrix each diagnal component of which is the inverse of the square root of the corresponding (positive) eigenvalue. If we substitute x with W F z in f(x), we obtain

f(x) = z^H (F^H W^H A W F) z / ||z||^2

since x^H B x = z^H F^H W^H V D V^H W F z = z^H F^H D' F z = ||z||^2 where D' is a k-by-k major submatrix of D.

Hence, the problem becomes the problem of finding the maximum eigenvalue of the (positive semidefinite) matrix, F^H W^H A W F!

Having discussed so far, I am wondering whether we need an assumption of the containment relathionship between the nullspaces of A and B..

Thank you.