In case A Matrix is positive definite, we have X'*A*X >0 , for all X non-zero

defining M = (inv(W) - K*inv(K'*W*K)*K') , we have A=K'*M*K, therefore,

X'*A*X = X'*K'*M*K*X, defining K*X =Y we have X'*A*X =Y'*M*Y , therefore,

X'*A*X >0 , for all X non-zero implies Y'*M*Y >0 , for all possible Y, which

contradict the simulation result that M is Negative definite.

Dear Ho Minh Dai

The notations used in your question are not clear!!

Try to rewrite your question in a clear format.

If Peter's guess is true for your question, then your matrix is the zero matrix, and consequently, it is not positive definite.

Best regards

I agree with Issam Kaddoura's answer.

Dear Debopam Ghosh Peter Breuer Issam Kaddoura Mehryar Husyan Pour Shad

Thank you very much for answering my question. I will rewrite the question as follows:

If W is a diagonal matrix whose diagonal elements are positive, and matrix K has full rank. Prove that

A = KT(W-1 -K(KTWK)-1KT)K

is a positive definite matrix.

Best regards,

Ho Minh Dai

Dear Ho Minh Dai

Unfortunately, the matrix A is not a positive definite matrix

Counterexample: K is full rank

KT = [ 1 2 1

0 0 1]

and W = diag(a , b, c )

after calculation , we obtain

A = [4(a-b)²/(ab(a+4b) 0

0 0]

But we can prove the following:

Lemma: A = AT, i.e., A, is symmetric with real eigenvalues.

Proof: is direct by using W = WT, W-1 = (W-1 )T (diagonal matrices)

Best regards

Dear Issam Kaddoura

Yes, as you said, the matrix A is not a positive definite matrix. I was mistaken.

Thank you so much for the help.

Best regards,

Ho Minh Dai

Your matrix A is nonnegative definite, that is $x^TAx \ge 0$ for any $x \ne 0$.

This is obvious from the following simple identity:

$$

A = K^T (I - K(K^TWK)^{-1}K^TW) W^{-1} (I - WK(K^TWK)^{-1}K^T) K

$$

Note that, $Ax=0$ if $x$ satisfies the equation $Kx=WKy$ for some $y$.

The example in my answer 2 days ago, show that

A is neither positive nor a nonnegative definite matrix.

It has the property A = AT, and consequently,

it has positive or negative real eigenvalues.

The 2x2 matrix

A = [ d 0

0 0]

with the only positive element d in (1,1) position and the remaining 3 zeros

is nonnegatve definite matrix because it is real symmetric and its two eigenvalues

are d and 0. I cannot observe any negative eigenvalues of A

I. E. Kaporin

The identity is true. But how you show that A is nonnegative definite using this identity?

@Issam Kaddoura

Given $x$, denote

$$

z=(I - WK(K^TWK)^{-1}K^T) Kx

$$

Then obviously

$$

x^TAx = z^TW^{-1}z

$$

Since $W$ is symmetric positive definite, $x^TAx$ is nonnegative

and it turns to zero if and only if $z=0$

@ Ho Minh Dai

Unfortunately, for the sizes indicated, that is $K$ be an (2n-1)xn matrix of full column rank,

the matrix A will always be singular. This is so because the equation $Kx=WKy$ always has

a solution with nonzero $x$ and $y$ in this case.

The matrix A can be positive definite only if $n+n \le m$, where $m$ is the first dimension of $K$.

Indeed, the sufficient condition for A to be positive definite is the full column rank of $ [ K | WK ] $.

@ Peter Breuer

The topicstarter himself wrote:

A = K'*(inv(W) - K*inv(K'*W*K)*K')*K

and it can be rewritten identically as

A = K^T (I - K(K^TWK)^{-1}K^TW) W^{-1} (I - WK(K^TWK)^{-1}K^T) K

The latter explains everything

Dear Ho Minh Dai

More investigation of your matrix leads to the following observation:

For any n, A has exactly one positive eigenvalue and all other eigenvalues are zeros. Several examples for different values of n confirm the nonnegative definiteness of A.

Best regards

Dear Peter Breuer Issam Kaddoura I. E. Kaporin

Thank you for your interest in the questions.

My goal is to prove that the WLS method is more efficient than the OLS method, which means that the diagonal elements of the covariance matrix for estimating the linear regression coefficients obtained by the WLS method are no more than the diagonal elements of the covariance matrix obtained by the OLS method. Therefore, I want to prove that the difference between the covariance matrices obtained using OLS and WLS is non-negative definite. I. E. Kaporin's answer helped me prove this.

Best regards

Dear Ho Minh Dai,

It was a great pleasure to help you!

Best regards,

I. Kaporin

Dear I. E. Kaporin

By the way, could you please respond to the question below: what is the condition for matrix A to be positive definite?

The matrix K has dimension (2n-1)xn, not (2n-1)xn. Please explain to me the following:

“... The matrix A can be positive definite only if n+n≤m, where m is the first dimension of K.” (Please could you refer me to an articles or books where I can find such property above).

Best regards,

Ho Minh Dai

Dear Peter Breuer

By the way, could you please respond to the question below: what is the condition for matrix A to be positive definite?

Is that if n > 2, n + n ≤ 2n - 1, then does the matrix A always be positive definite?

However, if the matrix W = a2I, where I is a identity matrix, then A will become the zero matrix for all n ≥ 2.

Can I conclude that if n > 2 and W do not have the form a2I, then A is always positive definite?

Best regards,

Ho Minh Dai

Dear Peter Breuer

Sorry, can you explain to me what im() and /\ stand for in im(WK)/\im(K)=0?

Best regards,

Ho Minh Dai

You may follow : https://www.youtube.com/watch?v=cfn2ZUuWPd0 for understanding positive definite matrices. In terms of data science please follow : https://towardsdatascience.com/what-is-a-positive-definite-matrix-181e24085abd

Dear Peter Breuer,

I am totally agree with you. I would like to mention, here, in RG no one have sufficient time to solve assignment problems. In such scenarios one may suggest some links, to understand and solve such problem itself. You may have noticed some beginner researchers asking about their projects with open problems to answers via RG members at the very beginning. Probably they do not understand that it is their own PhD project to initiate and solve....however., we may say experience is a good teacher...it getting better with the passage of time.