I have found that sometimes inverse and pseudo inverse of a covariance matrix are the same whereas it is not true always. is there any relation between pseudoinverse and nonsingularity?
Consider the 3x3 diagonal matrix D= diag(2,1,a) where a is a small positive number. Then D^(-1) = diag(1/2,1,1/a). Notice that 1/a can be very large if a is very close to zero. Now, computationally, we never work with real numbers; rather, we work with floating point numbers (essentially, real numbers modulo base changes, truncations, rounding,...). Suppose that your machine has an error tolerance eps. That is, when eps >= a >=0, your machine sets a = 0. In this case, D^(-1) no longer exists since 1/0 causes all manner of logic issues. On the other hand, the pseudo-inverse (typically the unique Moore-Penrose inverse) is well-defined, with pinv(D) = diag(1/2,1,0) for the given example. In theory, for any square matrix A, pinv(A) = A^(-1) when A is invertible, and when A is not invertible, pinv(A) inverts the invertible portion of A and suppresses the zero-eigenvalue component of A. More specifically, every square (nxn), real or complex matrix A has SVD (singular value decomposition) given by A = U S V where U and V are unitary (real and orthogonal when A is real; and further, U = V* when A is hermetian, and U = V^T when A is real and symmetric). The matrix S is diagonal with the diagonal entries nonnegative and in decreasing order s1 >= S2 >= ...>=Sk > 0. The rank of A is k, and when k is less than n. Then pinv(A) = V* pinv(S) U* where pinv(S) is the diagonal matrix diag( 1/S1, 1/S2, ... , 1/Sk, 0, ... 0). It turns out that there are very good algorithms for finding the SVD for real, symmetric matrices, which includes covariance matrices. These algorithms were developed in the 1960's by Gene Golub and others (see for example, Golub and Van Loan's book on matrix computations).
SO, to summarize, when A is invertible, the pseudo-inverse (and every other respectable generalized inverse) is, in theory, the same as the inverse of A. When A is not invertible (or when A is within a small distance from a matrix that is not invertible AND the null space of that nearby matrix is not close to orthogonal to another eigenspace), then the pseudo-inverse is much more robust than the inverse.
The pseudoinverse can be shown to solve an inconsistent Ax=b by finding the least squares "solution" that minimizes the 2-norm of Ax - b; and it solves a consistent Ax = b with multiple solutions by finding the unique solution x of minimum 2-norm.
Hope that helps.
The purpose of constructing a generalized inverse is to obtain a matrix that can serve as the inverse in some sense for a wider class of matrices than invertible ones. Typically, the generalized inverse exists for an arbitrary matrix, and when a matrix has an inverse, then its inverse and the generalized inverse are the same.
- Badges
- Science topic
- Similar topics
- Mathematics
- Algebra
- Matrix