If A and B Are non-singular then we have determinant of (A^2-B^2)=-determinant of AB , Am I correct? If so, then the problem is If A and B are singular, how can we prove?
The relations given above clearly say that det ( A) = det (B). I do not know how you infered det(A^2-B^2) = - det(A. B) ? Here either both are singular or both are non-singular.
Given your assumptions, the case that either B or A is non-singular can not happen.
Starting from A^3=B^3 multiply by B from the right and insert A^2B=AB^2. This gives A^2B^2=B^4. If B would be non-singular, we can obtain A^2=B^2, which is excluded in the assumption. By symmetry, the same statement is valid for A.
Clearly det(A) =det(B)=0 since if either is invertible then the other must be as well (since A^3=B^3 so det(A)^3=det(B)^3) and so A^2B=AB^2 reduces to A=B which is a contradiction. Now, with det(A)=det(B)=0 and the below statement:
O.k so after some thought it is quite clear from assumtions that A and B can't be of full rank so that det(A)=det(B)=0.
This is because the fact that since if det(A)=det(B) \neq 0 then this would imply the existence of the inverse of A and B so that the equation
A2 B=AB2 would result after multiplication by A-1 from the right and after multiplication by B-1 from the left to equation A=B, which would be a contradiction to the original claim A does not equal B. So the only possibility is det(A)=det(B)=0 ???
So looked like I only complicated things...
So in fact I think the last commentor Nitin Bisth is correct in his claim.
Nothing can be said about det(A^2-B^2).
Prasanth are you sure you have the facts dorrect ? Could you double chek them ? Of course it might be that my mind is again playing tricks on me :-).