if P is larger than N, I am not sure that the ML is still optimal because the Hessian matrix of the Likelihood Function could no more be definite positive, hence some variances of the estimated x() might become infinite

if P is less than N, the ML is definetely optimal with regard to bias and the variance of the estimated parameters, but if that is valid for estimation, it is not for prediction.

The ML becomes suboptimal when the test statistics are not sufficient to describe all variables, i.e., not all variables are well described by their respective probability density functions. Hence, some approximations would be necessary, such as quasi-LRT, which turns the overall ML detection process suboptimal.

In constrained problems (i.e. where the solution should fall within certain bounds) the ML method may become something of a problem in that the ML estimate lies outside of the bounds.

Also see 

http://www.econometrician.org/ut/attach/2002/05/27/158-fulltext.pdf

I think that you answered your own question. As long as the thing is a "Gaussian" you are golden. However there are things in the nature that are not such, for instance if you have your system dominated by shot noise, eg in you are doing imaging in very low light, the solution might be different. So to my opinion the first task is to recognize the dominating "interference" that you have in your signal... 

Why do you mean by optimal?  Do you mean that it is the Minimum Variance Unbiased (MVU) estimator? (for any vector of unknown deterministic variables: x).

In this case, if H(t) is a known (N x P) observation matrix (with N > p) and rank P, x is a (P x 1) vector of parameteres to be estimated, and w is an (N x 1) zero mean Gaussian noise vector with covariance matrix C_{w}, then the MLE can be expressed as:

(x)_{ML} = (H' * C_{w}^(-1) *H)^(-1)*H'*C^(-1)*y

And (x)_{ML} is the MVU and attains the Cramer-Rao lower bound.

Ref: Kay's book, Statistical Signal Processing Part I: Estimation Theory

to be gaussian or not has nothing to do with the optimality of ML, as long as you use the proper likelihood and not the gaussian likelihood (in every circumstances).

Dear Adam, 

Yes what you said is true.

Dear Jean, 

Thank you for your rich answer. I've seen what you said before when P > N.

What you said is true only when the noise is non-gaussian. If the noise were to be Gaussian, then even though P > N, we have that ML is optimal, i guess. 

But yes what you said is right

Dear Jean Maccarlo,

That's true only if P

Dear Miguel,

Thank you for your detailed response !! 

Dear Sami,

Yes what you said is logical

Thank you Petrus

Dear Ahmed,

I never knew that, thank you

Perhaps when there are two very large and "sharp" local maxima that are well separated in the state space but have a very similar likelihood. In these cases, you might be better off with an estimate that incorporates both maxima, not just the global maximum.

Most welcome, I hope it was insightful.

Just a point to add is that you need also to consider the statistics of the vector x(). The optimal estimator is the MAP (Maximum a posteriori). And it only degrades to ML when the vector x() is of uniform distribution.

Other than that, as all discussed it depends on the definition of the likelihood function and if n() is Gaussian.

I hope this helps.

Thank you Jochen

Yes Ahmed indeed it was

Thank you Youssef

In Non-Coherent detection (when we dont have transmitted phase exactly at reciver) , we have suboptimum detection although we used ML detector.

I mean ML detector in this condition will be sub-optimum

If you drop the Gaussian assumption for the additive noise, then ML could not be the optimum estimator. For example, the so-called robust estimators perform better under heavy-tailed noise processes (such as the alpha-stable noise family except for Gaussian) because they can more easily deal with the outliers.  

Dear Jochen,

To find answers to all your questions, I can suggest the following excellent review paper:

"Robust estimation in signal processing", A. M. Zoubir, V. Koivunen, Y. Chakhchoukh, and M. Muma, IEEE Signal Processing Magazine,  pp: 61-80, July 2012.

Thank you Mohammad

Thank you all for your useful answers