The best model is the one with the likelihood value that is closest to "0". In you case the best model is M1 as it has a lower mean.

Though, considering that M1 also had both the highest and lowest likelihood in its separate runs, I would do the test again adding more "runs". This will give you a higher level of confidence as to the best model.

Dear Russell,

Thank you very much.

I already have two trees to compare (I assume each tree as a model) one tree is under constraint and the other unconstrained. I just want to compare between them.

I tried tree puzzle software for the KH, and SH tests. It is quick and easy rather than analyses my data again analysis. What do you think?

Thanks again

After you perform marginal likelihood procedure your analysis is regular maximum likelihood analysis and you should apply regular likelihood ratio test (http://en.wikipedia.org/wiki/Likelihood-ratio_test).

Also there are some remarks:

1) It does not make any sense to calculate an average of likelihood values. What you need is product of likelihood values or sum of logarithms i. e. the combined probability of both data sets to appear simultaneously.

2) In general the hypothesis with larger likelihood is better (if it based on some physical considerations).

3) If you have continuous hypothesis distribution (parametric hypothesis) and likelihood dependence on that parameter is Gaussian, than -2ln(L) is a chi^2 distribution and you can use it features to make some quantitative results.

4) In general case the only way to provide correct quantitative rejection probabilities is simulation.