I have two algorithms. The first one has a low time and high mean square errors but the other one has a high time with low mean square error. How can I decide the best ? is there a method to compare between them ?
I am not sure I understood your question,It depends for what you want, It is your decision,if you need accurate results you should relay on the one have low mean square error but the CPU time is high on that case you should be careful for the specifications of the computer you use (CPU, RAM). On the other hand if you do not care of the accuracy of the results you choose the one with low accuracy and less CPU time, you can say generally (Accuracy and CPU are two main measures for the efficiency of the algorithms).
There is a nice way to visualize such tests on algorithms - in particular one that weighs accuracy over time spent:
http://arxiv.org/pdf/cs/0102001.pdf
This is - I think - the last version of the manuscript before it was published in the journal Mathematical Programming. If you do not have access to the published paper online, the above pdf should include all the details anyway.