It can't be decided systematically. The problem is known

as halting problem. Related is Rice's theorem. It says

that any nontrivial property of (recursively enumerable)

sets isn't decidable.

http://en.wikipedia.org/wiki/Rice%27s_theorem

A professor asked us: "What's the difference between

an operating system and a correct program?"

It was a rhetorical question.

How the halting problem is solved at the MIT:

http://web.mit.edu/newsoffice/2011/infinite-loops-interrupt-0802.html

Regards,

Joachim

Your question: Does a case exists where the algorithm does not correctly halt with either a yes or a no answer?

This is undecidable. In simple cases, one can use loop invariant or invariant before and after a recursive call (recursive sorting) to prove the correctness of an algorithm. In general, as Joachim pointed out this cannot be determined.

Best regards.

Thank You very much, colleagues!