The important in this theorem is that f should be continuous in [a,b].
If your function, defined in (a,b), can be continuously extended in [a,b]
you can apply the theorem.
For a counter example look at f(x)=1/x defined in (0,1). Even though f is continuous in (0,1) you cannot extended it continuously in [0,1] thus you cannot apply the W. theorem
The important in this theorem is that f should be continuous in [a,b].
If your function, defined in (a,b), can be continuously extended in [a,b]
you can apply the theorem.
For a counter example look at f(x)=1/x defined in (0,1). Even though f is continuous in (0,1) you cannot extended it continuously in [0,1] thus you cannot apply the W. theorem
First, look at the behavior of the function at the boundary values "a" and "b". If the function happens to be continuous at both [a,b] and (a,b) then you can surely apply the theorem. Continuity of a function can be check with three basic steps, a function y = f(x) is continuous at point x=a if: i) f(a) is defined, ii) ii.) lim_{x ->a} f(x) exists and is finite, and iii) lim_{x ->a} f(x) = f(a). I hope this helps, good luck.
Actually, this is related to the question whether the given function is uniformly continuous on (a,b) or not. If yes, then you can extend the function to a continuous function on [a,b], if not, you cannot. Just remember that a uniform approximation by uniformly continuous functions (in our case polynomials) yields a uniformly continuous function. Actually, this is not much different from the answers you got so far, but it points out where the trouble is coming from.
It is possible to generalise the Weierstrass approximation theorem so that continuous functions on (0,1) can be approximated by polynomials uniformy on closed subintervals of (0,1).
The normed topology of uniform convergence in [0,1] is replaced by the seminormed compact-open topology on (0,1).
If the function is continuous at the end points a and b, then the Weierstrass theorem is true over the closed interval and thus is also true over the open interval. Otherwise, it is not guaranteed that the function can be approximated uniformly over the open interval (a,b).
If the function is not continuous over [0,1] but has bounded variation over [0,1] then the Weierstrass theorem is satisfied at points of continuity and the Bernstein approximation polynomial converges to the average of limits from left and right at points of discontinuity. The last article in my list of publication (On Hermite-Fejer interpolation ...) explains a similar situation.
A closed interval [a, b] can’t be replaced by open interval (a, b) in Weierstrass Approximation Theorem!
Classically this well known result of Weierstrass that any continuous function on a closed and bounded interval can be approximated by a polynomial function is phrased as the set of all polynomial functions defined on [a, b] is dense in the space of all continuous functions defined on [a, b] with the uniform norm.
The trick for proving the Weierstrass Approximation Theorem is to construct very sharply peaked polynomials that integrate to 1 on [a; b]. This can be done because the interval is closed and bounded.
This theorem shows that the approximation holds on the space C [a; b] with its natural norm. Formally this means that the approximation is point wise.
There have been lots of good answers already, (especially by @Aurelian Gheondea) but here is an example why things go completely wrong on an open interval. W.l.o.g. assume a = 0, b = 1. Consider a continuous function f that blows up at 0, e.g. f(x) = 1/x. Any polynomial P on (0,1) extends to [0,1] and so is bounded. Thus
|| f - P ||_\infty = sup_{x \in (0,1)} |f - P| = \infty
You can approximate f on every increasing sequence of subintervals [\epsilon, 1- \epsilon] as well as you like but that just does not get you closer to approximating such a function f in the sup norm. This is why for most approximation problems of continuous functions on topological spaces people use the topology of uniform approximation on compacta (or a cofinal collection of compacta, i.e. a collection of compacta such that for each compactum you can find a larger compactum in the cofinal collection, eg. on the unit interval the collection of all closed intervals or the collection of intervals of the form [\epsilon, 1-\epsilon] , as in the answer of @Miguel Carrion Alvarez)