No, it does not solve the problems with stepwise selection methods. Everything you said about stepwise is correct, but I'd like to add what I see as the biggest problem is that in many cases we don't have very clear or rigorous criteria for whether to include a predictor in the model or not, so each individual step may not be generalizable. That is, if you sample data from the same population multiple times, and run a stepwise selection with the same exact predictors on each sample, you can (and, in fact, almost certainly will) get DIFFERENT end-models. 

Now, running an ANOVA on the same set of data after model selection won't be confirmatory (except in a technical sense, like checking to make sure the code for model selection worked). In fact, you are almost guaranteed to get the same conclusion running an ANOVA on that data, although it depends on exactly what type of model you used in the model selection process, but I'm assuming based on the context you had a fairly simple regression framework. That is, when you run an ANOVA, you aren't running a DIFFERENT model, you are essentially running the SAME model but with a different parameterization. If anything, it would be more surprising and revealing about the data if your ANOVA disagreed with the stepwise-selected regression model. 

The only way ANOVA would be used as confirmatory in this situation is if you applied the ANOVA to a DIFFERENT data sample from the same population. That is, you fit the stepwise selection method on one sample (your 'training' set) and confirm the results by applying ANOVA to a different sample (your 'validation' set). 

Thanks, Ryan - I did feel a bit like the ANOVA was 'fruit of the poisoned tree' but I wasn't sure. 

When I was reading about this online, it said that it was difficult to explain WHY step-wise is such a bad idea.  I appreciate your help elucidating the problem!

-J

HI Jaime

The stepwise model is fine for a preliminary look at one's data, and for "weeding" out variables that aren't significant when adjusted for the effects of other variables in the model. This is, in fact, what stepwise models were designed to do.  It is NOT a remedy for too many variables and not enough data.  In fact, the adjusted R2  should have been quite a bit lower than the non-adjusted.  I'm actually surprised that the ANOVA results were the same! DIdn't all those other variables adjust the effects of the bivariate relationship? 

But there are bigger problems here, if I read your question correctly. If understand correctly, the D.V. has three categories. To me that sounds ordinal, not interval, and shouldn't be used with OLS in the first place (or ANOVA either, for that matter.). The appropriate procedure would be a reasonably constructed  ordinal logistic regression model or, failing that, a multinomial one.  

The stepwise model is bad, but I think there are actually bigger problems with this analysis. 

Hope that helps

--BF

Thank you, WIlliam. Is there any scholarly article or textbook source you can refer me to that I can use to cite this information?  "I found it on the internet" is not always acceptable to those in the scientific community.  ;)

-J

I'm running survey data right now. I'm actually using a Stepwise method right now. I'm using it to see what is "significant". Then, I am going back and looking at those significant terms. If something is significant with 5-10 people, out of 2,000, I'm removing that term from consideration and rerunning the stepwise procedure. I'm also looking at VIF among the terms too. I wish I could use a best-subsets method:-(

I'm using Ordinal Logistic Regression on my data. It's rated 1 to 5. Ordinal Logistic Regression is specific to data where the response is ordered. As long as 2>1, 3>2, 4>3, 5>4, OLR will "work". I did choose demographic data that is related to my female friends as a base line/reference group. I'm not using any follow-up procedures. I'm making references  back to the reference demograhics.

Thanks again to everyone who replied.  I was able to make a convincing argument thanks to your expertise!  :)