My DV is measured two times per observations, once pre- and once post-treatment (interval scale). The IV has multiple levels and is categorical.
I am interested in how the different levels of the IV differ in their effect on changes from pre- to post-treatment.
First possibility:
An OLS regression of IV with change scores (post minus pre measurements) as DV.
Second possibility:
An OLS regression of IV and pre-treatment measurement of the DV with post-treatment measurement as the DV.
Third possibility:
An OLS regression of IV interacted with pre-treatment measurement of the DV with post-treatment measurement as the DV
What are the up- and downsides of these three approaches? Do they only differ with respect to the hypotheses they test, or are there any technical/statistical con's to some of these appraoches? Especially concerning the second possibility, I came accross criticisms because of endogeneity, i.e. correlation of unobserved effects in the error term with the first-round measurements of the DV as the IV. Also, the second and third possibility, in my specific case, have very high R-squared due to the inclusion of pre-treatment measurements, compared to the first possibility. I feel that this comes at some costs, though.
Thanks in advance
Hi,
Firstly, let us focus on the first choice and the second choice.
In the first choice: You actually model the change in score (y2-y1 where y2 is the measurement post treatment and y1 is the measurement pre-treatment) with respect to a covariate (let say x, and x is a categorical variable).
So the information that this model could be answered (after testing all the assumptions for a regression model and all these assumptions are satisfied) is what the relation between x with respect to the change in the score. So you could not capture the relation between the pre-treatment measurement and post-treatment measurement if any.
For the second option, it can have the advantage that you might see how the pre-treatment measurement could affect the estimate of the post-treatment measurement. I am not sure in which field are you working on but in many fields, the base line (the starting point) is very important to predict a measurement later on.
So personally, I favor the second option over the first one.
Now if we look at the second one and the third one, it is good to start a model where an interaction is there. In this case there are only two covariates (pre-measurement = y1 and x) so it is quite straightforward to look at the model with interaction and compare with the model without interaction to see which one in your case is better to answer the question at hand.
I hope it helps.
Thao.
Hi,
Firstly, let us focus on the first choice and the second choice.
In the first choice: You actually model the change in score (y2-y1 where y2 is the measurement post treatment and y1 is the measurement pre-treatment) with respect to a covariate (let say x, and x is a categorical variable).
So the information that this model could be answered (after testing all the assumptions for a regression model and all these assumptions are satisfied) is what the relation between x with respect to the change in the score. So you could not capture the relation between the pre-treatment measurement and post-treatment measurement if any.
For the second option, it can have the advantage that you might see how the pre-treatment measurement could affect the estimate of the post-treatment measurement. I am not sure in which field are you working on but in many fields, the base line (the starting point) is very important to predict a measurement later on.
So personally, I favor the second option over the first one.
Now if we look at the second one and the third one, it is good to start a model where an interaction is there. In this case there are only two covariates (pre-measurement = y1 and x) so it is quite straightforward to look at the model with interaction and compare with the model without interaction to see which one in your case is better to answer the question at hand.
I hope it helps.
Thao.
your 3rd is tricky because the interaction do not allow to assess simply a "treatment" effect, which straightforwardly the effect od treatment in your 2nd option. For this reason I would advocate 2 upon 3.
The method 1 is to be avoided in general because it is equivalent to 2 with a "slope" nailed at one instead of being deduced from the observations - a value of this slope below one occurs very frequently and is known as the regression to the mean phenomenon, quite often seen when experimental errors are of some importance. You may have a good hint of the phenomenon when comparing the slope of method 2 to one (no test please).
But what about the problem of endogeneity I mentioned?
Post-test scores (DV) are affected by pre-test scores (IV), plus DV (NOT IV) is potentially affected by the treatment (the changes from pre- to post caused by the treatment are what I am interested in). However, the error term is highly correlated with the IV, since it captures unobserved factors of both the DV and IV (I am not that good a statistician to say that I fully understand this argument, but I think my intuition gets the basic concept).
The reason why I am sceptical is that a regression with change scores as DV (in MY case) and Treatment as IV has a very low R². So does a regression with the DV being post-treatment measurements and Treatment as IV (without pre-treatment measurements as IV). However, as soon as I additionally include pre-treatment measurements as an IV (additional to treatment IV), the R² gets very high. I know this is because, in my case, pre- scores are very predictive of post-scores (and this is not really interesting). I just feel that I am "manipulating" my model to "look good" (high are squared), when acutally my treatments are not very good in explaining the outcome variable (due to very high variance and a generally low treatment impact).
Before all I should apologize if I didn’t understand precisely what you said. I will pick some points and comment them.
Pre-test scores are correlated with post-test scores : this sounds quite common to me, it is just the differences between individuals which show up.
Low R2 with a model DV= change scores and IV=treatment alone : that is probably due to 1) a change scores that is blurred by the use of a pre-test value as a definite value (coeff One in the difference pre-post) ad a low effect of treatment alone, because there are (?) large differences between individuals.
It should be useful to plot
1 - post-test vs. pre-test (for each treatment) to see if the pre-test is correlated with post-test as I assume (might be wrong)
2 - change scores (pre minus post or the converse, not important) vs pre-test values, for each treatment. If there is a correlation on this graph, that could be due to a possibly medium to large regression to the mean phenomenon. In this case your model 2 would be mandatory.
Might be useful you make these plots available.
People interested in this should give Allison, PD, 1990. Change scores as dependent variables in regression analysis, American Sociological Association 20, 93-114. a read.
He points out differences and similarities, as well as implications of model choices.
Note that the decision of which model to take very much depends on the question you want to answer, and whether your goal is inference or prediction.
http://www.jstor.org/stable/271083?seq=1#page_scan_tab_contents
In my opinion, the best on the topic is van Breukelen's 2013 paper. It gives a precise and comprehensive overview of change scores, ANCOVA and Lord's paradox. Also, it provides guidance for model selection and interpretation.
https://www.ncbi.nlm.nih.gov/pubmed/26745598
Article ANCOVA versus CHANGE from baseline in nonrandomized studies:...
I am sorry - but I did not publish this paper. Perhaps there is second person with my name.
Best
Manfred J. Holler See www.CCR-Munich.de
- Badges
- Science topic