I did the following full factorial experiments to find the significance of variables and fit model. I analyzed the data using the standard least squares model and generalized linear model (normal distribution/ identity link function). The results of both analyses are attached. Why do they show completely different p-values while the model estimates are exactly the same? What can I say about the significance of the variables?
X1 X2 X3 Y1
++− 1 1 -1 67.5
−++ -1 1 1 0.5
+−− 1 -1 -1 8.9
−−+ -1 -1 1 0.4
+++ 1 1 1 56.9
−−− -1 -1 -1 8.7
+−+ 1 -1 1 6.6
−+− -1 1 -1 69.4
000 0 0 0 37.1
(all variables are continuous)
Generally, how can I find that which model (GLM or standard least squares) should be used for analysis when I have no idea about the response distribution?
----------------------
Y2
0.034
0.001
0.011
0
0.144
0.007
0.035
0.021
0.053
consider Y2 as another response of the designed experiments above. when you check the distribution of Y2 , you will find 0.144 as outlier. does it mean that least squares model can not be suitable as it is affected by outliers?
Hamed,
since the outcome is continuous, you may assume the residuals are independent identical normal distributed, rather the Y1, Y1 is normal for each given level, you have no way to verify.
As your data coding, use linear regression or GLM, Say SAS reg, or SAS GLM, give similar results. See following coding and attachment.
Question? what is 0 0 0 dtabd for x1 x2 x3? If no meaning you may delete it.
With it, the model will assume the difference between 1 and 0 , 0 and -1 for each factor x1,x2,x3 are the same. If not, you may change coding.
hope help
data
try;
input
note $ X1 X2 X3 Y1;
cards
;
++- 1 1 -1 67.5
-++ -1 1 1 0.5
+-- 1 -1 -1 8.9
--+ -1 -1 1 0.4
+++ 1 1 1 56.9
--- -1 -1 -1 8.7
+-+ 1 -1 1 6.6
-+- -1 1 -1 69.4
000 0 0 0 37.1
;
data
try; set try;
y2=log(y1);
proc
reg;
model
y1=x1 x2 x3;
run
;
proc
GLM;
class
x1 x2 x3;
model
y1=x1 x2 x3/solution;
run
;
Why do they show completely different p-values while the model estimates are exactly the same?
Actually, both analyses should give identical results. Maybe the problem here is that the GLM tests are performed using the Chi² distribution. Maybe useing F-tests here would be better.
Generally, how can I find that which model (GLM or standard least squares) should be used for analysis when I have no idea about response distribution?
The response distribution (more specifically: the distribution of the *errors*) must be known or assumed. If it is not known, the assumption should be reasonable. There is not much more to do. One may look at the actual residual distribution to see if it is strongly contraditory to the assumption, one may also simulate data following the assumption and see if the "real" reasults are in strong contradiction to the simulated results.
does it mean that least squares model can not be suitable as it is affected by outliers?
If 0.144 is a physically impossible or really non-sensical value, you should anyway remove it. Otherwise you actually have too little data to judge from the data alone that this is an outlier. Typically the best option is then todo the analysis onec with and once without this value; if the results are grossly different, some careful reasoning is required (otherwise it doesn't matter anyway).
Outliers affect all analyses, not only least-squares. But only for least-squares it is relatively simple to detect outliers (given enough data). This does not mean that least-squares is more sensitive to the presence of outliers than other methods, except, for sure, robust methods (that usually have other drawbacks).
Paul, can you elaborate how ODA exactly applies to this problem? How does ODA decribe the relationship between the predictors and the response?
It was kind of hard for me to see the results of your outcome one my screen. Something that I have seen when using center points in DOE and not using the DOE analysis option in the software is that the software can think each level, -1, 0 and 1 are all categorical levels. I've had sofware give error messages if I use a design like yours and look for interactions. The software assumes that when you ask for X1*X2, you will have X1[-1]*X2[-1], X1[-1]*X2[0], X1[-1]*X2[1], X1[0]*X2[-1], X1[0]*X2[0], X1[0]*X2[-1], X1[1]*X2[-1], X1[1]*X2[0], and X1[1]*X2[1]. Since you don't have these 9 different combinations, the software might try to compensate some other way.
What software did you use in this analysis? Is there a reason you didn't use the DOE section to analyze the data? If you remove the center point from the analysis, GLM and Least Squares should give the same results. When you add in the center point, I can see things being different for the reasons above.
I did get a chance to analyze the Y1 data quickly. Since you only have a single center point and not a series of axial points, you can't really analyze for a quadratic term. I found that terms X1, X2, X3, X1X3 and X2X3 were statistically significant. The center point is not significant. As Jochen will tell you, there is a difference between statistical and practical significance. Which is for you to decide.
Because of the small number of samples in the analysis and the fact I pooled insignificant terms, the statistical significance of some of these terms might be (most likely are) bloated.
Typically a glm function uses different estimation (such as ML) than regression, and may report z-scores instead of t-scores. So you would expect to see some difference there. But there is another issue with this dataset which is a problem. You have 8 data points and with a full factorial you are also fitting 8 parameters (X1, X2, X3, X1*X2, X1*X3, X2*X3, X1*X2*X3, const) so you're basically overfitting. E.g. (the m are the X coded 0/1 instead of -1/1)
. drop if descriptor=="000"
. anova y1 m1 m2 m3
Source | Partial SS df MS F Prob > F
-----------+----------------------------------------------------
Model | 5078.11389 3 1692.70463 3.89 0.1113
m1 | 463.601246 1 463.601246 1.06 0.3604
m2 | 3599.76141 1 3599.76141 8.27 0.0452
m3 | 1014.75124 1 1014.75124 2.33 0.2015
Residual | 1741.3251 4 435.331275
-----------+----------------------------------------------------
. regress y1 m1 m2 m3
------------------------------------------------------------------------------
y1 | Coef. Std. Err. t P>|t| [95% Conf. Interval]
-------------+----------------------------------------------------------------
m1 | 15.225 14.7535 1.03 0.360 -25.73727 56.18727
m2 | 42.425 14.7535 2.88 0.045 1.46273 83.38727
m3 | -22.525 14.7535 -1.53 0.202 -63.48727 18.43727
_cons | 9.8 14.7535 0.66 0.543 -31.16227 50.76227
------------------------------------------------------------------------------
glm with ML
. glm y1 m1 m2 m3, family(gaussian) link(identity)
------------------------------------------------------------------------------
| OIM
y1 | Coef. Std. Err. z P>|z| [95% Conf. Interval]
-------------+----------------------------------------------------------------
m1 | 15.225 14.7535 1.03 0.302 -13.69132 44.14132
m2 | 42.425 14.7535 2.88 0.004 13.50868 71.34132
m3 | -22.525 14.7535 -1.53 0.127 -51.44132 6.39132
_cons | 9.8 14.7535 0.66 0.507 -19.11632 38.71632
------------------------------------------------------------------------------
glm with iterated least squares
. glm y1 m1 m2 m3, family(gaussian) link(identity) irls vce(oim) noheader nolog
------------------------------------------------------------------------------
| OIM
y1 | Coef. Std. Err. z P>|z| [95% Conf. Interval]
-------------+----------------------------------------------------------------
m1 | 15.225 14.7535 1.03 0.302 -13.69132 44.14132
m2 | 42.425 14.7535 2.88 0.004 13.50868 71.34132
m3 | -22.525 14.7535 -1.53 0.127 -51.44132 6.39132
_cons | 9.8 14.7535 0.66 0.507 -19.11632 38.71632
------------------------------------------------------------------------------
Note that -anova- and -regress- give the same p-values and glm gives the same coefficients and quite similar p-values except for m2, but generally m2 is significant whichever way you look at it.
Now see what happens when fitting the full factorial model:
glm with ML
. glm y1 m1##m2##m3, family(gaussian) link(identity) noheader nolog
------------------------------------------------------------------------------
| OIM
y1 | Coef. Std. Err. z P>|z| [95% Conf. Interval]
-------------+----------------------------------------------------------------
1.m1 | .1999998 19.8 0.01 0.992 -38.60729 39.00729
1.m2 | 60.7 19.8 3.07 0.002 21.89271 99.50729
|
m1#m2 |
1 1 | -2.100001 28.00143 -0.07 0.940 -56.9818 52.7818
|
1.m3 | -8.3 19.8 -0.42 0.675 -47.10729 30.50729
|
m1#m3 |
1 1 | 6 28.00143 0.21 0.830 -48.8818 60.8818
|
m2#m3 |
1 1 | -60.6 28.00143 -2.16 0.030 -115.4818 -5.718202
|
m1#m2#m3 |
1 1 1 | 42.4 38.34254 1.11 0.269 -32.75 117.55
|
_cons | 8.7 14.00072 0.62 0.534 -18.7409 36.1409
------------------------------------------------------------------------------
glm with iterated least squares
. glm y1 m1##m2##m3, family(gaussian) link(identity) noheader nolog irls vce(oim)
------------------------------------------------------------------------------
| OIM
y1 | Coef. Std. Err. z P>|z| [95% Conf. Interval]
-------------+----------------------------------------------------------------
1.m1 | .1999998 19.8 0.01 0.992 -38.60729 39.00729
1.m2 | 60.7 19.8 3.07 0.002 21.89271 99.50729
|
m1#m2 |
1 1 | -2.100001 28.00143 -0.07 0.940 -56.9818 52.7818
|
1.m3 | -8.3 19.8 -0.42 0.675 -47.10729 30.50729
|
m1#m3 |
1 1 | 6 28.00143 0.21 0.830 -48.8818 60.8818
|
m2#m3 |
1 1 | -60.6 28.00143 -2.16 0.030 -115.4818 -5.718202
|
m1#m2#m3 |
1 1 1 | 42.4 38.34254 1.11 0.269 -32.75 117.55
|
_cons | 8.7 14.00072 0.62 0.534 -18.7409 36.1409
------------------------------------------------------------------------------
. regress y1 m1##m2##m3, noheader
------------------------------------------------------------------------------
y1 | Coef. Std. Err. t P>|t| [95% Conf. Interval]
-------------+----------------------------------------------------------------
1.m1 | .1999998 19.8 0.01 0.994 -251.3829 251.7829
1.m2 | 60.7 19.8 3.07 0.201 -190.8829 312.2829
|
m1#m2 |
1 1 | -2.100001 28.00143 -0.07 0.952 -357.8919 353.6919
|
1.m3 | -8.3 19.8 -0.42 0.747 -259.8829 243.2829
|
m1#m3 |
1 1 | 6 28.00143 0.21 0.866 -349.7919 361.7919
|
m2#m3 |
1 1 | -60.6 28.00143 -2.16 0.276 -416.3919 295.1919
|
m1#m2#m3 |
1 1 1 | 42.4 38.34254 1.11 0.468 -444.7882 529.5882
|
_cons | 8.7 14.00072 0.62 0.646 -169.196 186.596
------------------------------------------------------------------------------
"which modelshould be used for analysis when I have no idea about the response distribution"
What you really care for is the distribution of the residuals and that gets tougher in multiple-regression situations. You try to make reasonable assumptions. The other question is how to choose a model, for which there is no definitive answer. In principle you can compare R2 between different least-squares solvers and AIC between ML solvers, but you must judge carefully.
- Badges
- Science topic