I am not a (habitual) useR, but I understand that the default setting for the t.test command in R is to return an unequal variances t-test that is called the "Welch Two Sample t-test". But for reasons I won't go into here, I found myself wondering if the test it calls Welch's t-test actually uses Satterthwaite's (1946) degrees of freedom, NOT Welch's (1947) degrees of freedom. As the Stata documentation shows, they are not the same. I've put the two equations side-by-side on a slide (see attached image).
Source: https://www.stata.com/manuals/rttest.pdf (page 10)
Because Stata has options for both the Satterthwaite & Welch t-tests, I decided to compare the results from R's t.test to Stata's ttest. I used this example to do so:
- https://rcompanion.org/rcompanion/d_02.html
My results can be seen in the attached text file. (View with a fixed font.) In order to get the result R calls Welch's t-test, I have to use the unequal option for Stata's ttest command. And it uses Satterthwaite's (1946) df, not Welch's (1947) df. When I use the welch option, I get a different result.
So, I reckon that whoever maintains the t.test command should either change the labeling, or start using the Welch (1947) degrees of freedom. But I look forward to hearing what you useRs think about it! ;-)
See: https://stats.stackexchange.com/questions/305127/rs-t-test-unequal-variance-degrees-of-freedom
Thank you, Sal Mangiafico. I had not seen that discussion. I have not (yet) taken the time to read it thoroughly, but from a quick skim, it seems that:
- Welch (1938) and Satterthwaite (1946) proposed the same adjustment to the df, and it is implemented via the unequal option in Stata, with var.equal=FALSE in R, and it is also the "equal variances not assumed" t-test computed by SPSS.
- Welch (1947) proposed a different adjustment to the df, and it is what Stata's welch option implements.
If I've got that right, I suppose that when we report a Welch t-test, we ought to specify which one, Welch (1938) or Welch (1947).
Here are JSTOR links to the relevant articles, for anyone who might be interested.
- https://www.jstor.org/stable/2332010 (Welch, 1938)
- https://www.jstor.org/stable/3002019 (Satterthwaite, 1946)
- https://www.jstor.org/stable/2332510 (Welch, 1947)
Thanks again for alerting me to that StackExchange thread.
Since R is open source, it's simple to look up the actual code. The relevant lines in the function stats:::t.test.default are:
stderrx
A bit of digging and Welch seems to have shown that the Welch (1947) isn't superior:
Aspin, Alice A., and B. L. Welch. “Tables for Use in Comparisons Whose Accuracy Involves Two Variances, Separately Estimated.” Biometrika 36, no. 3/4 (1949): 290–96. https://doi.org/10.2307/2332668. [see point 4 in the Appendix by Welch]
So it makes sense that the Welch-Satterthwaite correction is the standard.
The t.test() command in R is however unusual in that it doesn't include references - it should really reference Welch (1938) and Satterthwaite (1946) and probably Student too.
Interestingly enough, Satterthwaite is mentioned in the R documentation: "If TRUE then the pooled variance is used to estimate the variance otherwise the Welch (or Satterthwaite) approximation to the degrees of freedom is used."
But I have the impression (as a non native speaker) that this formulation rather suggests that you can choose between these options, which is not the case. The documentation should be clearer at this point and cite the specific references imo.
Rainer Duesing , have you considered https://www.r-project.org/bugs.html or to file this issue in ?
Following up on Thom Baguley's post, the documentation for Stata's ttest command (https://www.stata.com/manuals/rttest.pdf) attributes the Welch (1938)/Satterthwaite (1946) adjustment to the latter only and does not even mention Welch (1938). Re the welch option, it says this:
welch specifies that the approximate degrees of freedom for the test be obtained from Welch’s formula (1947) rather than from Satterthwaite’s approximation formula (1946), which is the default when unequal is specified. Specifying welch implies unequal.
I reckon that documentation should be edited too.
PS- I just posted to Statalist suggesting that the ttest documentation be updated as described above.
https://www.statalist.org/forums/forum/general-stata-discussion/general/1734987-unequal-vs-welch-options-for-ttest-why-no-mention-of-welch-1938-in-the-documentation
Good work, Bruce Weaver .
Yeah, it would be a good idea † to ask for updates on the R function.
- "Welch–Satterthwaite" is probably the most common description for the method used. (And would avoid the "Welch (or Satterthwaite)" language.
- It would be good to include the references for the method. (This is pretty common in R documentation.)
† Typical work conversation. "Someone should really do this thing." "Yup, someone really should." "Do we have a program associate for this ?"
Sal Mangiafico and Jochen Wilhelm herewith I volunteer to contact the R team. Anyone in for Stata? ... I am looking in your direction Bruce Weaver ;-)
I've already posted to Statalist, Rainer Duesing.
https://www.statalist.org/forums/forum/general-stata-discussion/general/1734987-unequal-vs-welch-options-for-ttest-why-no-mention-of-welch-1938-in-the-documentation
Sorry Bruce Weaver I missed that in your last post. I submitted an email to the R-core team with our proposal.
I received an answer from the R team and they will consider the changes in the output description and the documentation, as well as adding the references. But they will not include the Welch (1947) option, since there does not seem to be any situation, where this correction may have an advantage. Does anyone know a paper, which shows under which circumstances Welch (1947) may be preferrable?
During the discussion with the R contact person, we found another approach to calcualte the t-value by Cochrane and Cox (1950), which is used in SAS. https://documentation.sas.com/doc/en/statug/15.2/statug_ttest_details14.htm
Does anyone know this? I didn't and I do not have access to the book.
Cochran, W. G., and Cox, G. M. (1950). Experimental Designs. New York: John Wiley & Sons.
- Badges
- Science topic
- Similar topics
- Methodology
- Research Methodology
- Research Papers