When statistics packages (e.g., SPSS) are used to obtain intraclass correlation coefficients (ICCs), they provide the results of an F test. In my experience, the F value in these tests is almost always highly significant, but I confess to not knowing what this signifies. Can anyone help me, please?
Also, could anyone tell me whether the F-test results should be included when reporting ICCs, and why, please.
Good morning. I have books; SPSS analysis without anguish and an SPSS old manual & Armitage and Berry - Statistical methods in medical research. However, the latter book is falling apart, the binding has disintegrated so the pages are falling out. The other two I'll have to look for as when we had people stay at Xmas, all my stuff got thrown into the spare room literally in garbage bags. I can look for these as I'm sure the answers will be found.
Deborah J Hilton thanks, in advance, for the possibility you might come to my rescue. I'll be interested to see whether you and your books can help out. My searches have all been in vain.
Robert
Hello Robert, I've got the Armitage and Berry book. You go to pages 163-5, 464. I have not found the other two books as yet, but I'll look. All the pages of that first book are falling out. I'm just on my way out. Do you have it in your library, are you going to the library today?
Deborah J Hilton: Thanks for pursuing this. I retired from my university 15 years ago and unfortunately I don't have library access any more. I think the local campus of my former university's library wouldn't have many books on stats, anyway. Books seem to have become an undesirable commodity in that library. It has moved with the times . . .
I understand. Yes, it is true, people no longer value paper books, whether it be young students at school, kids at kindergarten or statisticians or librarians. I think it is actually very sad really, as I think technology means people are rushing around too much with information, always looking at computers & devices & not using books or magazines. We went recently to a photography show, & it displayed a photographer's work, but they also had a mock display of her office which included many books. I said to them that I thought this was good. I think what is happening with society & technology is a concern. I'll try & look later today if I can for an answer.
I'll look for the other two books. I agree it is very sad that people no longer value books, very tragic and upsetting really.
Robert Trevethan
Deborah J Hilton
Hello and respect
I enjoyed reading your posts and I agree with you.
But about the answer
Pay attention to the following hypotheses
H0 : ICC = a0 vs H1 : ICC > a0
The F test you mentioned is actually testing this hypothesis.
Abolfazl Ghoodjani, thank you for trying to help, but (at the risk of revealing my ignorance very publicly) I don't know what a subscript o stands for. Could you help out, please?
Apart from that, in my experience, the p value associated with the F test in the context of ICCs almost always indicates a high level of significance (often
Dear Robert Trevethan , of course you are my teacher and I learn from you.
- I mean a0 constant number (0 0.5
Abolfazl Ghoodjani, thank you for trying to help me, but I get the feeling we're on different planets. Maybe my planet is a weird one, but let me explain where "I'm at":
ICCs values are conventionally placed into categories ranging from poor (definitely so if < .50) to very good / excellent (up around > .90). Hypothesis testing doesn't really come into the scene.
To illustrate my problem, I am attaching some output relating to a poor ICC that came from some research I am currently involved in. There, I have highlighted in yellow the results that are most pertinent to the ICC. In this case, the ICC is .49, which would be regarded as poor. I have highlighted the results pertaining to the F test in turquoise. There, it is obvious that the p value is < .001 (highly significant).
Part of my original problem is that I don't know how to interpret these F-test results. I wonder whether you, or others, can tap into that problem. I hope so!
Thanks, again, for trying to help.
Robert Trevethan Do the same analysis with test value = 0.5
What answer will you get?
value for your F-test will be different.
So I want to reiterate that the result obtained for the F-test (and of course the probability value) depends entirely on the following hypothesis: what number you choose a0 to be.
H0 : ICC = a0 vs H1 : ICC > a0
In the sent output, have you tested if the ICC is zero? (H0 : ICC = 0)
SPSS software is not aware of everything when the ICC is estimated to be 0.491, it is known that there is a significant difference and it is not equal to zero
Abolfazl Ghoodjani, thanks, again, for trying to help me. However, I continue to think we are on different planets.
When researchers investigate ICCs we/they do not engage in hypothesis testing - which is what you are proposing.
Instead, an appropriate formula is selected for calculating the ICC (there are several different formulas), the ICC is calculated, and then it is evaluated in relation to criteria, e.g., < 0.50 indicates poor reliability, values from 0.50 to 0.75 indicate moderate reliability, values between 0.75 and 0.9 indicate good reliability, and values ≥ 0.90 indicate excellent reliability.
I am still left wondering what the F-test results signify and whether they need to be reported.
Robert Trevethan Hello and respect
Maybe the following links can help you.
https://www.jstor.org/stable/41304112
https://ncss-wpengine.netdna-ssl.com/wp-content/themes/ncss/pdf/Procedures/PASS/Confidence_Intervals_for_Intraclass_Correlation.pdf
Article Intraclass correlation – A discussion and demonstration of b...
Abolfazl Ghoodjani, thanks again for trying to help. I've just read the first of the items above, but it doesn't seem to answer my concerns. I'm familiar with the third item, but I'm pretty sure it doesn't address my concerns, either. Unfortunately, the second item gets me an error message.
I'm still up in the air!