when the unit of an analysis is at the team level, it is critical to demonstrate high within group agreement (rwg) to justify using the group average as an indicator of a work group variable.
1. How to calculate rwg in SPSS?
For measuring ICC 1( Interclass correlation) ICC2(Inter-rater reliability) which options at Scale-reliability (2 way mixed, or 2 way random/absolute agreement, consistency) are appropriate for ICC1 and which for ICC2.
Any reference or guidance would be greatly appreciated.
Hi Zinat,
This blogpost may be of general help, a link ist attached.
Please also consider the on-site comment section below the actual article, there you can find some interesting ideas and examples that are discussed.
BR,
Hannes
http://neoacademic.com/2011/11/16/computing-intraclass-correlations-icc-as-estimates-of-interrater-reliability-in-spss/
Hi Zinat,
This blogpost may be of general help, a link ist attached.
Please also consider the on-site comment section below the actual article, there you can find some interesting ideas and examples that are discussed.
BR,
Hannes
http://neoacademic.com/2011/11/16/computing-intraclass-correlations-icc-as-estimates-of-interrater-reliability-in-spss/
Hello Zinat,
Please follow the following link
Best
https://www.youtube.com/watch?v=v2d6eyYoh4M
Hi Hannes and Witold thank you for answering to my question. The links sent were helpful.
Whenever the team is the unite of analysis and data is collected at the individual level ,it is needed to be aggregated. Researchers suggest that an aggregation should be justified . For doing this : 1) The James et al. (1984) rwg procedure is used to estimate the within-team agreement and to determine empirically whether individual scores on variable X etc., could be aggregated within each team .
2) computing ICC 1 : inter-rater reliability or the amount of variance in individual level responses that can be explained by group level
3) and ICC2. which is reliability of group means.
How t0 compute rwg via SPSS.
Further guidance about calculating Rwg ,and ICC 1, 2 would be greatly appreciated.
Hi Renata, Thank you for answering to my question and sending the Excel tool. I've downloaded that and would work with that. I appreciate your time and assist.
Hi Zinat,
I could not find the excell tool in the URL you mentioned. If you have the tool, may I have it?
Thanks.
Hi Mehmet,
You can download from the link below:
http://www.sbuweb.tcu.edu/mscole/articles.html
I also attached the zip file.
PS: thanks again to Renata for sending the tool to me.
Thank you Renata, Hannes, and Witold,
The excel tool is very handy and the Lander (2015) article was worth reading for a quick introduction to ICCs.
Another useful article is the one by LeBreton and Senter (2008).
LeBreton, J. M., & Senter, J. L. (2008). Answers to 20 questions about interrater reliability and interrater agreement. Organizational Research Methods, 11,815-852. doi: 10.1177/1094428106296642
Best,
Nicol
I want to test the concordance between raters of a risk assessment instrument. My methodology is that there will be 5 raters and 7 scale items rated on a Likert 6 point scale. I want to utilize Inter Class Correlation test? Is this appropriate or is there another test statistic that can give me levels of agreement.
Hi Ron, as far as I know you can assess the agreement among the raters by calculating Rwg. For example, Bledow & Frese(2009)used rwg to assess some items for measurement development purposes. The tool attached on this page gives you rwg, and also ICCs.
Hi Zinat,
I have downloaded the tool you shared here and used it. It's great, thank you very much. However for another scale I have this tool is not suitable as it says your response items should be between 5 and 11. I have scale with 2 response kinds; one between 0-4 and the other 0-12. Can you recommend another tool for rwg and ICC calculations perhaps?
Many thanks.
Hi Pinar,
Sorry for my tardiness in replying. I am afraid to say I do not know another tool for your scale.
Hi Zinat (and others),
Do you need Excel 2007 to run the Excel tool? I cannot get it to calculate the statistics on my Macbook with Excel 2016.
Thanks,
Ulrich
Hi Ulrich,
I am afraid that I do not know the answer to your question. Sure, expert people in computer programs could answer your question.
Yes. You have to run it on a Windows PC to get the macros to work properly. It will work with Excel 2007 and Excel 2010.
Hello Zinat,
do you also know how to calculate the rwg score in STATA?
Further, thank you so much for providing the tool.
However, since I have 22 items - I am not able to use it. Do you know another one?
Hello Natali,
Sorry, I do not know how to calculate the rwg score in STATA.
Hope you will figure it out very soon. .
Hello Natali Blem ,
I am also interested in making these calculations in STATA (rwj, ICC 1/2). I would like to calculate agreement between 15-20 healthcare workers per hospital and for several different hospitals. Have you found any helpful sources for STATA?
Thank you - Chris
Hello Christian Laugen,
I am using now SPSS to calculate the ICC Score since this was easier for me.
Further, does someone now what the the cutoff point of the ICC scores is, which allows me to aggregate the individuals to a group level?
- Badges
- Science topic
- Similar topics
- Mathematical Sciences
- Statistics
- Statistical Software
- SPSS