I have obtained two ratings (creativity and technicality) from six expert judges based on art-related outcomes. I am following Amabile's (1979) Consensual assessment technique to identify assess creativity. Research suggests that (ideally) ratings of creativity should not correlate with that of technicality in order to demonstrate discriminant validity (i.e. they are not measuring the same thing). Since the data collected is ordinal in nature, I have looked at Spearman's/ Kendall’s Tau correlations. However, I am concerned with the correlation and ICC approach as it does not capture paired differences (what I mean by paired here is same product two independent ratings). Correlations are also problematic given the very small sample size (8 ratings). Could anyone suggest what might be the best approach to identify where ratings for creativity and technicality differ (taking into account each rated product)? Thanks!
Dear Kelvin,
Your small sample size severly limits what you can do using statistical approaches, particularly correlational approaches. First, it is very unlikely you would even find a statistically significant relationship. The alternative, no relationship possibility, could not be considered as the probability of a Type II error would be VERY large. My suggestion is you consider a qualitative approach with these data starting with an analytical content analysis. This approach would not establish the relationships you might seek, but could indicate if they might be feasible.
Good luck,
Jim McLean
If you want to evaluate differences you can use t-test or anova test method, through the JASP program you can easily include the bayes factor to quantify the accuracy of the alternate hypothesis of difference and the null of lack of difference.
Article The whole is more than the sum of its parts – assessing writ...
Article Examining the reliability of scores from the consensual asse...
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics