Hi all, I´d like to get your feedback on a problem I have. The study is about the evaluation of idea quality.
I have a set of 100 ideas from an online ideation campaign:
- 30 business ideas,
- 40 technology ideas,
- 15 service ideas and
- 15 marketing ideas.
Due to the different competences and knowledge needed to properly evaluate the ideas, I´d like to split up the evaluation of the ideas according to their topic (business, service, marketing and technology) and present them to different rates. Every idea will be evaluated against 9 criteria on a likert scale from 1-7. Every idea will be assessed by 3 experts.
In the end, however, I´d like to calculate a mean score across all 100 ideas for every criteria.
Now I´m wondering how to proof reliability of the evaluations by ICC? Which ICC to use to use and how?.
My first idea is to calculate the ICC for every topic evaluation session. In case the ICC is above the threshold of .7, the rating is reliable.
Anybody any idea on that? Thank you in advance Sven
It is certainly a challenging research. Congratulations! I think if the Likert scale will be 0 to 7 points use the limit of 0.7 as a reliable rating will be lower than expected. I suggest using a rating of between 6 to 7 for at least 90 to 95% confidence, as is the case with sample calculations, for a higher degree of reliability. Again, congratulations on the challenge of measuring ideas.
Thank you for all your valuable contributions! It helped me to solve the problem!
Sven
- Badges
- Science topic
- Similar topics
- Mathematical Sciences
- Statistics
- Probability
- Probability Theory
- Reliability