Hello Konstantina,

I agree that checking for comparable internal consistency reliability values across scenario sets makes sense: McDonald's omega or Cronbach's alpha could be used. I'm not sure I agree with the advice to skip the factoring check.

It's not clear what you mean by "...combined two models to create [the measure]." if that implies you adapted items from two existing scales, then I would suggest that factoring probably is worthwhile. Otherwise, how do you justify the use of a single score for the set of 14 items (regardless of how it's computed)?

In any event, If all respondents answered the measure (all 14 items) for all four scenarios, then treat those scenario scores as a repeated measures dimension in your anova.

Good luck with your work.

Hello David Morse,

Thank you for your answer!

I used Cronbach's alpha for reliability and I did correlation tests using Spearman's. I did that for the total scale (14 items) and for the constructs.

I also don't agree with skipping the factorial check, but yeah, that was the advice.

I indeed adapted two existing scales. I would mostly say one scale and one theoretical framework with 5 dimensions that define the type of questions/items you can use. That's why I was concerned about skipping the factorial analysis.

Nevertheless, reliability is pretty high above .8 for all the constructs and above .9 for the overall scale, and correlation coefficients are moderate to strong.

I thought of doing what you suggested, treat the scenario scores as repeated measures and basically compare them, but also use the constructs to see any significant differences, hoping that this will offer more valuable information to my research.

Thank you again!

ANOVA is a comparison of means but a Likert scale is ordinal data. With continuous data

For more information can use this link

https://www.google.com/url?sa=t&source=web&rct=j&url=https://www.quora.com/Can-ANOVA-be-performed-with-a-Likert-scale-or-a-Likert-type-scale&ved=2ahUKEwjfm--Uq4DxAhUFKewKHZ-9BJ0QFjACegQIHBAF&usg=AOvVaw0CnyjtckGh6-1iZFeVm7_K

Thank you Radhwan H Ibrahim , I'm aware, but this is a requirement from my institution and not my initiative. I have seen it is a controversial topic, and that's why I was hesitant, but there are researchers supporting that is possible to use ANOVA for analyzing Likert scales.

I am not sure about your wording, "4 identical scales and 56 questions." What do you mean by scale identicality and questions? Why are you using four identical scales?

Best,

Mohialdeen Alotumi I used the wrong word "identical", I mean I use the same scale for all 4 assessment scenarios. So in total, I have 56 questions (14 per scale) and 4 constructs per scale (with 3-4 variables/ questions each).