The changes in the Chronbah's alpha in different studies could be attributed to reasons such as the new setting, different participants, the translation of the instrument, its validity in the new culture, the desinge used..etc.

The researcher has to report this, possibly as a limitation and

it will be useful to run EFA to examine the structure of the instrument in the new setting

Thx Muayyad,

Just another question. I used two newly translated tests in my study (I translated them with the help of two specialists, using proper translation and back translation process) the Cronbach's alphas are good for both tests aside for the sub score that I mentioned (originality) from the creativity test. My goal is not validating these two tests but to use them as part of the study. But as originality reliability is low, should I report on exploratory FA for both tests? Only one test? perhaps only the problematic sub test of originality? Naturally, as I want to keep it an article of ~10 pages with main focus given to the study and less for the validation of the tools, can I report on validation process that included EFA only for the originality factor? 

Dear Gilad,

It seems that originality is quite different from the others two. I ask if you really need to test reliability - reliability means "repeatability" or "consistency". A measure is reliable if it give us the same result over and over again – do you need to test that? (the proportion of intercorrelation between the 3 variables?). If yes, try to perform alternative measures, like split-half or test-retest (or even parallel forms). Alternatively, inform in your research that alpha coefficient depends on the number of items in the scale.

Good luck with you research!

Gilad,

similar to Lisete's post, I am wondering if a factor model (and, hence, internal consistency via alpha) is really appropriate here. While I am no expert on creativity, it seems to me that those three indicators are not necessarily reflective of the same latent construct.

In other words (as Lisete pointed out), can we expect to find the same scores on all three indicators from individuals with a certain level of underlying creativity?

Aside from the factor structure, I am wondering if the scores on the indicators satisfy the assumption of normality. If not, this may be an issue in factor analysis. See Chapter 9 in Brown's excellent book: Brown, T. A. (2015). Confirmatory factor analysis for applied research. Guilford Publications.

Oh no, they are not the same at all. This are three different constructs. Three factors actually (fluency of ideas, flexibility of ideas and originality of ideas) each construct is made of three items. Reliability was calculated for each construct alone (meaning, between three items). For originality the Cronbach's alpha was low (0.6) which is reasonable given that many times when one have original answer for one of the three questions he don't have for the rest two.

EFA can show that at least the three items have high loading, so they are valid. What do you think?

Thanks Lisete,

Just thinking now on reporting alpha of 0.6. It's really not a good one but I can explain the reason behind it easily, it's not a product of bad measure but probably the nature of it (don't have much previous studies with it, it's a new one). 

ok, go ahead

 Your reliability is low because the items don´t measure the same kind of creativity (explain this in your research)

Hi some cues:

- number of items (generally a low number of items in a subscale could decrease Alpha in that subscale);

- construct issues (the argument of Lisete)

- number of respondents

- direction of hierarchy scale

- pleas pay attention to assymetry and kurtosis of those items

All best

Paulo

Gilad,

sorry about the confusion.

Still, I think the fact that people may have high originality on one item but not on the others is a problem. Given that all three items should be affected by the same underlying factor, they should not be relatively independent like that.

Also, I am doubtful EFA is appropriate here. While this may be a new sample, you do have a "theory" about the factor structure.

Heiko,

I agree with you that it's the same underlying mechanism and therefore we perhaps assume in theory the items should be effected in similar way from it. But, there is:

1. The operative definition of originality which is strict and set a threshold of suggesting an ideas that were presented by no more than 2% of other participants in the sample. The definition itself is problematic (though this threshold, or a bit softer one are very common in creativity tests) because than if one suggested idea that was also raised by say 3% of the sample population, it's not considered as original and then his score on this item is 0 and on the other it may be 1 or higher (depends on the number of original ideas he came up with) - therefore you get low reliability. This is  type II error.  Also, if one suggested an idea that no one else in large sample came up with, still he got the same score as one who presented an idea similar to that offered by say 19 more others in a sample of 400 participants while the very original participant is more expected perhaps to produce original ideas for other items as well.

2. The notion that novelty of ideas is positively correlated with originality assume that there are shared mechanisms to both. And studies does support this, the more ideas you can produce the more likely you are capable of producing also original ideas. But it doesn't means that this relation is mandatory. Originality is also a way of thinking and not only about being able to come with many ideas.

Finally, I guess that being original requires insight, one term in creativity research is incubation time (meaning, you can't come up with original idea as fast as you come with non original idea, you don't have schema for original ideas. Studies in other realms of decision making and cognitive processes show that many times most of the processing is being done without consciousness, then we have the moment of insight or the moment when we change perspective about the problem, so it's very much different in my opinion than coming up with many ideas as possible which requires you to extract information and perhaps relay more on processes that have to do with executive functions and etc. Actually, people with attention deficit for instance are more original than average (because they are being flooded with information from many different categories and therefore make original associations, but it takes more time than just taking data from categories I guess) . 

Yet again, currently creativity tests assume that the reliability of originality scores based on different items  given for a limited in time creativity test should be high. Regardless of the type of questions and etc. These tests do have small to medium positive correlation with real life creativity, but they are not more than indicator. 

There is also the statistical issue: When only ideas that were suggested not more than 2% of the times in the entire sample are original, you get many participants who score 0 on their total originality score, specifically when the number of items is small. Actually, most of them score 0.

Roughly 30% of those whose originality score is >0 scored 1 or 2, many times suggesting 2 originals ideas for the same item (so, their reliability is very low).

The rest produce at least one original idea for more than one item, still their reliability score is small if they didn't score above 0 for all items and suggested close number of ideas for each item. Now, in my opinion real reliability test should somehow take this into consideration, Meaning that a score of 0 should take a meaning for itself and that the weight that should be given for the correlation between originality items that been scored above 0 is different. But I don't know such test.