Hello Sanchit,

This is an interesting issue because alpha (properly referred to as coefficient alpha, not Cronbach's alpha, apparently) is more complex than many people realise. The attached article, although more than 20 years old, might help you.

It might be necessary to determine whether you have subdomains within your 20 items, but, if you don't, you'll almost certainly get a "respectable" alpha even if there is not much association among your items.

Thank you Robert Trevethan Sir for your reply.

Firstly,

I have 20 independent risk factors in project management. They are rated based on their severity and likelihood. I have no subdomains to my 20 items. What can be done in this case?

Secondly,

The 1996 paper says alpha is a function of interrelatedness of the items in a test? In my case of risk factors, what is implied by interrelatedness, as every risk factor will have some likelihood and severity towards cost overrun?

Thirdly,

The 1996 paper mentions that in addition to alpha, intercorrelation among the scales should be calculated. Does this mean I have to calculate correlation among the five point scale levels of the Likert scale? How to evaluate that?

Hello Sanchit,

Robert Trevethan 's advice is a good reply to your first query, so I'll just try to touch on your second query.

1. You might well have subdomains, though you may not "see" any rationale for their existence. One way to explore this possibility is via factor analysis. It is possible that more than one feature/attribute contributes to how your respondents rated each of the 20 risk factors. That said, the number of responses you have (20-25) really isn't enough for stable estimates from a factor analysis.

2. The nub of evaluating score reliability in the first place (via internal consistency estimates such as Cronbach's coefficient alpha) is that all individual item scores contribute consistently to the overall score, making it a better indicator of the target attribute of interest. So, evaluating relationships among items (via item-item correlations, or item-total score correlations) is one common step in item and scale analysis.

3. If you have multiple scales/dimensions. then understanding how these relate to one another is important. For example, if scales are essentially independent or unrelated, it makes no sense to combine them into an overall score. For this reason, if factoring your items suggests a multidimensional measure (more than one scale/dimension/factor/construct), then evaluating how these do or don't relate to one another is useful.

Good luck with your work.

Article The Use of Cronbach’s Alpha When Developing and Reporting Re...

Article Making Sense of Cronbach's Alpha

Article Statistical power as a function of Cronbach alpha of instrum...

Sanchit Saran Agarwal. You have received some good advice up above from David Morse, and I think that the Taber article in the post by Cristian R-V is useful. The second article he mentioned might be beyond your needs.

In answer to your first question, if you are sure that there are no subdomains (I would like to know how / why you are sure - is it from prior research with similar samples?) I would not worry about obtaining coefficient alpha because it is almost bound to be misleading / useless with 20 items. (Of course, reference to alpha is so endemic in the literature that many people think it's good to obtain and report it - so you might want to joint the club even if doing so isn't enlightened. Behaving like a sheep is often endorsed among researchers.)

With regard to your second question, you could check the interrelatedness of your items by obtaining a set of interitem correlations. Ideally on a scale they should be within the range of .15 to .50, and most should be > .30. If there are too many low interitem correlations, the items are likely to be measuring disparate things. If there are many interitem correlations > .50 there is a risk that some items are redundant.

This is related to your third question (where I think you used the word scale when you intended to write "item"). It should be easy to obtain a set of interitem correlations if you are using a stats package, and if you are using SPSS you can certainly obtain a set of corrected item-total correlations that will also provide you with a sense of the extent to which all of your items are measuring the same kind of thing. These correlations are easily obtained within the output obtained when seeking a coefficient (Cronbach's) alpha. There are YouTube presentations that should help you with that if you're unsure.

Dear Sanchit Saran Agarwal , beyond the invaluable feedback by respected Robert Trevethan , Cristian Ramos-Vera and David Morse on your query, the question of Reliability (Internal Consistency or the value of Alpha Coefficient) will arise only if you are developing (or using from literature) a set of items to measure a latent construct.

According to my understanding, you have "Construction Project Risk" as a concept and you have asked 20 questions regarding this concept from 20-25 experts. The scenario is not at all suitable to discuss about Reliability of these "Items". Rather, how do you define "Construction Project Risk"? On what basis you constructed 20 "items"? Why do you say all 20 "items" measure distinct aspects of "Construction Project Risk"?

I will suggest you to share details about your research because I think there is a confusion whether you are qualitatively exploring a concept or quantitatively measuring a construct.

Thank You Robert Trevethan David Morse Cristian Ramos-Vera Muhammad Zia Aslam for your detailed replies. I am humbled.

I believe I have not explained my problem statement in detail:

Objective: Risk identification and analysis of hydropower projects (not any case study specific) in India that can lead to cost overrun during construction.

1. I have identified 20 independent risks (Items). All of them can occur together or in any other combination during hydropower construction.

2. The risks are identified through literature review and discussion with relevant experts. Muhammad Zia Aslam

3. I have identified different experts from India (20 to 25 in number) to give likelihood and severity to these risk factors by

(Scale - Very low to very high, score 1 to 5 respectively).

4. Based on expert scores, I will rank these risks.

5. Robert Trevethan David Morse I might be wrong but the risks are not subdivided further as they are the events that occur during construction which directly lead to cost overrun. I am not aggregating them into different categories, my aim is to rank these individual risks based on their likelihood and severity.

Robert Trevethan David Morse Does the interitem correlation means to check the validity of expert opinions? Validity tools given in literature are Kendells concordance test, Interrelator agreement (IRA), Convergence test.

David Morse Muhammad Zia Aslam Does my case allow for reliability testing?

Hello Sanchit,

With reference to the first of your mentions of my name above, I am afraid that I don't know enough about your specific topic to offer further advice.

With regard to the second mention of my name, Interitem correlations are simply concerned with the amount of association among your items. It is quite easy to obtain a correlation matrix of items with each other. This is more to do with reliability (consistency of items in terms of the size of their inter-relationships) than to do with validity.

Hello again, Sanchit,

Robert is correct about inter-item correlations. But, let's focus on the data you have in hand at present. One aspect about which I was unclear is, did you elicit two judgments about each construction risk (one for likelihood of occurrence; one for severity), or just one?

What you can do, with the data you have thus far, is to assess: (a) rater consistency in judgments of a given risk (Cohen's kappa has been extended by Fleiss and others to multiple judges, and would work well to characterize the degree of agreement); and (b) scale the relative values (most crudely, as a rank order, or more elaborately, as with item response models such as the graded response model). Whether the scale "works" when applied to evaluating specific projects remains to be seen. If IRT calibration "works" for the scale, that's usually promising evidence for unidimensionality (though, there are multidimensional variants available).

Good luck with your work.

References

Fliess, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76, 378-382. doi:10.1037/h0031619

Nelson, K. P., & Edwards, D. (2015). Measures of agreement between many raters for ordinal classifications. Statistics in Medicine, 34, 3116-3132. doi:10.1002/sim.6546

Thank you David Morse, I require two separate judgements, one on likelihood and other on severity using the same list of items. In this case, can I individually check the rater agreement for likelihood and severity and then use these values for any further analysis?