No, only tau-equivalent or parallel measurement data.

Liberal alternatives is ordinal alpha but I recommend McDonald's Omega.

Matt

Thanks Matthew.

Never even heard about that before! But does McDonald's Omega help to determine the validity of such questionnaire as well?

Thanks!

Cronbach's alpha and McDonald's Omega are both based on correlations, so you cannot use dichotomous data such as Yes-No questions.

Dear Jorge Teixeira

The test is not suitable for Yes/No questions.

This paper sheds some light on the assumptions that need to be fulfilled for proper use and interpretation of Cronbach's alpha as an index of reliability/internal consistency of a questionnaire.

Article Making Sense of Cronbach's Alpha

Thanks everyone for the answers!

The questionnaire has some questions in style of a likert-scale. For those I will use a Pearson Correlation.

For the questions with dichotomous answers, I will use Yule's Q.

But I am still having troubles what is the correct test to assess reliability for the questions with answers "often, once in a while, once or twive, never". Can anyone share some light here? Is there any statistical test?

Thanks!

Cronbach's alpha is the most commonly used statistic for Pearson's correlation, but you should not mix that together with the dichotomous items.

David L Morgan could you elaborate why you think mixing dichotomous and polytomous/continuous items is not warranted? After all, the KR-20 formula is mathematically equivalent to coefficient alpha, and is specifically for dichotomous items.

Regarding your point about correlations, the phi correlation is also mathematically equivalent to Pearson's correlation form when dichotomous data are used, so correlations for these types of data are perfectly kosher and well known.

Coefficient alpha is about how much some unweighted composite scores (i.e., a simple sum of the items, coded using some numerical representation such as 0,1,2,...,k) correlates with the underlying trait of interest. Hence, whatever items you use to create this composite are perfectly valid so long as they are linearly related to the trait you are trying to measure. The downside of alpha is that dichotomous items will implicitly be given less importance because polytomous+ items will change the variance of the composite score quicker. This is why measures like McDonalds omega are often better, because the correlations between the items and trait are better included, thereby given different weights to the items.

The assumption behind alpha (tau equivalence) is based on a series of continuous variable to generate an average correlation. I think using ordinal variables (Likert-scored items) already pushes that definition, and so going all the way to dichotomous variables is a step too far.

David L Morgan Tau equivalence just means that all items are given the same weight when forming a linear composite, but the error variance can differ across items. This means for composite/total score X = w1*x1 + w2*x2 + ... + wk*xk, where the w's are weights and x's are the numerical coding of the items in a test with k items, coefficient alpha assumes that all the w's are equal; in the simplest case, letting all w's = 1, you get the usual sum-to-total score. Reliability, as defined in classical test theory, is how much this composite X correlates with the underlying trait of interest (T); i.e., rxx' = cor(X, T).

The item values used to create this composite, whether they be 0-1, 1-2-3-4, continuous, or whatever, does change the process of forming a linear composite with equal weights. This is why the KR-20 formula, which coefficient alpha is just a generalization of, is perfectly fine to use for dichotomous items so long as tau-equivalence (and other assumptions, like unidimensinality, uncorrelated residuals, and the like) is true. Nowhere is it required that the x's need be continuous.

As I stated above, the only downside of mixing dichotomous and polytomous/continuous items is that the items with fewer categories may get under-represented because they don't affect that composite score as much as more informative items. This is why McDonald's omega is generally a better solution since dichotomous items can be given more weight if they strongly relate to the trait of interest. Or better yet, fitting the test with latent variable methods such as those from item response theory would circumvent this problem altogether.

No, tau equivalences means that all items can be given the same weight because of the assumptions behind it. I think you are putting the cart before the horse.

Jorge - I agree with David L Morgan that Cronbach's alpha is the most commonly used statistic for Pearson's correlation. Cronbach's alpha is the most common measure of internal consistency ("reliability"). It is most commonly used when you have multiple Likert questions in a survey/questionnaire that form a scale and you wish to determine if the scale is reliable. A “high” value for alpha does not imply that the measure is uni-dimensional. A negative number indicates that something is wrong with your data—perhaps you forgot to reverse score some items. The general rule of thumb is that a Cronbach's alpha of .70 and above is good, .80 and above is better, and .90 and above is best. Cronbach's Alpha ranges from 0 to 1, with higher values indicating greater internal consistency (and ultimately reliability).

Hii

Five point scale most suitable for Cronbach's alpha.

David L Morgan Tau equivalence is just a mathematical definition, and is either true or false in empirical applications. Hence, whether a composite should be composed based on tau-equivalence depends entirely on the degree to which this assumption is true. If one assumes tau equivalence and uses coefficient alpha anyway (which assumes this), when in fact this assumption is false (the items should be unequally weighted), then the reliability estimate becomes biased. Fortunately, in the special case of alpha, this ends up that the estimate provided is a lower bound of what the true reliability is (again, assuming all other assumptions have been met as well).

In any event, what does your reply have to do with requiring continuous response data for tau-equivalence to be valid?

The major problem I have here is in trying to substitute the dichotomous versions of correlation coefficients for pearson correlations as a way to meet assumptions in tau equivalence. In particular, if you use phi, it often cannot obtain a value of 1.0 due to the marginal distributions. The assumptions of tau equivalence are based on covariances, that have only "analogies" at dichotomous level.

I see your concern, but this is a property of a correlation not an issue in reliability as related to coefficient alpha. Covariances for dichotomous data are still valid covariances, and as you say can be used to either support or reject tau equivalence (e.g., by fitting a factor analysis model and studying whether the loadings are all the same magnitude). This is why estimates like McDonald's omega are also valid for dichotomous data when one is interested in creating a weighted composite score as the unit of measurement. The same is true for basically all estimates of CTT reliability as well.

That said, the bounded nature of the phi correlation does become problematic in contexts other than creating a linear composite score, like when the focus of the investigation is on the item-level properties themselves. This is generally where item-factor analysis and item response theory come into play, where the focus is more centered on the behavior of the items rather than some (usually linear) composite variable. So, while I agree with you that phi correlations may be a problem in some contexts, when forming a reliability estimate for an unweighted composite X as a proxy for T the type of data used to create this composite is not really relevant.

Interesting

So this means we could just substitute phi's anytime we need genuine correlations? Good luck publish a regression analysis with that approach.

Between two dichotomous variables, yes, that's exactly what I'm saying.

So you would have no problem if all the independent variables as well as the dependent variable in a regression with dichotomous? After all, you could easily form a correlation matrix for that analysis using nothing but phis.

No, I would not say that. The DV in regression is assumed to be continuous so that the residuals implied by the difference from a linear composite are conditionally normal. The IVs can be anything though, because again they are used to form some linear composite (here we see this reasoning again that we did before). For regression you form the composite Xhat = b1*x1 + b2*x2.... + bkxk, where again any of the values of the x's can be continuous or discrete.

But you would have a perfectly fine correlation matrix, by your terms where phi is acceptable any time you would ordinarily use a true correlation coefficient. Just by examining the correlation matrix, you would have to no way of knowing that the DV was dichotomous.

Correct. That's why it's an assumption that needs to be satisfied beforehand. Looking at a covariance or correlation matrix alone doesn't tell you whether the variables are continuous or not. The regression model has no idea what variables are what in the system of equations, just that statistics like variances/covariances are computable.

We generally use chronbach for items(statements ) used to judge a construct or latent variable(e.g. stress).THis value tells us if the scale(questions for measuring the construct) is Reliable.The same can only be told when on collects a data of about 50 samples on this set of questions.Questions like Yes No or age or income are called observed variables which can be measured unlike latent variables(stress).Thus they dont need a statement to be measured and thus they dont need a chronbach to measure the same.

Jaspreet Kaur thank you for your explanation