I suggest reading:

Estimation of reliability: a better alternative for Cronbach’s alpha

By

Kimmo Vehkalahti, Simo Puntanen, Lauri Tarkkonen

Available at:

http://mathstat.helsinki.fi/reports/Preprint430.pdf

If your likert scale is 5 and you have dichotomous answers for less than 20% of your questions, you can set them to 2 (for 0) and 4 (for1) and then do the job. However, if your dichotomous answers are extreme you can set them to 1 and 5, respectively.

Andres,

I would recommend methods other than alpha generally, and especially for your situation. There are a number of assessments of internal consistency based on a latent-factor model. Bentler's rho is fairly easy to use.

Pat

Alpha really just looks at the consistency of the responses across the items in the scale. I am not sure why the different lengths would be an issue here exactly. The number of questions seems to some extent unimportant in the sense of asking about scale performance. What are you trying to say about the scales? If you are asking a question that alpha addresses these data would be of interest.

This may be of assistance in viewing this as well.

http://socrates.berkeley.edu/~maccoun/CoefAlpha.pdf

There is often a misunderstanding that the metric addresses unidimensionality. It does not really address dimentionality really. Other methods are needed to address those issues. We see this persist for example in things like ...

http://www.ijme.net/archive/2/cronbachs-alpha.pdf

Hi everyone,

Reliability always depends on the number of items.

There's an unbiased alpha that could help to compare those scales.

On the other hand, are you looking for internal consistency or reliability? There are other approaches to reliability such a test-retest correlations or ANOVA procedures (Hoyt's approach to reliability), also I think about Guttman's lambda coeficients.

Also it could be interesting taking into account the Factor Analysis approach to reliability.

My piece of advice is, in sum, try to perform several approaches to reliability, so you can have a global view of the psychometric funtioning of your test.

If you need more help, please tell me.

Good luck with your research! :)

Andres,

First make sure that Cronbach's alpha is appropriate for the type of data that you have. It should only be computed for continuous variables. If you have ordinal data, it will underestimate the true reliability. For ordinal data, you want to use Zumbo's ordinal alpha or ordinal omega. Both estimators employ a polychoric correlation matrix under the assumption of latent multivariate normality. If this assumption is untenable, then I would recommend an estimator I developed called bootstrap split-half reliability. If you use the Spearman correlation matrix, rather than the raw data or Pearson correlation matrix, as an input it will provide you with a nonparametric reliability estimator.

With regards to comparing the reliabilities of two surveys of different lengths, I employ the Spearman-Brown prophecy formula to equate the lengths of the two surveys.

Hope this helps,

Cristian

Principal Component Analysis is the response: if the items generate a relevant (high percentage of variance explained) 'size' (all the loadings have the same sign) component, then the items can be summed up to generate a comprehensive score.

PCA was demonstrated to be suited for binary and ordinal data (see fundamental works by Cox and Jolliffe). Moreover we can check the consistency between different choices of items e.g. different questionnaires with different number of items) by correlating the emerging component scores.

I would need more information, since the Cronbach alpha would not be an issue here for the different scales, no matter if they have different items per scales. Remember that the internal consistency just tell you the correlation among the items. If you want to know how well the scales are performing it could be possible you need another type of analysis. Evaluate their structure validity with factor analysis, rash models if you want to know how the individual items are performing. Maybe your are not giving enough information to understand you doubt.

Here is further information on the use of Alpha...

http://www.ats.ucla.edu/stat/spss/faq/alpha.html

If the scales supposed to evaluate the same construct, divergent and convergent validity could be useful. Also if the scales are related to a mental health condition, evaluate how they are performing as screening instruments for the condition, but then you would need to diagnose the condition in order to evaluate each scale and their raw scores in terms true positives, true negatives, false positives, false negatives considering different cut-off points. I have an article Rivera Medina in 2005 that discuss this method evaluating the CDI and BDI as screening measures for major depression in adolescents.

Hello,

It is possible to assess reliability and validity in a Confirmatory Factor Analysis (CFA) framework as well. I recommend running a CFA in Mplus and using the standardized factor loadings to compute average variance extracted (AVE) to help you in this assessment of your scales. There are lots of online references and, I believe, an excel macro that can help you calculate AVE.

Cheers,

Pat

Hello Andres,

When there are less than three items I usually indicate the significance level of the correlation coefficient. Indeed, This is an alternative for Cronbach’s alpha widely used in literature. Moreover, I had read that Tarkkonen's rho could be useful as a replacement of Cronbach’s alpha.

Example:

https://www.researchgate.net/publication/258190192_Medicin_de_la_Intencin_en_la_Actividad_Sexual_en_Adolescentes_una_Aproximacin_de_Acuerdo_al_Gnero_del_Caribe_Colombiano

Article Medición de la Intención en la Actividad Sexual en Adolescen...

It's already been said, but I'll say it again.

The number of items should not impact your ability to compare cronbach's alphas UNLESS one or more of your scales have very few, semi-heterogeneous items. It might also be valuable to look at the convergent and discriminant validity (as stated above). If these scales are measuring the Abe construct, consider doing this: do an EFA that extracts 1dactor and output the factor scores (or create a unit-weighted factor score) and correlate that factor score to each scale. The correlations will indicate how much each scale accounts for the common variance.

Not sure if that helps.

Hola.

El coeficiente Gilmer-Feldt es una buena opcion, asi como omega desde la metodología factorial. No dependen del nro de items.

I would be tempted by the generalizability calculation method instead of that the reliability. Generalizability is intended for multiple variables influencing a single outcome, and it takes into account the respective variances of each of the variables. The G coefficient is similar in interpretation to the Cronbach's alpha, but is more relevant for multiple factors affecting a single outcome.

Hello Andrés:

Chronbach's Alpha is a good and trustworthy tecnichal resource to begin a diagnosis of psychometric properties of scales. Yes, one of its limitations is that it is sensible to the number of ítems on a scale. I recommend Rasch Analysis to establish unidimensionality of the scales compared and the contrubution of each individual ítem to the scale's overall metric properties. There are a number of extensions suitable for Rasch Modeling if the ítems are ordinally graded categories. Also, the R Project has a number of packages and documentation for the implementation Latent Trait Modeling beggining with Rasch and extensions.

I suggest reading:

Estimation of reliability: a better alternative for Cronbach’s alpha

By

Kimmo Vehkalahti, Simo Puntanen, Lauri Tarkkonen

Available at:

http://mathstat.helsinki.fi/reports/Preprint430.pdf

As noted before, it is not clear what your request is. If you are wanting to compare the reliabilities of two scales to see if they are different, then you would want to use the Feldt test. It uses a W statistics to determine whether two reliabilities are different, irrespective of the number of items on each scale (this is included as part of the analysis). There is a paper by Feldt and Chin, I believe it is in 2006 in Educational and Psychological Measurement.

My input here highlights the (mis)use people (mainly psychologists) make of Cronbach’s alpha.

It is common practice to look at the value of a Cronbach’s alpha and (because the value is high enough, say .81) jump to the conclusion that the concept is well measured by the items at hand. Now, as noted in the paper pointed out by Dr. Hossein Arsham, this is complete nonsense if the model is not one-dimensional.

In other words, what one ought to do is to first test whether the model is one-dimensional, and if yes, then, well, maybe look at the value of Cronbach’s alpha. One is better off with a true one-dimensional model and a Cronbach’s alpha value of, say, .67 than with a Cronbach’s alpha value of, say, .81 and a model that is not one-dimensional.

Not sure whether this shades any light on the question raised by you, Andrés, but it is worth mentioning because I see this misuse of Cronbach’s alpha way too frequently!

Cheers,

SCM

Dear researchers.

Following the discussion, yes my scales are uni-dimensional. I did EFA on them.

Thus, what is the next step for comparing the reliability of scales?

Thanks for your answers.

Dear colleague,

Performing a Rasch Analysis on your separate scales is (roughly) equivalent to doing an EFA on them to extract exactly one factor. It is hard to see at a distance if your EFA´s did extract only one dimension with all the items loading clearly on it. I assume that is the case.

The advantage of Rasch modeling is that you can determine how each ítem contributes to measure the latent construct you seek to capture with the scale. This happens if each ítem meets the model fit criteria. Also, if the ítems are ordered categories, each category threshold must meet the criteria. Another advantage is that you can considerably shorten the scales (economy of elements) keeping only those ítems that fit best with no loss of information.

I agree that Cronbach's Alpha is frequently missused. See it only as a preliminary way to begin your psychometric analysis.

Spearman-Brown profecy formula. It forecasts the reliability if the scale has a different number of items. If you want to compare two scales with a different number of items, you have to forecast the alpha of one of the two scale using the same number of the other scale.

I think your next step is to rank order the scales by reliability and "eyeball" them. The scales with larger alphas are estimated to be more reliable. The reliability scale is continuous but error grows rapidly as reliability drops, so people have heuristic thresholds like 0.70 or 0.75 or 0.80 being acceptable. If all the scales are above such thresholds or if the differences are small, then differences may not be meaningful.

You could also calculate confidence intervals but IIRC the sampling distribution of alpha is hairy.

If you really, really wanted to remove the effect of different numbers of items, you could use the Speaman-Brown formula to "step up" the shorter scales to the same length as the longest scale. But this is absolutely the wrong way to compare the scales you are using... Instead, it compares some hypothetical longer scales "just like yours (but longer)" to each other.

One possibility is the Spearman-Brown prophecy formula. But probably the simplest way (and the one making the least assumptions) is to remember that Cronbach's Alpha is just a stand-in for the average intercorrelation of all items: "standardized item alpha". When classical test theory was developed, it was too burdensome to calculate the whole tableau of n*(n-1) interitem correlations; so Cronbach's Alpha was designed to estimate a lower boundary for it. But with modern computers, the task is trivial, and the average interitem correlation is a natural measure of reliability.

Why not use re-test reliability? Administer the test twice to the same sample, maybe o weeks apart, to a group of say N = 100 participants and correlate the mean raw scores. This measure does not depend on the number of items / length of the scale as you are correlating total means of both occasions for each of the two tests.

If I understand the Question right, the goal is to comparatively evaluate two instruments intendend to measure the same construct with seperate sets of Items.

If alpha is used as a measure of reliability it is intended to reflect the proportion of true score variance reflected by the indicator Items under the assumptions of classical test theory. In the days when alpha etc. were developed, many methodological  advancements we have today were not available.

I would propose to rather use confirmatory factor analysis. If you have some confidence, that a latent variable estimated this way is a measure of the respective construct, you could use the average variance extracted as an alternative to alpha - reliability (though AVE is conceptually somewhat different) for the purpose of comparison. I am not sure wether there exists a test to compare  two AVEs, so I would maybe use bootstrapping.

One advantage of this is that for a given set of Items, the higher the AVE the higher the reliability estimate should be. However, when you add items with similar indicative power to the set, AVE will be unchanged while reliability estimates grow.

Moreover you have many other means of comparing the adequacy of both models. One would be to test wether both sets of items represent the same latent variable, or if the data rather suggests 2 correlated latent variables. In the latter case the comparison of relibility / AVE may become pointless.

Another suggestion, though not exactly related to your question, would be to consider controlling for common method biases (e.g. by including social desirability scales, unrelated constructs with the same answering format, or whatever applies to your field of study), as reliability estimates like alpha, as well as AVEs are overestimated in general if such effects are present. Your measures may be differentially biased and though comparison of reliability / AVE might suggest superiority of one measure actually the other measure may be better if bias is controlled.

A simple alternative may be to compare the means of inter-item correlations, which are the other component of the Cronbach's alpha formula

In SPSS, Split Half and Guttman coefficient. Better to refer them.

https://www.researchgate.net/publication/233052664_A_Nonparametric_Coefficient_Of_Internal_Consistency

Article A Nonparametric Coefficient Of Internal Consistency

It depends upon the scale of the items and the assumptions you would like to make. Zumbo's ordinal alpha is one of the best alternative measures to use if the variables are measured with an ordinal scale (e.g., Likert).

Unlike Cronbach's alpha, which assumes continuous data, ordinal alpha employs a polychoric correlation matrix to compute reliability. However, it assumes that your data are bivariate normal at the latent level. Furthermore, it also assumes tau equivalence (all factor loads are equal and error variances are equal). If you do not believe that a latent factor is appropriate, Zumbo also developed the ordinal theta, which is appropriate in the PCA context.

If you do not feel comfortable with the latent normaility assumption, you can compute a nonparametric reliability estimator by employing the Spearman rank correlation matrix rather than the polychoric correlation matrix.

If you do not like the tau equivalence assumption, you can combine the polychoric correlation with Raykov's rho.

Finally, if you just hate assumptions in general, there is a method that I developed called the Gugiu bootstrap split-half reliability. This is a nonparametric reliability estimator that can produce nonparametric CIs about the estimator.

Hope this is of help,

Cristian

"A Nonparametric Coefficient of Internal Consistency"  covers your situation. You can download from here:

https://www.researchgate.net/publication/233052664_A_Nonparametric_Coefficient_Of_Internal_Consistency

Article A Nonparametric Coefficient Of Internal Consistency

It seems Tarkkonen’s rho is a better alternative to Alpha. Anybody knows how to calculate Tarkkonen’s rho through a simple and straightforward way? The calculation in his thesis is far too complicated for me at least.. Thanks a lot in advance.  

You could also consider using the average inter-item correlation... Clark and Watson (1995) suggest it should be between .15 and .50.

Composite reliability.

Cristian Gugiu your interpretation of ordinal alpha is incorrect, and reflects common misconceptions related to this estimator. See this paper for details on why ordinal alpha is only an estimator of theoretical/hypothetical reliability rather than the tests actual reliability: https://journals.sagepub.com/doi/abs/10.1177/0013164417727036?journalCode=epma