First, there is no problem with using descriptive statistics. Second, to use inferential statistics you would need to combine related items into scales. Are you familiar with that process?

Hi David L Morgan

Thank you for your response,

I am not sure what type of descriptive statistic can be used as I am not sure what type of data I have - I believe it is ordinal categorical but I don't see how I can take something such as a mean and calculate it for a 3 point Likert scale with the following answer options,

• Less likely
• No effect on likelihood
• more likely

Yes, I am somewhat familiar with inferential statistics. I have themes/groups which I have grouped the Likert items/questions into. However, is there a way to test the questions to see if they fit the group? Furthermore, I have only one sample, does that mean I am unable to do any correlational analysis techniques. Or does the correlational analysis happen between scales (Grouped Likert items)

Additional assistance appreciated!

Jameel

Hey

After the collection of the data you have to organise the data and score it (as per the rules), then you can apply descriptive and inferential statistics, but in applying inferential statistics there are some sorts of assumptions which need to be fulfilled first, then only you can apply the appropriate statistical technique.

Hi,

I can say its safe to do descriptive statistics (frequencies, percentages, means and standard deviations) on your data.

You can also safely do correlation analysis using a compilation of statements representing a theme with another compilation representing another theme.

Otherwise; if you want to compare one statement with another, appropriate statistics will be using a Chi square test. Sometimes people use  the t-test to compare responses on Likert scale but it is not appropriate

Jameel Isaacs. I think that to see whether there is any "validity" to the groupings you have made you probably need to begin with exploratory factor analysis, or, if you are very confident about which items belong in certain groups, begin with confirmatory factor analysis.

Because those procedures are a bit complex for beginners, I suspect that's why David Morgan asked you if you are familiar with the process[es] of forming scales.

I guess it depends on how impressive you want your research to be. Scanning items and deciding, subjectively, what goes with what, is one strategy, and it might be acceptable for a basic kind of assignment. However, in my view it would be far too "lightweight" for anything beyond an undergraduate project.

All the best with what you are doing - and feel free to come back with more queries if I/we might be able to help more.

Oh, and you might need to tell us a bit more about your research so that we can contextualise our suggestions more helpfully. Do you want to investigate variables other than what is being measured by your scale, for example?

For descriptive statistics on individual items, you could use anything associated with ordinal but not interval data, so that would exclude means and standard deviations.

With regard to creating scales from Likest-Scored items, there have been so many questions here about that topic, that I have collected some resources on it:

https://www.researchgate.net/post/How_can_one_help_people_create_simple_scales_from_Likert-scored_items

Venkateswar Meher

Thank you for your response.

I have three answer options as per the above, if I code the statements as 1, 2, 3 and compute the mean. Let's say I calculate an answer of 2.4, how does that translate to back to my answer options?

Regarding the inferential statistics, are you referring to normal distribution assumptions so that parametric or non-parametric tests can be used?

Daglous Gesora

Thank you for your response.

So to confirm, would I conduct the above mentioned descriptive statistics on the individual Likert items (statements)? As an additional point, would these statistics provide any value if I only have a 3 point Likert item? I do not want to do it for the sake of doing it. David L Morgan mentions that I would have to exclude means and standard deviations as I am working with ordinal and not interval. Can we clarify on this point please.

"You can also safely do correlation analysis using a compilation of statements representing a theme with another compilation representing another theme." -- I believe this will definitely provide value - thank you for helping validate the possibility of doing this.

"Otherwise; if you want to compare one statement with another, appropriate statistics will be using a Chi square test. Sometimes people use  the t-test to compare responses on Likert scale but it is not appropriate." -- Will try this as well, I suppose some serious logical thinking will come into play in determining what statements to correlate.

Robert Trevethan

Thank you for your response.

Thank you for the clarification around the validity testing of the Likert scales. I think I will conduct the EFA just to be sure and subsequently the CFA. If you have any links to good resources on this I would greatly appreciate it!

"I guess it depends on how impressive you want your research to be. Scanning items and deciding, subjectively, what goes with what, is one strategy, and it might be acceptable for a basic kind of assignment. However, in my view it would be far too "lightweight" for anything beyond an undergraduate project." -- I am looking to really dive into the analysis, I do not want to do a light touch analysis for my research. I really want to do a thorough and proper analysis. What kind of analysis would you recommend for masters level? I also want to be objective and eliminate as much bias as possible.

More information about my research:

I am attempting to determine Voter apathy in South African student populations.

I have 25 statements, which I created from predetermined segments of an election which I believe would contain critical factors in affecting a students likelihood of voting. Each statement had three answer options, more likely, no effect on likelihood and less likely. I have 700 odd responses to each question.

Ultimately I would like to build a new voting framework based on the data I have received through my responses. But I would also like to look at what hidden truths I can extract.

Looking forward to any further help.

Jameel Issacs

First your score your items accordingly and did the raw score of each sample,

Then if you want to just describe the data without comparison then you can go for descriptive statistics, but if you want to study the statistical significance differences between or among the variables then you can apply either para.etric or non-parametric statistics if the assumptions are fulfilled.

Your nature of the data

And your hypotheses may be helpful to use the appropriate statistical technique.

Jameel Isaacs. Hello Jameel - It sounds as if you have 700 people in your data set - yes? If so, that's good. Your sample is large enough to do both exploratory factor analysis (EFA) with about half your sample, and also confirmatory factor analysis (CFA) with the other half.

I'm a bit concerned about your having offered only three response options, but maybe you can sail ahead with some confidence despite that. (If others would like to come onto this thread and provide warnings, I'd not object to them doing so.)

There are a number of good articles about conducting EFA. Here is one of the most helpful:

Costello, A. B., & Osborne, J. W. (2005). Best practices in exploratory factor analysis: Four recommendations for getting the most from your analysis. Practical Assessment, Research & Evaluation, 10, 1–9.

I could give you more advice, but, given you're a master's student, I don't want to encroach on the territory that your supervisors might regard as theirs. Two days ago, I submitted an article to a journal in which I wrote quite a lot about best practices in EFA and CFA, but I don't want to make that public via RG given the prospect of copyright issues. If you wanted to contact me privately on RG (do you know how to do that? Let me know if not), you could give me your email address and I'd then be happy to send you some of the manuscript that I've just submitted - and perhaps offer some more advice that's specific to your particular research.

I hope that's helpful.

Robt.

You should not calculate a mean on the individual items because they are ordinal data, and that statistic requires interval-level data.

The goal in creating multi-item scale is to produce a measure that does meet the assumptions of interval data, so that you can not only report means for such scales but also use them in inferential statistical tests.

The total score of the ith person score on an instrument are assumed interval in nature and therefore amenable to parametric statistics

Jameel Isaacs, your concern on means has been well answered by David L Morgan

Chuck A Arize, actually the total scores can only be considered to be interval if: 1) they are unidimensional (i.e., they measure only a single dimension, as assessed by an Exploratory Factor Analysis), and 2) they have a high degree of reliability (as assessed by Cronbach's alpha).

David L Morgan

Thank you, the total score of the ith person can be treated as an interval measure for any parametric analysis and t-statistics on the means will be appropriate for his 700 observations. Also, Cronbach alpha will pass easily given his large observations. My post refers to the validity of t-test of the means. I expect him to know that apples and oranges are different. That is, why I simply focused on the relevant point. Even without looking at his data, I can say that with 700 observations the issue of dimensions is minimal as long as the means are used with WLS estimator

I notice that Cronbach's alpha has been referred to in a couple of recent posts above. Although this metric is widely cited, presumably, as an indication of homogeneity among items on a scale, it is very much dependent on the number of items in a scale and has even been demonstrated to be high despite subsets of scale items having very little, or even negative, correlations with each other when those subsets of items are combined into a single alpha calculation. The limitations of this metric (alpha) have been written about for at least 20 years, but many researchers seem to be unaware of those limitations.

I've just submitted an article in which I discuss these issues, and I would be happy to provide some relevant references from that article if anyone is interested. If my article is accepted for publication, I would be happy to provide the reference for it, too.

Chuck A Arize The number of observations has nothing to do with the value for Cronbach's alpha, because it is solely determined by the average correlation and the number of items that compose the scale.

Robert Trevethan Since alpha is determined by the average correlation and the number of items, it would take a very large number of items to overcome a low average correlation. That could occur, but it would not apply to most practical applications -- especially for survey research, where time limitations tend to keep the number of items to a reasonable length.

David L Morgan. I'm interested in following up this with you, David, and please feel free to get back. I was basing what I wrote both on publications that I respect and on personal experience. The publication that I think most clearly demonstrates that alpha is related to number of items in a scale is by Cortina:

Cortina, J. M. (1993). What is coefficient alpha? An examination of theory and applications. Journal of Applied Psychology, 78, 98–104. https://doi.org/10.1037/0021-9010.78.1.98

It was this article that inclined me to write that the number of items affects alpha had been known for at least 20 years. The following two articles also deal with the issue:

Cho, E., & Kim, S. (2015). Cronbach's coefficient alpha: Well known but poorly understood. Organizational Research Methods, 18, 207–230. https://doi.org/10.1177/1094428114555994

Streiner, D. L. (2003). Starting at the beginning: An introduction to coefficient alpha and internal consistency. Journal of Personality Assessment, 80, 99–103. https://doi.org/10.1207/S15327752JPA8001_18

My most recent personal experience refers to the Teacher Sense of Efficacy Scale, which is generally believed to comprise three independent (which often seem to be regarded as orthogonal) dimensions, each of 8 items. The three subscales typically have alphas in the .80s. When all three subscales are combined into a single total scale score across all 24 items (something that, in my view, many researchers do despite it being anomalous to do so), the single alpha value is typically in the mid to high .90s.

To me, this quite clearly demonstrates that alpha is highly dependent on number of items, and at times is not "troubled" by disparity in the constituent items.

I think that Chuck Arize and I might differ in that I regard alphas as increasingly meaningless when there are, say, more than 15 items in a scale, whereas he seems to think alphas remain meaningful, but it's merely necessary to know that larger numbers of items produce higher alphas. At least I think that's the difference in our approaches. Maybe I have misinterpreted what he wrote - in which case I would welcome some clarity from him.

My focus was on t-test usage for a test-score.For Cronbach alpha, I did not discuss whether it was meaningful or not. For me, it is simply a descriptive statistic and has little or no relevance on the regressand which can be measured with error. Further, I noted that the variation in a large sample of 700 would be enough for the issue of dimensions to have minimal effect on the outcome of the estimation as long as WLS estimator was used on the means.

Chuck A Arize. Sorry, but I can barely understand anything that you have written immediately above. It seems we're on different planets.

Robert Trevethan

Since I saw your comment while I was about to sl eep, I simply thought you understood, please read it again. I am in bed already

Robert Trevethan, there is no doubt that alpha is related to the number of items in a scale, because that is part of the formula for calculating alpha. The practical question is how many items you need to have before a "low" correlation can turn into alpha of .7 or .8. I mostly work with survey research, where it would be highly unusual to have as many as 20 items in a scale, due to time constraints. But I also know that some researchers who work with other methods (and "captive populations") do use much larger item sets.

David L Morgan. Thanks for getting back. In retrospect, I think we were on slightly different paths (but at least we were on the same planet - your response makes sense). I wasn't addressing the issue of how many items are needed before low interitem correlations will yield a "decent" alpha, but, rather, whether high alphas were a useful / accurate indicator of reliability (presumably homogeneity) in a scale.

By way of background, I am currently doing quite a lot of research about teacher sense of efficacy. The most common scale in that field is the Teacher Sense of Efficacy Scale (TSES), which, although it has a short form of 12 items, is often used in its full version with 24 items, even when participants are not "captive". Citing an alpha value for all 24 items combined is a common mantra in the literature about the TSES even though I'm sure alpha is meaningless under those circumstances and, I'm sure, cited only because everyone else cites it.

For what it's worth, a colleague and I are currently conducting research with a scale that has 40 items in it - and it's administered online, so the prospective participants aren't captive. Our response and retention rates with preservice and inservice teachers seem to be quite high (up above 60%). I might be deceived, but I think that's attributable to the way we have framed our request for participation (courteous and informative) and the care with which we designed the survey. I hope our work will be published within the next 6 months, but we need to do some more analyses first.

Thanks, again, for your thoughtful, informed response.

Robt.

Robert Trevethan I think have always been careful to say that alpha is not a measure of homogeneity. That being said, however, very high alphas (e.g., .9 and above) only leave room for very highly correlated factors.

Your experience with online surveys is interesting. In a telephone survey (which typically lasts about 30 minutes), a 24-item scale such as the one you describe would simply eat up too much valuable time. As for the 40-item scale that you describe, do you believe that is uni-dimensional, and if so, why would it take so many items to describe a single construct? If it is multi-dimensional, then I would hesitate to describe that as a single scale.

David L Morgan. Thanks for your thoughtful, and thought-provoking, responses. I, like you, would not regard alpha as a measure of homogeneity, but I suspect that a lot of researchers believe it is. However, I'd be interested to have your reflections about what the difference is between reliability and homogeneity in this situation. I'm not challenging you - merely inquisitive and personally puzzled. I need to do some more thinking about this. Maybe you can help.

First up, it's interesting that, in the TSE literature, most researchers sing the praises of their research for producing alphas up above .90 on 24 items while, nevertheless, analysing their data as if those data comprised three independent, and distinctly different, domains. I've just submitted an article in which I criticise that kind of thinking.

The research we originally did with 24-item surveys, and are currently doing with 40-item surveys, is all online, not by phone - well, apart from the fact that a lot of participants seem to have responded using their cell phones. Our new 40-item scale seems, on the basis of both EFA and CFA (we have enough participants to use both forms of factor analysis with different participant groups) to be pretty robust in terms of having two clear factors.

I wonder whether, under the circumstances, it is better to refer to two (separate) scales, or to a single scale that comprises two subscales. I'm not sure whether that's merely semantics or not. I think of scales such as the WAIS and WISC that are called (single) scales despite having subscales.

I appreciate your point about whether it's necessary to have as many as 40 items to assess a construct. That's a problem for us, particularly because I'm sensitive to the problem of redundancy on scales. We chose our items, and worded them, very carefully to avoid redundancy - and hoped that FA would help us cull some of those items. But they all hang in there with good loadings (all above .50 on their respective factors) and pretty high communalities.

There is one interesting background feature in our research: So far, all of our samples have been Chinese, and there might be a greater tendency among them to integrate things that westerners would compartmentalise. We need to investigate that more.

If you had time & inclination, I'd appreciate additional comments / reflections from you. Our interchange might be helpful for other researchers - even if neither of us attracts a lot of recommendations for our posts! I'm interested in promoting good research, not in gathering recommendations on RG, particularly when those recommendations can easily be bogus. :-)

Robert Trevethan If one does an oblique rotation, it will show the degree of correlation between the factors produced. I would have to do some arithmetic to determine the range of correlations for a 3-factor solution that produced an alpha of .90, but I suspect those r's would be .7 and above -- well within the range where one would have to worry about multicollinearity.

But the best way to examine issues like that would be to move out of an Exploratory Factor Analysis and into either a Confirmatory Factor Analysis or a Structural Equation Model. A CFA or SEM approach would allow for direct statistical tests of the amount of improvement for things like using a 1-factor or 3-factor model, etc.

As for the 40-item scale, the fact that each of the items does a good job of capturing the underlying construct is not the same as saying that you need all of the items to capture that construct. It might be interesting to compose a scale from either the 5 or 10 top loading items, and then see how strongly that correlates with each of the remaining items.

From a practical point of view, however, what interests me most is the whether online interviews would make it practical to use longer scales, especially for measures that involve sub-scales. As an example, what is known as the "Big Five" personality inventory involves, as you might imagine, five sub-scales. It was originally developed by psycho-metricians who used students as a captive audience to answer as many as 100 questions. For survey purposes, it is often reduced to a 5 x 4 short form of 20 items. But measuring something as complex as a personality dimension with only 4 items strikes me as dubious.

So, if respondents will accept longer versions of scales in online interviews, that could definitely be an advantage for that mood of survey interviewing.

David L Morgan. You are right: In the literature that I am criticising, the factors are usually correlated, but that's revealed subsequent to factor analyses that have been done using varimax rotations. For some reason, most researchers in the field use varimax rotations despite also (anomalously) arguing that the three factors are correlated with each other.

In their CFAs, the researchers also usually get quite good results, but I'm interested whether those results are enhanced by outliers in the data because the data are almost always very skewed.

You are also right in saying that all items capturing a construct is not the same as all items being needed to do the job. The issue we are tussling with is whether we simply want to distinguish between people on the basis of, say, half a dozen items that might not represent the breadth of the construct, or whether we want participants' scores to be capable of reflecting the full range of aspects that prior research and our own pilot work has indicated to be relevant to the construct.

I agree with you about a small number of items to assess a construct being "dubious". We certainly don't want to measure our target construct with anything like as few as four items. Our problem is that all 40 items seem to be hanging in there despite overt lack of redundancy - and we'd done pilot testing to ask teachers whether they thought a larger set of items was relevant or not. We used their responses as part of the process to arrive at our 40 items.

For what it's worth, in our most recent research, we had not only the 40 items from our own scale(s), but also a burnout subscale (one of the good old Maslach subscales) and the 9 items from the Big Five agreeableness scale. Pretesting indicated that respondents could fill in those items and some demographic items (about 60 items all up) within 10 minutes. I'm sure we lost some participants because of the effort involved (we're finalising our analysis of this, but it looks as if we had around a 70% retention rate based on those who didn't take up the online invitation as well as those who gave up part-way through), so our levels of participation are gratifying given what appears to be the norm in this field where response / retention rates seem to be about 33% or lower, even when incentives are offered and only around 30 items are involved.

I have a question: Should the interfactor correlations that are produced as part of an oblique EFA be regarded as akin to the correlations between the same domains based on participants' responses subsequent to a varimax rotation? I suspect the answer is "pretty much, yes", but I'm not sure. (I hope you can see what I'm asking here.)

Thanks for your responses. To me, this is what RG is about - not like the streams of vacuous self-serving Facebook-like posts on some RG threads.

Hello All,

I have summarised the post follows,

• Data: 25 Likert style statements with 3 answer options
• Responses: 700+

On individual items, the following data summary is valid

• Data Type: Ordinal categorical
• Descriptive Statistics: Anything associated with ordinal data except for means and standard deviations
• Inferential statistics: Item to item comparison: Chi-Squared (Non-parametric restriction)

To test validity between groupings, EFA unless confident then begin with CFA

• Conduct EFA with half the sample and CFA with the other half if your sample is large enough
• Resource: Costello, A. B., & Osborne, J. W. (2005). Best practices in exploratory factor analysis: Four recommendations for getting the most from your analysis. Practical Assessment, Research & Evaluation, 10, 1–9.

The goal in creating a multi-item scale is to produce a measure that does meet the assumptions of interval data, so that you can not only report means for such scales but also use them in inferential statistical tests.

Total scores required to be,

• Unidimensional (They measure only a single dimension) - As assessed via EFA
• Have a high degree of reliability – As assessed via Cronbach Alpha

Resource: https://www.researchgate.net/post/How_can_one_help_people_create_simple_scales_from_Likert-scored_items

Notes around Cronbach Alpha:

• Careful not to regard alpha as a measure of homogeneity. i.e. Alpha cannot tell you if all the items measure the same underlying construct
• It does not imply that the measure is unidimensional: EFA can be used to check for unidimensionality.
• Alpha is a measure of internal consistency.

Resources on Cronbachs Alpha,

• Cortina, J. M. (1993). What is coefficient alpha? An examination of theory and applications. Journal of Applied Psychology, 78, 98–104. https://doi.org/10.1037/0021-9010.78.1.98
• Cho, E., & Kim, S. (2015). Cronbach's coefficient alpha: Well known but poorly understood. Organizational Research Methods, 18, 207–230. https://doi.org/10.1177/1094428114555994
• Streiner, D. L. (2003). Starting at the beginning: An introduction to coefficient alpha and internal consistency. Journal of Personality Assessment, 80, 99–103. https://doi.org/10.1207/S15327752JPA8001_18

David L Morgan & Robert Trevethan is the above correct? And can we take chat further offline? I believe I have many more questions but do not want to complicate the thread.

My only comment would be with regard to the entry "Careful to regard alpha as a measure of homogeneity. i.e. not a measure of homogeneity." I would refer to alpha as a measure of internal consistency.

David L Morgan I have updated accordingly. Is it better aligned?

Should say "not a measure of homogeneity." Homogeneity is basically the same as uni-dimensionality (i.e., alpha cannot tell you if all the items measure the same underlying construct).

Jameel Isaacs. I think you've got a pretty good grasp on things, though I'm suspicious about alpha (apparently it's not really correct to call it Cronbach's alpha, and, instead, perhaps coefficient alpha or just alpha).

I could be a bit obtuse about alpha, but it seems to me that if something is internally consistent it is also homogeneous. In fact, Cho and Kim (2015) have argued that alpha carries a number of meanings, none of which is clear or justified. So, although this metric seems to be worshipped in some circles, I'd treat it with a pretty high degree of caution.

Here are some additional references about alpha that you might find helpful - and even entertaining:

Boyle, G. J. (1991). Does item homogeneity indicate internal consistency or item redundancy in psychometric scales? Personality and Individual Differences, 12, 291–294.https://doi.org/10.1016/0191-8869(91)90115-R

Hattie, J. (1985). Methodology review: Assessing unidimensionality of tests and items. Applied Psychological Measurement, 9, 139–164. https://doi.org/10.1177/014662168500900204

McNeish, D. (2018). Thanks coefficient alpha, we’ll take it from here. Psychological Methods, 23, 412–433. https://doi.org/10.1037/met0000144

Panayides, P. (2013). Coefficient alpha: Interpret with caution. Europe’s Journal of Psychology, 9, 687–696. https://doi.org/10.5964/ejop.v9i4.653

Sijtsma, K. (2009). On the use, the misuse, and the very limited usefulness of Cronbach’s alpha. Psychometrika, 74, 107–120. https://doi.org/10.1007/s11336-008-9101-0

That should keep you busy!

Apart from that, pls feel free to contact me privately via RG if you'd like to. I'm happy to keep on providing you with help if I possibly can. If you give me your email address in an initial RG message, that might be good. I find that email contact is more effective that communicating via RG messages.

I think of homogeneity as similar to uni-dimensionality, with the point that alpha is not a measure of whether all the items in a set are indeed measures of a single underlying construct. As far as I know, alpha was devised by Lee Cronbach, so that is why his name is attached to it -- but there might be more to it than that.

Anyway, alpha has a very simple interpretation in terms of factor analysis: if all the items for a factor had equivalent loadings, then its reliability would be equal to alpha. Another name for this is "unit weighting," meaning that each item in the scale gets an equal weight when they simply summed to get a total. This is why alpha is sometimes referred to as a "lower bound" for reliability: if there is substantial variance in the loadings, then it will underestimate the reliability of a properly weighted scale.

David L Morgan. With regard to Lee Cronbach and alpha, apparently he didn't "devise" it - and he even disassociated himself from it. Cho and Kim (2015) provide some interesting historical background concerning this metric. In fact, the first common misconception that they deal with in their article is that Cronbach developed alpha - and they quote him as having written in 2004 that "it is an embarrassment to me that the formula became conventionally known as 'Cronbach's alpha'".

I hope that's helpful.

Interesting bit of history. I will revise my future practice and just call it "alpha" without any attribution.

David L Morgan. For what it's worth, David, from now on I'm initially referring to it as "coefficient alpha" in my articles in order to indicate it is different from the alpha in p < .05, etc., but after the initial mention of "coefficient alpha", I just use "alpha". I'm yet to see what editors and reviewers say.