As a result of exploratory factor analysis in some research about anxiety experienced by students when they study a foreign language, we have identified four distinct domains within which the students' anxiety falls. When we obtain composite scores on each of those factors by adding responses on the relevant items and dividing by the number of items, the four scores differ noticeably.
The students are anxious on some domains, but not on others.
I am not sure whether we can legitimately assess whether the differences between the factors are significantly different from each other or not - e.g., whether we can say that a particular type of anxiety is significantly greater than another type of anxiety.
Something tells me that conducting that kind of analysis might not be permissible, but I'm not sure.
Incidentally, if it would be permissible to conduct a series of comparisons, I assume we could conduct either paired-samples t-tests or Wilcoxon signed-ranks tests. If my assumption is incorrect, I would appreciate information concerning why.
I think there would be six comparisons between the four types of anxiety - which I assume would necessitate some kind of adjustment to the critical value of alpha - perhaps by dividing .05 by six as a Bonferroni-type adjustment. Is my hunch about that correct?
I assume that the research question is: is the variability in means for the different ITEMS larger than would be expected by chance. If so, you are changing it to are the means of some items that are correlated different from some items that are not correlated within those samples. Am I correct on this?
Andrew Paul McKenzie Pegman: Thanks, Andrew, for trying to help. Actually, we had used Likert-type scales to obtain the data with which we conducted exploratory factor analysis (EFA), so, by conducting EFA, we are a stage beyond the "Likert level": We have moved from items to factors.
Furthermore, given that scores we have on each of the factors are at the ordinal level of measurement (and might even be regarded to be at the interval level for some purposes), I think we would be losing information if we were to use chi-square analyses, so I am reluctant to go down that track.
But, again, thanks for trying to help.
Daniel Wright: Thanks, Daniel, for pursuing this with me. I hope I've understood your question by responding to say that it's not individual items we want to compare, but the composite scores obtained from having added groups of items according to the factors on which they loaded.
After we added responses on items for each factor, and divided the result by the number of items in each case, the four factors, with their means (SDs) are the following:
Nervousness: 2.6 (0.84)
Self-Doubt: 3.0 (0.84)
Avoidance: 2.3 (0.86)
Discomfort: 3.1 (0.99)
I'd like to check whether it's permissible to see whether our participants score significantly higher or lower on some / all of the above factors than on other factors. For a start, I suspect that the scores on Avoidance are probably lower than are the scores on all three other factors.
My concern is that, because we're comparing scores on different factors, it might be similar to assessing, say, whether students' height is different from their intelligence - which would clearly be silly because the two variables have different bases.
When put that way, my concern is that the four factors might be regarded as having different bases. But that might not be the case: They are all based on five-point Likert response options.
I hope I've explained our situation appropriately. I'm still wondering whether it would be legitimate to conduct paired comparisons across the four factors.
Andrew Paul McKenzie Pegman, sorry, I didn't for a moment intend to be rude. I simply can't see how your suggestions apply in our case. Maybe I hadn't explained our situation adequately.
Andrew Paul McKenzie Pegman , your response to Robert Trevethan thanking you is surprising. There are many procedures that you can look up using a chi-sq table, so that part of your "answer" isn't answering anything. Without knowing what procedure you are talking about, we don't know if it is "standard" or not.
Robert Trevethan , so the issue seems to be whether these composites having been determined in part by EFA, if there is a bias for high and low ones to cluster together. This would likely depend on the distributions. It would be less problematic if the groupings were just determined by SMEs.
Daniel Wright, here's my ignorance showing: Sorry, but I don't know what you're referring to by SMEs. Could you explain, please, how SMEs might provide a better idea of how grouping would occur.
Having said that, I wonder whether we're drifting from my original problem. In essence, I want to see, for example, whether the participants' scores on Discomfort are significantly higher than are their scores on Avoidance.
Sorry. SMEs being subject matter experts. My concern is not knowing what the constructs Discomfort and Avoidance are, expect by the EFA. It were scores on two separate sub-scales, so you knew beforehand how you would make them, that would be different.
Daniel Wright, thanks for the definition of SMEs. I'm used to them being small to medium enterprises because of editing work I do for people in business studies - but I was pretty sure you weren't referring to those SMEs. So I drifted off thinking of possibilities such as standard measurement errors that didn't make much sense to me .
To come closer to answering your question, there is quite a lot of prior research about the topic we're investigating (different types of anxiety experienced by people when they learn a foreign language). Although a lot of the EFAs in that prior research are badly flawed in my view, I think we're on quite solid ground in thinking that one type of anxiety (and factor) is discomfort in our data, and another type of anxiety (and factor) is avoidance. Each of those factors is indicated by a distinct subset of items from an original set of 33 items.
My concern is whether it would be permissible to use stats such as paired-samples t-tests or the Wilcoxon signed ranks (matched-pairs) test to see whether the students were higher or lower when comparing their scores on the four different factors. I don't want to be erroneously comparing apples with oranges.
I hope that makes sense.
It is comparing apples and oranges, but no problem with that (Adult Dan eats way more apples than high school Dan, who ate way more apples than current Dan). First step would be a scatterplot, and draw the diagonal. The t-test is just distance from the diagonal. And of course you can transform a t value into a chi-sq value ;o)
Andrew Paul McKenzie Pegman, I notice you've removed a couple of your posts higher up, which could result in some readers of this thread being mystified about some of Daniel Wright's and my posts.
Given that you had indicated, in one of the posts you removed, that you were sure your initial suggestion was correct, I wonder whether you would be willing to explain how you think I could conduct the analyses using Likert scales and chi-square tests. I acknowledge that I could well have been premature in thinking that your suggestion(s) might not work.
Daniel Wright, thanks for persisting. My concern, however, is a bit different. I'm not wanting to find out whether the three-aged Dans eat the same number of apples or not, but something closer to whether adult Dan eats more apples than he does oranges.
Hmm. I think that's the issue if we're converting the situation to fruits . . .
- Badges
- Science topic