I have a multiple response item consisting of 10 response options. Participants were instructed to choose 3 of them.
I want to know which one of those response options was picked the most.
Of course, I can simply see in the raw data which one was picked most often, but is there any way to statistically verify that?
For example, option A was chosen 50 times.
Option B was chosen 20 times. Option C was chosen 7 times. Option D was chosen 11 times. Option E was chosen 27 times.
Is there a way to determine whether option A was chosen statistically more often than any other response option? I've been breaking my head over google search results for the last couple of hours, but to no avail.
Hi Martijn,
Yes, one possible answer to your question is one of the first techniques, both historically and pedagogically, in statistics. It is called Fisher’s Exact method.
The test compares how many samples out of the total number of possible permutations would have results with that many A’s or greater and so on with the rest of the responses. Given how the number of permutations can explode exponentially the test is only feasible for small sample sizes.
With a larger sample size you could perform a chi-square test to test whether the distribution of choices will occur in five percent or less of repeated samples. That is known as the p-value and .05, approximately two standard deviations from the mean of the normal, is the standard for statistical significance.
My advice would be to Google Fisher‘s Exact test and the chi-square test and figure out what precise version of these tests is appropriate for your experimental design.
If you supply more particulars of the experiment, sample size, whether the categories are correlated, etc. I or another researcher with a knowledge of statistics can help determine the most appropriate test.
Best,
Aran Canes
The item in question asks participants to choose three things they find most important about some kind of service (further elaboration is irrelevant). They can choose three items out of a list of ten. These items could be loosely related, but there's no theoretical nor any statistical reason to assume there's anything more to it. Think of items like "friendliness", "timeliness", "cost transparency" and the likes.
The sample size is about 150 respondents, which is currently still growing. We are aiming at about 250-300 total. So it's not a very small sample size. Chi-square is something I came about in my Google searches as well, and though I have some experience with this type of analyses, I'm not sure such a test would fit my purpose. To my knowledge, a Chi-Square test is most suitable when crosstabs can be applied. That is, one variable is represented in the rows, and the other in the columns of a table. In other words, at least a 2×2 design. In my case, I only have one row, with ten columns representing all of my response options.
Usually, when I experience such problems with my analyses, I simply go for a trial-and-error approach and see what sticks. In the case of Chi-Square tests, I was hoping it would somehow work out if I didn't enter a variable for the rows. It turned out all the Chi-square test did, was test if the number of times a single response option was chosen differed significantly from the number of times it wasn't chosen. In other words, it tested within each response option rather than across. For example, if about half of my respondents found "timeliness" to be an important aspect of the service quality, the Chi-square test returns a high p-value. The difference between the number of people that did and didn't tick that box is very small, since about half of them ticked that box. Conversely, large within item differences yield low p-values. Is there a way to force the Chi-square test to look between items and only look at the frequencies? Maybe a new database file has to be created in order to do so?
It must be noted that I am not very experienced with statistics. I've done some research here and there, but this is about as far as my knowledge reaches.
To perform a chi-square test on this data you will need to divide each respondent and each response into chosen/not chosen. Then you will have s 5x2 table with 450 data points. The chi-square test will determine if the frequencies are, using loose language, likely to be the result of random selection or exhibit a non-random pattern.
Good morning Martijn,
Yes, it should be a 10x2 table. I thought there were only five choices but now I see you wrote that there are ten possible responses. Please let me know if the chi-square test does not satisfy the needs of your research.
Best,
Aran
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics