Hi, I'm currently analysing a survey for a university course design, and some guidance would be very much appreciated.
The students who are enrolled in a (year-long) course were given a link to participate in an online survey (which consists of a series of questionnaires using a Likert scale) over the course of two semesters. Since this wasn't compulsory, we only have 66 responses from semester 1 and 116 from semester 2. Some students might have submitted their responses in both semesters, or just one of the two semesters, or none.
And with this premise, I would like to find out whether there is any significant difference in students' response in semester 1 and semester 2. So, for example, I want to compare the mean/median rank of Q1 in semester 1 with the same Q1 in semester 2.
Would this be possible in any way? What kind of test should I use? Thank you very much in advance.
If you wish to compare means across all the questions from the questionnaire you can use paired sample t-test
Your biggest problem is likely the low response rate. The responses you get may not at all be representative for the population you aimed to sample. You should critically evaluate how the probability of giving a response is linked to the content of your survey.
Hello it also depends how many options to chose you have in your survey. More detalies needed for modeling the calculation with less variables.
To verify the difference, use the T test if your dat are normally distributed.
If the T test are insignificantly different, then the data may be combined into one large sample collected from two locations. However, if you find that there is a significant difference, then the data should not be combined.If that is the case, you may want to run two separate regressions and compare their equations and investigate their respective F ANOVA.
Annie, first of all, you have a big trouble with your research design, if you want to compare the responses of the same population a paired design must be followed strictly. You said that you are interested in compare Q1 between semesters, but if you have a low response rate, then the sample size of the Q1’s would be reduced drastically, so the sample size problem would be worst. None t test would be done, even if you can paired your responses, the Likert scale is ordinal, then it’s preferable to use a non parametric test.
Best regards
I agree that in the optimum design both surveys would have the same (number of) respondents. One should strive for such optimum designs. Still, we sometimes must take our losses, and do with the outcomes of less ideal designs.
Having said so, as things are, I do not think that you could organize your data such that you could apply the paired t-test. With your kind of data I would have tried Welch’s t-test. Welch’s t-test can be applied to unequal sample sizes (i.e. unequal numbers of respondents) and the variances of the samples do not need to be identical either. Welch’s t-test is explained in https://en.wikipedia.org/wiki/Welch%27s_t-test
Do not report two-sample test p-values, because they won't be correct.
If some students may be in both groups, then the samples are not independent, which invalidates the two-sample t-test. Additionally, the differences in samples sizes between semesters is probably not random, so you aren't comparing groups than have been sampled in the same way.
I suggest you simply describe the data, giving mean(SD), median(min-max), and maybe overlapping histograms for each semester. The proportion of students responding the the survey each semester is also relevant. This is the answer I have given when I encountered very similar questions at my own institution.
PS: Too late now, but problems like this are why it's good to discussion design of a study before collection data.
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Sampling Design
- Sample Size