Well, that's a three-aspirin headache for sure!

You are certainly stuck with the problem of being unable to identify individual students, so you can only analyse the two groups of responses. I would use robust variance estimation as a way of dealing with the unknown degree of heterogeneity in the data.

Before you throw away the people who answered the same for all items, have a look and see if there is a pattern – for example, all 18 items indicated no anxiety at all. This may indicate that there is a test floor or ceiling effect – data piling up at the extreme end(s) of the scale.

There are a number of ways of approaching the analysis, which you could do by quantile regression or ordered logistic regression. One interesting alternative is fractional regression, where you rescale your scores between zero and one. This takes account of the bounded nature of the scores. Even if you don't have Stata, the folks at Statacorp make their helpful help materials available online. See https://www.stata.com/manuals14/rfracreg.pdf

Hello Nicole,

I certainly agree with Ronán Michael Conroy that this is a harsh introduction to the realities of data collection in which hindsight is so much clearer than foresight!

I would suggest that:

0. You consider investigating whether a pooled / summated / averaged response across your questionnaire items makes sense in the first place. In other words, evaluate the dimensionality of your measure (say, via factor analysis).

1. You run a comparison of the two groups at pretest (using possibly a Mann-Whitney or, if you're sufficiently confident that the summated items score behave as a continuous variable, an independent t-test). Any differences here would suggest that the groups were not equivalent at the outset.

About the only option I see for the posttest is to look at the posttest results, again as two independent groups. You could compare a robust confidence interval estimate of any differences against the a priori difference that was observed.

2. You'll need to define some minimal level of scale completion in order to accept a case as having furnished the desired data. Whether this is 50%, 70%, 90% or some other fraction is up to you and your team to decide (and defend). For any value less than complete data (100%), I'd suggest you either: (a) compute an averaged score for the items that were completed and use this instead of a summated score; or (b) look into whether imputation (something more profound than mean substitution, please!) makes sense given the nature of the missing data in your set.

3. I agree that arbitrarily tossing cases for using the same response option may be unfairly characterizing legitimate responses as meaningless. Some scales include "validity" items, which would call for a different response so as to help flag cases where thoughtless "Christmas treeing" completion methods were used.

Good luck with your work.

Thank you so much for your help, we will start researching those analysis suggestions. Ronán Michael Conroy David Morse I very much appreciate it!

In addition to factor analysis, as suggested by David Morse , I'd recommend Mokken scaling, which produces subscales from your data that meet a strong definition of unidimensionality. It's a nonparametric, iterative procedure, and can be done in Stata or R. It's good for extracting subscales from an item pool, or for simply identifying and discarding substandard items (items that fail a test for scaling based on Loevinger's H).

For question 2, I suggest to keep record of how many students did not answer. As you are testing the Anxiety so may be due to anxiety(fear of reporting) they did not answer. You can make graph for it.

Dear Professors Ronán Michael Conroy and David Morse . Very much thank you for your advice. After performing a Factor Analysis in each group (control group pre test, control group post test, experiment group pre test and experiment group post test), RStudio indicates that I should have 3 factors for each group. In addition, it indicates that the number of components are 1 for the control groups and 2 for the experiment groups. Here, I some how got lost because, it is confusing for me how to apply the Mann Whitney test if I have three factors. My guess is that I just coerce the number of factors to 1 when obtaining the factors, so I will just have one single variable? Am I right? If yes, is it ok to apply a normality test to determine the normality of this new variable and if it is normally distributed, can I apply a t-test for independent samples (comparing control and experiment groups)?