Hello Olivia,

You want to determine whether an individual student's score on a single item differs from their score on the same item on a second occasion? Without knowing what kinds of items (right-wrong? 9-point bipolar scale? 5-point Likert-type response scale? 0-100 confidence rating?), I'll have to be a bit vague, but the overall principles will be the same:

If so, I'd suggest: (a) you set a threshold for score agreement (e.g., exact agreement? agreement within +/- 1 unit?); (b) count the case as "agreeing" it it meets your threshold, and as different or disagreeing otherwise; then (c) report the results for your pool of 3 persons and 7 items (such as, what % of 21 possible agreements were observed?). Note that this is purely a descriptive approach. An N of 1 case (at a time), observed on one variable, offers no opportunity for any meaningful statistical test.

If you want to know, did the set of 3 persons have a change in scores across occasions on item "X", then you could run a statistical test, but the power of said test (no matter which one you chose) would be very low unless the differences were profound in magnitude. The same would be true if you combined the scores across the 7 items; the sample size is just too small. (Unless this is all just an exercise to show you can apply tests to data sets, in which case the appropriate test might be: (a) Wilcoxon signed rank sum test; (b) dependent t-test; (c) McNemar test; or (d) something else, the choice depending on the type of item involved and the specific research question.)

My advice would be to stick with the descriptive reporting and interpret the results accordingly.

Good luck with your work.

Hello Olivia,

You want to determine whether an individual student's score on a single item differs from their score on the same item on a second occasion? Without knowing what kinds of items (right-wrong? 9-point bipolar scale? 5-point Likert-type response scale? 0-100 confidence rating?), I'll have to be a bit vague, but the overall principles will be the same:

If so, I'd suggest: (a) you set a threshold for score agreement (e.g., exact agreement? agreement within +/- 1 unit?); (b) count the case as "agreeing" it it meets your threshold, and as different or disagreeing otherwise; then (c) report the results for your pool of 3 persons and 7 items (such as, what % of 21 possible agreements were observed?). Note that this is purely a descriptive approach. An N of 1 case (at a time), observed on one variable, offers no opportunity for any meaningful statistical test.

If you want to know, did the set of 3 persons have a change in scores across occasions on item "X", then you could run a statistical test, but the power of said test (no matter which one you chose) would be very low unless the differences were profound in magnitude. The same would be true if you combined the scores across the 7 items; the sample size is just too small. (Unless this is all just an exercise to show you can apply tests to data sets, in which case the appropriate test might be: (a) Wilcoxon signed rank sum test; (b) dependent t-test; (c) McNemar test; or (d) something else, the choice depending on the type of item involved and the specific research question.)

My advice would be to stick with the descriptive reporting and interpret the results accordingly.

Good luck with your work.

If a person's score is different, it is different. Your problem is interpreting that difference.

If you test a person twice, you expect their scores to change because of the inherent error involved in the testing process. If they get the same test twice, there is the added bias that on the second occasion they are familiar with the test. If, on the other hand, you give them different tests on each occasion, each test will have a different expected score because they are, well, different tests.

Furthermore, if you have done something between the two tests – say, given a teaching session – then there is no credible null hypothesis that says that your teaching session will have zero effect on knowledge. Indeed, even if interventions don't change things like attitudes, they help the person to identify the socially acceptable answers to the evaluation questions!

In this kind of situation, you have to specify the minimum change in score that counts as real and worthwhile. Real means that it's unlikely that the errors and biases involved in testing and retesting could have produced a change this big. And worthwhile means that a change of at least this size is required to justify the costs of the intervention.

And, of course, as David Morse has pointed out, you have so few people that you will have difficulty in drawing any conclusions.

n=3 is pretty small. What is your research question? Did you conduct a power analysis prior to collecting data?