In general, you want to be sure that the pre- and posttests are comparable. If you are concerned about the effect of repeated testing, one way to do this is to make 120 multiple choice questions and then calibrate two equally difficult tests by having completely different students (not participants) answer all 120 items and randomly dividing the questions amongst the two ensuring that the average score and standard deviations comes out more or less the same across the two tests. Then, you should, theoretically at least, have two equal tests. That is what I did for a similar phrasal verbs experiment I conducted. You could also borrow my questions from my study (send me a message) or work off of some of these items.

The trick part about phrasal verbs is that it is not always the items themselves that are difficult, but the meaning senses chosen for the question.

When doing your analysis, be sure to use a combined model. A common mistake I see is to do several t-tests. You need a two-way ANOVA (assuming normality) and to look at the interaction effect.

For some of my papers on phrasal verbs:

Spring, R. (2018). Teaching phrasal verbs more efficiently: Using corpus studies and cognitive linguistics to create a particle list. Advances in Language and Literary Studies, 9(5), 121–135. https://doi.org/10.7575/aiac.alls.v.9n.5p.121

Spring, R. (2019). Using short animations to help teach phrasal verbs from a cognitive typology standpoint: Implications from two years of comparative data. STEM Journal, 20(4), 105-122. https://doi.org/10.16875/stem.2019.20.4.105

Spring, R., & Takeda, J. (2024). Teaching phrasal verbs and idiomatic expressions through multimodal flashcards: Confirming the importance of multimedia-based instruction. Journal of English Teaching through Movies and Media, 25(2), 40–53. https://doi.org/10.16875/stem.2024.25.2.40

The first paper is the one that has a pretty well balanced 60 question phrasal verb pre and posttest. Let me know if you want the items.

However, the methodology should look more like what can be found on this page (in case you't not familiar - if you are, please disregard):

https://springsenglish.online/stats/

I'm sorry for the unclear title of my original question. I realize now that it didn’t fully capture my main concern.

To clarify: My key question is actually about the overlap between the content in the intervention and the two tests (pretest & posttest).

Specifically, is it acceptable for the intervention to directly use the same set of phrasal verbs that are tested in both the pretest and posttest? Or would that be considered a threat to validity?

Thank you again for your time and valuable suggestions

I personally would not reuse the pre-test list in the intervention because that allows for simple memorization of the answers for each item, rather than learning of the general principles in the intervention.

You could use a variation of the suggestion from Ryan Spring and develop a second list for the intervention, and then use the original list for both the pre- and post- tests.

Hello Rean Nguyễn - I think it is fine to use some of the same phrasal verbs, potentially even in the same meaning senses. Depending on what, specficially, you are trying to test, there might be no way around it. You can, however, change up the questions. Some of this gets taken care of by statsitics if you have a control and experiment group: both groups have the same repeating advantage, so if there is a difference in improvement, the treatment was still probably more effective than the control.

Another thing done in a lot of langauge learning studies (which I didn't have the insight / foresight to do in my 2018 paper, though I wish I had) is to include a third, delayed post-test. For example:

Pre-test

(3 weeks of treatment)

Post-test

(3 weeks of nothing)

Delayed post-test

That tests two things: (a) how much staying power the treatment had (the end goal of langauge learning is to remember it much later and to have truly acquired the skill or langauge chunks), and (b) having the delayed post-test probably reduces the amount of impact from having simply memorized the pre-test questions.

To answer your question (Specifically, is it acceptable for the intervention to directly use the same set of phrasal verbs that are tested in both the pretest and posttest? Or would that be considered a threat to validity?): Yes, this can of course be done. What the post-test measures is then the learning that results from both the intervention (on, say, phrasal verb 1-20) and (in sense of 'plus') the learning that results from the pre-test (on phrasal verb 1-20). Every test is always a learning exercise, because students process the foreign language while doing the test (although we tend to not think about tests in this way). - Also: You can solve your whole study design question by turning the whole thing back on its feet. Do not ask starting from the study design (does it work for x?), but from the research question. What do you want to find out about? What is the appropriate design to find out about this (which design works to examine y)? Then everything will fall into place naturally.