Hello Joanne,

The basic design sounds as if you have one, not four, dependent variables even though you are identifying four scores (Stim_A, early; Stim_A, late, Stim_B, early, Stim_B, late). It's a 2-within subjects factor design (e.g., repeated measures, with four total combinations.

Can you use categorical covariates? Yes. And, with only two levels, all you need do is code them as 0,1 and proceed.

Would it be better to incorporate them as blocking (between-subjects) factors? Maybe. A lot of the recommendations from literature on blocking vs. ancova hinge on: (a) the strength of the linear association (stronger argues for covariate, lower and/or nonlinear argues for blocking factor); and (b) the number of levels of categorical nuisance variables (implications for error df in the model, which would be more critical with small N; this latter concern isn't likely an issue for you).

The real question is, if you didn't plan to incorporate the two extraneous variables into the analysis prior to data collection, should you add them after-the-fact by having found correlations with the outcome scores? The usual admonition here is, avoid this practice. The reason is, that by cherry-picking among a veritable infinity of possible demographic differences, you're possibly overfitting your model in a way that may not generalize and therefore biasing your results.

It's too bad you didn't incorporate the reversed sequence of conditions in each study as part of a randomized, counterbalanced design. That would give you a stronger basis for asserting whether stimulus sequence affects results.

Good luck with your work.

As I read it, the basic design is a 2 (order) x 2 (stimulus type) x 2 (trial position) mixed design with order as a between-Ss factor and the other two as within-Ss factors. And you are asking about the feasibility of adding two more between-Ss factors, each with 2 levels:

• English v no English
• Exposure to multiple languages (yes/no)

Do you want to include just the main effects of those additional variables, or do you want to include some or all of the possible interaction terms too?

What is your sample size?

What was the response on each trial, and how was response accuracy measured?

How did you deal with the 20 trials per stimulus? Did you use a mean for the 20 trials? Or a median? Or some other summary statistic?

Thanks for clarifying.

Thank you to the researchers who kindly responded with advice. I will address both separately:

To Mr David Morse

I apologise that my initial explanation was unclear. To clarify, I included questions on language history during data collection due to the empirical evidence suggesting that language ability can influence performance in similar tasks to mine. I collected demographic variables like age and gender only for group matching purposes, not as variables of interest in the main analysis. The remaining questions were on their language history.

Including these questions was a precaution I took because I wasn’t able to be too selective when recruiting participants due to resource constraints, so it was more pragmatic to control for any potential differences statistically. Running correlation analyses was a precautionary step to explore if past research findings might have affected my results. I appreciate that this isn’t an ideal situation and also share your concern that it can look like cherry-picking. I am not sure myself, if I’m honest. Do you have any input on this?

It is worth noting that the ANOVAs yielded significant main effects with large effect sizes, with or without controlling for the demographic variables. This is partly because there weren’t as many eg. non-native English speakers (non-significant) to influence the data, and partly because the mean differences were not that large. The significant results are obviously “cleaner” if I were to only use, say, native English speakers, but my intention is to at least address that these differences do exist.

Regarding my study design, yes it involves participant randomising and counterbalancing. Participants are randomised according to their demographic variables (including language history) into one of the two groups, and within each group, the trial order is also counterbalanced between groups. So both groups should be matched (at least in theory!). The reason why I did not incorporate both groups together is that the impact of Stimuli A on B is not necessarily interchangeable with B on A. This is something I am exploring as there are inconsistent findings from the literature. If I were to incorporate the groups as another factor, the analysis would become much larger and too complicated.

But yes, it was a consideration I pondered on extensively early in the design process, so I can completely see your point! :)

To Mr Bruce Weaver

Thank you for your advice! I have provided more information on my study rationale and design in my response above on why I am analysing the groups separately. I hope it makes sense but if not, please do share your feedback.

I can see what you mean, a mixed design does seem the most appropriate and have followed your recommendation and ran two 2 (Block) x 2 (Trial position) x 2 (Native language) x 2 (Language exposure) mixed ANOVAs (one per group).

To address your questions, I am interested in whether these additional variables have any impact on the size and relationship of my dependent variables. The ANOVAs for both groups showed significant main effects of Block and Trial position. The only significant interaction was Block x Native language interaction. Block x Language exposure was “close” to significant (p = .090), but I am treating it as non-significant.

I’m not sure if this is correct, but I then ran separate ANOVAs on native and non-native English speakers and found that in the latter, there was a non-significant main effect of Block. Native English speakers were significant. Main effects of Trial position were significant for all levels of all factors for both groups. Is analysing Native vs Non-native separately the correct thing to do?

My sample size is N = 48, with n = 24 per group. I estimated the minimum sample size using G*Power prior to recruitment and it cited 18 per group for 80% power at a medium effect size. I recruited 24 to allow exact counterbalancing of the trials (there are 24 different permutations).

In each trial, participants are to provide a response and I calculate the accuracy as a proportion. I then calculated the average of the early trials and late trials, respectively. The number of trials considered “early” vs “late” is a bit complicated to explain without going into detail on a specific line of research, but they are based on a different statistical analysis that I ran beforehand. This ANOVA is unfortunately only one part of the entire study’s analysis plan, so I am trying to keep any exploratory analyses as concise as possible.

Thank you so much for your recommendation as I had been very unsure whether to proceed with a mixed design.

To both, thank you once again for all your feedback. I hope my explanation adequately addresses your questions and concerns. Thank you again for your valuable input. Please feel free to let me know if you have any questions or further input, and I would be more than happy to provide more information.

Best regards,

Joanne

Hello again Joanne,

1. If you did balance groups by these features and randomize to condition, then the blocking approach makes more sense. Your original post wasn't clear on this (as well, it sounded like only the second sample was counterbalanced for order of stimulus condition.)

2. Given that, I don't think covariates are the way to go; the allocation design implies that the features should be built in as factors.

3. I'm not sure I see how running two independent study sets is any less complex than trying to build that in as a variable. You would be correct to be concerned about too many IVs in a sample of the magnitudes you're describing, however.

4. Finally, the follow-up tests of native speakers for block differences, and non-native speakers for block differences are two of the four possible simple effects tests partitioning information from the native X block interaction which you observed. So, yes, that's suitable. (There is a way to do this without having to run separate anovas, however, so the error terms may differ as a result.)

Good luck with documenting your project.

Hi David,

Thank you for such helpful input. Could I clarify your response in no.4 regarding the simple effects tests? Were you referring to t-tests? I've conducted paired t-tests on native and non-native speakers comparing their respective performances on blocks 1 and 2. However, I also noticed that there is an option for simple main effects in SPSS (I'm using version 28), which I also ran (with Bonferroni correction) to get pairwise comparisons.

I'm sorry for the continued confusion on my part. I'm less familiar with three-way ANOVAs and unsure of the right steps. Thank you so much, I genuinely appreciate your patience with me!

Best regards,

Joanne

Hello Joanne,

Yes, the simple main effects tests would subsume the specific t-tests you ran (e.g., the t-tests are two of the four possible simple main effects for that interaction). The only possible distinction would be which error term spss applied for the simple main effects.

Good luck with your work.

Hi David,

Thank you! That makes everything so much clearer. I apologise again for taking so long to grasp it. You have been very helpful, so thank you again!

Best regards,

Joanne