Hello,
I am currently preparing a preregistered report for experiments in different countries. I am studying motivations/values of students; therefore, I want to conduct several same experiments (containing mixed subject designs) in different countries.
I have no hypotheses regarding country differences. I just want to expand my sample size to be more representative in (in my case) western European countries.
My analyses are planned for each country separately and results should be presented for each country separately.
I have read somewhere that in this case, I should check if the factor country really has no influence on students’ performance, i.e., that my results are independent of the factor country.
I don’t think that I could just conduct an ANOVA using country as an IV to examine non-differences on students’ performance because then I would have to test if I can actually compare the data sets at all (measurement invariance; either ways my sample sizes for testing measurement invariance would be too small).
How are researchers usually facing this problem in cross-country/cross-cultural research? Especially when they don’t have any hypotheses regarding country differences and are not planning to conduct analyses to compare countries?
What could I state in my preregistered report?
Thank you very much.
Use Hierarchical/Multilevel models. See e.g.
http://www.stat.columbia.edu/~gelman/arm/
To expand on Jochen Wilhelm 's point.
If you do not have any prediction or model that incorporates "country" then you simply want to account for the variance on your estimates that it may produce. This is the same as when we account for the auto-correlational nature of data from the same person (i.e. more similar) than between persons (i.e. more different) by using a person random-effect in our model.
Thus, you can code the variable country (e.g, just have a nominal variable with each country name). And then when you run a mixed-effects model on your data, you have Country as one of the random-effects.
Small note, the number of level of a random-effects variable should exceed the number of variables in your model and levels of your fixed-effects (e.g, if you only have data from 4 countries in the end, but your analysis is conducted on 5 factors, then it is not appropriate as there will not be enough data to make stable estimates).
If you are doing regression analysis, then you should avoid creating a single "nominal' variable because the software has no way of knowing that this is a set of categories. Instead, you should create a series of dummy variables. If there are N countries, then you will need to create N-1 dummies, which will all be compared to the omitted country (so it is wise to choose a meaningful category for the omitted country, because all the other countries will be compared to it).
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Sampling Design
- Sample Size