Let's say that I'd like to compare male respondents' and female respondents' perceptions of government across 30 countries, and test if there is statistically significant gender differences in perceptions across countries. The data is collected from survey data conducted in 30 countries. Then, simple t-test results of each country sample shows that two group means are statistically different in all countries. But if I want to claim that my analysis shows that there exists statistically significant gender differences in any countries regardless of different values of country-level variables, including polity score, GDP, and gender equality index, what is my next step other than regression? I mean that I want to show there exists gender differences in any countries in my sample regardless of different values of the above country level variables. I don't need to do regression, but I just want to show group means differences across countries.
There are a lot of issues about whether NHST is appropriate here, but I think what you are looking for is a graphical method that helps you to say if Country X is "significantly different" from Country Y. So first adjust the size of the intervals for the multiple comparisons and then see many sources (my fav is https://www.bristol.ac.uk/media-library/sites/cmm/migrated/documents/graphical-presentation-of-a-collection-of-means.pdf). For t-tests, the pairwise.t.test function in R is also I think the sort of thing that you'd be interested in.
If I understood you right, Meta-analysis is what you are looking for.
As I understood your problem, you are looking for a way to make a gerenal statement about the mean difference between males and females across all countries (or irrespective of any particular country).
If so, the simples approach is to just pool all the data. However, the data within each country/study may be correlated, so that it can be advantagous to include a random intercept term for country/study in your model (what then is a mixed or hierarchical model). You should also include as covariables all known or resonably suspected variables that may affect the male/female difference. You may further include weights if the studies are of different quality.
A not on your wording: you were asking if there "exists statistically significant gender differences". This is an unfortunate wording. Statistical significance is not something that is "discovered by data", it is a characteristic of the data itself, and it tells you only that the sample size iyou have is large enough to interpret the sample estimate (here: a mean difference) given the noise/uncertainty in the data with respect to the tested hypothesis (here, presumably, that the expected mean difference is zero, in which case a "statistically significant result" would mean that you can at least interpret the sign of the sample estimate). It's an entirely different question if you are able to show a relevant difference. Defining what "relevant" means is a subject-matter question and required domain knowledge. If this is given (and it usually is not!), then one would check if the confidence interval (or actually the posterior interval) is completely ouside of the range of "irrelevantly small differences".
I would see this as a two level multilevel model with respondents at level 1 nested within countries - such models naturally take account of the expected dependence with countries in terms of the response variable and thereby get less biased standard errors. You can include variables at each level about respondents and countries. Including a constant and a gender dummy in the fixed part of the model will allow you to assess the average gender differential across the 30 countries , allowing the gender differential to vary at the country level will identify countries that reinforce or negate the average differential. It is also possible to model within country between respondent heterogeneity by gender. Chapter 7 in this demonstrates what can be done with categorical predictorsBook Developing multilevel models for analysing contextuality, he...
For background there is a lot of support material here : http://www.bristol.ac.uk/cmm/software/mlwin/mlwin-resources.html
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Regression