Hi Plamen, no the significance test would not have the same meaning as you will only have 25 cases (one for each country) rather than millions of cases if you had individual-level data and therefore a lot less statistical precision. Of course the individual data would also have to be individual per capita GDPs for each person (perhaps measured as a proxy measure, e.g. gross salary) as well as whether each individual person was religious or not, so perhaps this data would be very hard to get. Ideally you would want to create a multi-level model to account for clustering in the data (by country, but also by smaller units of people who are likely to be more similar to each other as they live in specific areas).

Do you have the results of regression models for individual countries? if so, you could work around not having the individual-level data by using meta-analysis of the results of each country (odds ratio or relative risk of religion given GDP per capita). Otherwise, if you know the distribution type and standard errors for each of your variables for each country you could simulate individual-level datasets for each country and run the multi-level model I suggested above.

Hope this helps, Tim

Hi Tim, thanks a lot for your answer. I actually just gave a hypothetical example to make my question simple. In my case I am performing a multiple regression analysis trying to explain which factors have importance on destination choice by Bulgarian migrants to other EU countries. I have 22 cases and the variables I am currently testing are: GDP per capita, unemployment rate, geographical proximity, cultural similarity index, presence of legal restriction on employment and hostility against immigrants. Most of the results are as hypothesized but only half of them are significant at the 10% level. The rest are very signification. The R2 is quite high, but I am wondering how to present the results of the significance test in my written work. Do I have to say that those factors are not proven to have importance on the choice making?

Hi Plamen, 22 cases is really not enough to do a multiple regression with 6 explanatory variables. As a rule of thumb you should have at least 10 cases per variable you add to a model in order to have adequate statistical power to answer a question. So I would say you can only say that you don't have enough data to answer which factors are important in choice making. I would also be very cautious about using the word 'prove', as even with high statistical significance (which you don't have) an association indicated by regression does not mean there is a causal relationship. In the absence of individual-level data I would probably rely on qualitative studies more than the quantitative data you have...

This is a really helpful answer. I am surprised that my professor didn't explain this to me.

I also start to believe that qualitative approach would be more appropriate as the results seem pretty random and the significance for some variables is close to 1.

But still wondering regarding the rule of thumb you mentioned. Would it be sensible to use only 2 variables in several models? In this case the rule would apply but, still, would the results make sense?

You could use only 2 variables in several models if you think that the other variables do not influence the relationship between the 2 variables (i.e. that you don't need to include more of the variables together in a multiple regression). However, you won't gain any power overall as you will need to correct for multiple hypothesis testing (if you do 10 models, just by chance one of them will have a p

Thanks a lot, Tim!

A couple of points that occur to me:

=> For your problem, the operative values are aggregate variable values since these are the values the migrants know about and I think you are hypothesizing determine their choice of destination country. Their personal income and employment status is not irrelevent but the choice of country is presumably based on aggregate values of the destination country so the simulation option probably doesn't work out.

=> With only 22 cases and at least 5 degrees of freedom in your model, you are way overexposed to overfitting. The rule of thumb is at least 10 obs per degree of freedom

=> And yes, using your approach, you would have to say the factors are not proven.

=> Another approach to consider is viewing your data as multinomial outcomes. That is a model that makes sense to me on first principles. Then you could write the success probabilities in the desired functional form and use maximum likelihood to estimate the effects of the variables of interest. Overfitting might still be an issue ...

Thank you for your ideas, David! I think with my statistical knowledge I won't be able to perform this analysis though...

And this is where you might want to collaborate with David to get the most out of your work by combining your interest and expertise in the subject matter with his expertise in statistics and modeling.

@Plamen/John It is an interesting problem and I have some interest in following up on this idea to see how it works out. I have a day job, so some schedule flexibility would be required. I actually am trying to build up my consulting experience since this will soon enough be my day job,

That would be really great, I am always eager to learn new methods in statistics but the problem is that I am really pressured by the time. I already decided to change the approach to a qualitative one and started writing. Thanks for the offer though!

A potential issue in your model is lack of independence among your predictor variables. I'm wondering if GDP per capita x unemployment rate, geographical proximity x cultural similarity index, or presence of legal restriction on employment x hostility against immigrants are not correlated. Significance tests for correlation, even Spearman rho, are pretty robust and could be used to drop-off some of candidate variables before modelling.

Perhaps they are statistically correlated, but not theoretically, apart maybe for "presence of legal restriction on employment x hostility against immigrants".

If statistically correlated, you should pick one, not both. It's a math issue, not theoretical, and you will notice no decrease in R2.

I understand, but I mean that they could be only randomly correlated. How can I prove my hypotheses if I don't use at least one variable for each? The R2 is pretty stable - at .60 for the full model. The adjusted R2 is not so high though.