My co-author and I are working on a paper using the Economic Freedom of the World and 19 countries (sample cannot be increased because it is focused on a specific Region). The number of years is 19, then our sample is relatively short. We are trying to design a "non-orthodoxal experimental review", and apply diff-in-diff, but: 1) we do not have a specific treatment year, countries can be treated or not treated during all the period sample (treatment is if economic freedom score decreased, then country 1 can be treated in year 3, 4, 8, 10 (if its economic freedom score decreased those years) and not-treated in the other years that its economic freedom score did not decrease); 2) we are using an interaction model, my two interaction terms are treatment (1 if decreased the score of economic freedom, 0 otherwise), and intensity (the % variation of the economic freedom score), and other controls. We have presented in some seminars and conferences the research progress and early results of this paper but suggestions just focus on the "size of the sample", which we know could be a caveat but as I said the research intention is just the Region, the early results are significant and show the expected signs. We have not received suggestions on our "non-orthodoxal experimental review" (different treatment years, countries that can be treated or not in different years, our interaction model specification, etc...) My specific question is: Does anybody could give us some insights or suggestions on the methodology and how to respond convincingly that increasing the sample size is not an option for this research.
You might try to use the reference in the attached screenshot for the additional information on methods. David Booth
Hi Rafael,
To your question: “how to respond convincingly that increasing the sample size is not an option for this research,” I think the best answer is that you are focusing on a specific region and therefore your sample size is limited by construction to the focus of your research. You already have given that answer, and I would only add the following: When your results are statistically significant, that is despite (not because of) the small sample size. Statistical significance takes into account the sample size, and it will be more difficult to reject the null hypotheses with a small sample size than with a large sample size. Just make sure that outliers or high-leverage observations are not driving your results. This is easier to check with your sample than with a larger sample, but you can always do some post-estimations graphs. For your case, I recommend leverage versus normalized squared residuals to investigate the individual influence of each observation on your results. If any observation is in the top-right (high-leverage, high residual) and therefore is very influential, try removing it and re-estimate your results. I really like the idea of bootstrapping that Dauji Saha recommended, and I would also consider Jackknife because you would be explicitly leaving one observation out at a time, which would make your case even stronger.
In terms of your "non-orthodoxal experimental review" methodology, it would be convenient to know your dependent variable and the goal of your analysis to provide you with a more thoughtful answer. However, so far and based on your explanation above, I can see that your treatment is endogenous. That is, you do not randomly assign country-years to your treatment groups but rather you observe which country-years had a decline in economic freedom. Therefore, your treatment may capture part of your outcome and consequently, your results may be biased and inconsistent. I do not see any problem with your treatment being in separate, non-connected country-years except for the endogeneity problem mentioned above. In simple terms, the reason why some of your countries enter your treatment in a particular year could be because of (or driven by) the value of the outcome variable around that year (exactly what you are trying to explain and investigate), and therefore the effect you measure for your treatment would not make sense. Does that make sense? In terms of interacting your treatment with the intensity, that is interesting. Still, you are implicitly focusing on only a subset of your already reduced sample (those that decreased their economic freedom). That subset can be small if most countries in your Region have increased their economic freedom over time and therefore their country-year observations would fall in your “not treated” group and you may end up having many more zeros than ones, increasing multicollinearity. The problem with that is that your results could be unstable, and it would be worth a deeper individual analysis of each of countries that you are “treating.” In addition, there could be some measurement “errors” in your treatment group. For example, the economic freedom of country A was over estimated one particular year and then the following year it declines only to correct the error, but the observation enters your “treatment” group nevertheless. So, analyzing those observations carefully, one by one, is key. Finally, in terms of the interaction itself, note that it is an interaction of the intensity (defined by you as the % variation of the economic freedom score) with a dummy variable that depends on the intensity itself (your dummy = 1 if intensity=0). Therefore, the two interactive terms depend on each other by construction. Also, the resulting interactive term only captures the intensity for your subgroup of treated country-years, a subsample that can be small as explained above, even if your total sample size appears large (because the zeros of the “non-treated” group are counted in the sample size), which gives you more reason to investigate your “treated” observations more carefully. As a simple suggestion, I would also consider a non-interactive model first, including just the intensity (both positive and negative sides) plus your controls before you consider a fully interactive model or even a model with intensity squared or cubed if you want to introduce non-linearities in the model. Motivate well the purpose of all interactions (or other non-linearities) as a follow-up of your simpler model. I hope these comments help you.
Kind regards,
Hugo M
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Sampling Design
- Sample Size