Yes, severe sample imbalance especially in a moderation analysis involving a categorical moderator like age with only a small number of older participants (n = 38) can raise serious concerns regarding the reliability, stability, and generalizability of your results. Here's a detailed breakdown of the issues and potential remedies:

The presence of severe sample imbalance in your study—particularly the small number of participants in the older age group—does raise concerns about the reliability of your findings. Your study follows a 2×2 factorial design, where both your independent variable and the moderator (age group) have two levels. Ideally, this design assumes balanced group sizes, which is what your G*Power calculation (with an effect size of 0.25, power of 0.8, and alpha of 0.05) was based on, requiring 32 participants per cell for a total of 128. However, in practice, your collected data includes 511 participants with highly uneven distribution across the four groups (243, 15, 230, and 23), meaning that the older age group has only 38 participants in total. This imbalance leads to two main problems: low statistical power in the smaller group and instability of estimates, particularly in subgroup analyses. In your case, while the moderation effect of age was found to be statistically significant, the subgroup regression showing significance only in the older group may be statistically fragile, as estimates from such a small group are likely to have large standard errors and can be influenced by outliers.

Moreover, an imbalanced sample can violate assumptions such as homoscedasticity (equal variances), which undermines the validity of standard hypothesis testing methods. Subgroup analysis—particularly when the group sizes are small and not pre-specified in a study’s hypothesis—can increase the risk of Type I errors, or false positives. These factors suggest that while your findings may be suggestive, they should be interpreted with caution due to the possibility of statistical artifacts arising from unequal group sizes.

To address these concerns, several remedial strategies can be considered. One effective approach is to use bootstrapping, a resampling technique that can generate more stable estimates of standard errors and confidence intervals without relying on large-sample assumptions. This can help assess the robustness of the interaction effect in your regression model. Another option is to apply robust regression techniques, such as using heteroskedasticity-consistent standard errors (e.g., HC3), which help correct for the unequal variances across groups and provide more reliable p-values. You might also explore weighting methods or propensity score matching to adjust for the disproportionate group sizes, allowing for more balanced comparisons. Additionally, conducting a sensitivity analysis—in which you test your model with and without the underrepresented group—can help determine the extent to which the imbalance affects the observed moderation effect.

Jamesloue Belecario Cajes Thank you very much for your detailed and thorough response!

Just reduce other size. The study can work with less old people. Just sample the other four people and watch the balancing on the new dataset. Instead of increase a data group reduce the other data, usually size is not really important to recollect Big data when You just need to describe properties and not quantities. For example on cern data size is just used to calculate constants. Good day.

Ivan Avila Caceres Thanks for the advice!