it depends on your study. if you're analyzing whole genome data (as GWAS) with small groups, a pVal of 0.05 is not enough to filter your variants. in this regards, lowering the groups will lower your statistical power and of course give you insignificant results at end. you should give us more details on your analysis plan.