I think that you should try to increase your population to improve the statistical power of the analysis, the idea would be to have more records of recombination events and this is only possible with more individuals. To infere associations of genetic variants with the phenotype is not that accurate with just 54 individuals... perhaps if you could have at least 100?