I think it is important to first get some more basic info before we can come to an answer. At how many populations/species are you looking? And most importantly, do you have a reference genome that may give you an idea about the linkage of your loci?
If you have a reference and an acceptable density of markers, I guess you can try to adapt some of the "commonly" used statistics into something that fits your system. See e.g. this paper:
http://arxiv.org/pdf/1307.4137.pdf
If no reference, or only a low density of markers, I guess you can still throw a guess by looking into shared and unique variants, population/species divergence and connectivity, and population size. But I don't think that has been done before.
Finally, if you are looking at diploids, deviations from HW equilibrium may perhaps tell you something about mechanistics that potentially underly balancing selection (e.g. a heterozygote surplus), and perhaps the homogeneity of allele combinations may tell you something as well?
Hi Francesca, I found this review useful to get the picture: "Detecting balancing selection in genomes: limits and prospects" http://onlinelibrary.wiley.com/doi/10.1111/mec.13226/full
Here, the requested information: I am analysing a single diploid species, and two populations. 50 and 100 samples have been genotyped through ddRAD (Peterson et al. 2012) and PoolSeq (Futschik & Schlotterer 2010) respectively. I obtained 76,837 (ddRAD), 755,810 and 61,270 (PoolSeq different flitering parameters) SNPs. There is no reference genome for my speces; for PoolSeq, that requires a reference genome, I used a cosely related species (mean alignment rate 80.36%), but I do not think it is advisable to use this for balancing selection analyses.
Thank you for the suggested references Tom and Rik. Unfortunately, most of the methods indicated there are not applicabile due to missing reference genome of the analyzed species, reference genome of closely related species that are appropriate for the study case, and candidate genes.
I'm currently using Bayescan to identify outliers (low Fst). I'm considering also to calculate Tajima's D bacause it is among the few statistics I can apply with my type of data, but it has many drawbacks due to demographic confunding effects. Do you think it is worth to do?
HWeq is commonly used to remove paralogs in RAD data filtering. Additionally, this method does not distinguish between the possible factors that make the locus deviating from HWE, hence I don't think it would tell me much more. What do you think?
I'm looking forward to receiving your comments and suggestions. Thank you again!