Some measure of heterozygosity might be a place to start, and at least in my mind seems to expand to polyploid individuals relatively well. Are we talking SNP data? If so, what is the typical range of alleles at a given locus? How rare is it to have four different bases at one locus? 

Hi colleagues and thanks for your responses!

To have 4 different alleles depends on the locus (we used microsatellite loci). I also imagine an expansion of "diploid" heterozygosity into this tetraploid case. If one allele is scored, say A, then there is no variation and individual "heterozygosity" would be =0. In the opposite case (4 alleles scored), it would be =1 (both cases unambiguous). The case with 2 alleles, say A and B, is not unambiguous because the genotype could be AAAB or AABB or ABBB. In spite of this, it should be possible to calculate a measure with some assumptions (in every case, an individual possessing 4 alleles is more diverse than another possessing 3 and so on: 3>2 and 2>1)...

I could find papers coping with this problem when calculating similarity between individuals (Kosman & Leonard 2005 in Mol. Ecol.). I couldn't find studies dealing with the equivalent of heterozygosity (at the individual level) for this case (well there must be some).

To the previous answer: I think, the fact that there is wide-range pollen flow makes it even more difficult to split the sampled individuals (scattered through a very large landscape) into populations...

All the best,

Charalambos

Yes, what you are describing is what I had in mind. Generating some scoring system between say 0-1 would be ideal. From there you would have to decide on how to score things at the individual level. How many markers have you genotyped? Is there much chance of any of them being in linkage? If the answer is that you have genotyped many, and that they are unlikely to be in linkage, then you could simply calculate a mean score for an individual if you are after a crude solution.

Dear, Charalambos,

I advise you to consider individuals as diploid and processed the data as such. You said it is tetraploid, in case it is a autoploidie so for a population of 20 individuals representing 40 you on maximizing the number of heterozygotes course if the population is panmictic because in this case there frq (AA) = p², frq (AB) = 2pq and frq (BB) = q². In case it is a alloploïdie (genome of two different species) so for a population of 20 individuals representing 40 you, in this case, if you have 4 alleles (this is rare because the genes in the two genomes may have a different) you take them in pairs by size, the closest one to the other belongs to the same genome (so individual in your matrix).

Maybe you could also treat genotype as discrete category (AAAA, AABA etc.) and run simple PCA, color code for the condition then. Should provide nice visualization (would also work for the 0-1 scale)

prcomp in R

Meanwhile, I found this paper, which gives a good idea: http://onlinelibrary.wiley.com/doi/10.1111/1755-0998.12431/full

This is also relevant...

http://openwetware.org/images/6/69/LVC_PHR242_110228.pdf 

This paper is very helpful

Article The effects of inheritance in tetraploids on genetic diversi...