I have some understanding of the Shannon index, primarily as it is (mis)applied in ecological work, but none concerning the system you are considering applying it to. From that perspective:

The Shannon index was developed as a measure of the information content of codes and only subsequently adopted as an index of species diversity. It can be (and has been) used with many kinds of data and I see no reason why it should not be applied to yours.

I would caution, however, that Shannon's equation can only correctly be applied to an infinite sample. The common practice of applying it to finite ones is wrong, though the magnitude of the resulting errors might not matter to you. There is a parallel index (H) that can correctly be applied to finite samples, Shannon's H' then being found as the asymptotic value of H as sample size increases, though that is very rarely done.

Next: "Exponential Shannon" (the exponent of H' -- which is also the first-order Hill Number) is a more meaningful measure than H' itself. A data set with double the diversity of some other data set will have twice the value of Exponential Shannon, whereas that doubling is not seen with H'.

However ...

I have to ask why you would bother with all that work. Quantitative ecologists stopped using the Shannon Index (or other indices of species diversity) half a century ago, when multivariate statistical analyses became generally available. H' (like all other indices of species diversity) treats every species (or, in your application, every transposon) as interchangeable, which they are clearly not. In contrast, a Bray-Curtis Similarity Matrix treats each as unique and different. You can then analyze such a Matrix in many ways: clustering, MDS, ANOSIM and so on. Perhaps you have reason to consider diversity for itself (as Shannon did when considering codes). If not, application of any diversity index involves throwing away much of the information content of your data and that is rarely a sensible thing to do.

If you do have reason to focus on diversity (for itself, rather than as an over-simplified summary of data), then you should probably look at other orders of Hill Numbers, not just Exponential Shannon.

I hope that is more helpful than confusing!

Trevor Kenchington

Thank you for your response Dr. Trevor John Kenchington. To clarify on looking at the diversity: Since my major purpose is to see if there is any variation in the transposon density/per gene between two conditions and based on my understanding, Bray Curtis gives the compositional dissimilarity between two different sites where it considers species in common between both sites along with total species. However, unlike ecological data, transposon data doesn't tell us if there is any common transposon unless you look at the position of transposon inserts. In that case, would Bray Curtis be appropriate index to move forward with?. Also, regarding looking at the statistical significance of those index values, do matrices like clustering, MDS, ANOSIM as mentioned above give certain significance values (like p values) as an identifier?

Since I am new to this, I am trying to figure out if whatever I am planning to do is a correct approach. I would again be grateful if you could clarify this.

With species-abundance data, the diversity indices (such as Shannon) treat the species from two sites as AA, AB, AC etc from the firsts site and BA, BB, BC etc from the second. Those indices ignore any connections, such as AA being the same species as BC. In contrast, the Bray-Curtis metric looks at particular, named species. The abundance of Homo sapiens at one site is compared to the abundance of H.sapiens at other sites, never interchanged with the abundance of Canis lupus, for example. I do not know enough about transposons to be sure but I suspect that you could only use Bray-Curtis if your data lists them by insert position.

But can you use even the diversity indices without that information? In species-abundance work, you need to know how many individuals of species AA are in the sample, how many of AB etc., so you need species identifications for each individual. The analytical approach then throws away the information on which species name corresponds to "AA" but that is only discarded after the specimens have been identified to species. I suspect that you would need to list your transposons by their positions in order to calculate a Shannon value and, once you have those data, it is just as easy to prepare a Bray-Curtis matrix.

Once you do have such a matrix:

Clustering leads to cluster diagrams. They are not sued in formal hypothesis testing and there is no associated "significance". Likewise, MDS ordinates data. It does not compare groups nor lead to probabilities other being different.

SIMPER and ANOSIM (as a pair) are used in comparing and contrasting groups. They do generate probability values. However, they were designed for analyzing ecological data. You should think carefully before applying them to your data. The nature of the uncertainties (the "error term") in your data may be quite different to what SIMPER and ANOSIM were intended for!

Trevor

Thank you Dr. Trevor John Kenchington. I really appreciate your help. I will look into suggested index and matrix and move forward with the appropriate one.