Hi everyone,
I have some issues to determine how to pick correctly the right distance calculation method in R.
I have two datasets : one containing habitat variables (worldclim data, so including negative values) and another one containing values for several phenotypic traits. For each dataset I would like to create a distance matrix in order to perform some Mantel tests between the two matrices (phenotype vs habitat). However, there are so many distance methods that I am a bit lost on which one should be picked. In addition I have some zeros in the habitat distance matrix (replicates which have the same abiotic data) which might restrict the use of some methods.
For phenotype information I created a distance matrix in R using several methods : bray-Curtis, Euclidean, Manhattan, Canberra, Minkowski, maximum --> it doesn't change the conclusion of the Mantel test whatever matrix I use. However, for the habitat matrix results vary. Since there are negative values I guess it is an issue for the Bray-Curtis when not transforming the data (R gives me a warning about misleading calculated values). From what I read, Euclidean distance seems to be used quite a lot for abiotic data. Is there any other method which would be more appropriate ?
If anyone could help me I would be very grateful !
Thanks,
Lola
Cara Lola,
if you have sparse matrix (many zeroes) the best choice is to use Jaccard index as metrics ( https://en.wikipedia.org/wiki/Jaccard_index ). Do not think of biotic/abiotic, Euclidean is perfect in any occasion in which you have a proper Euclidean space, i.e. when you have indpendent axes (variables) with continuous numeric values. Minkowski takes into consideration mutual correlation between variables (corresponds to an Euclidean metrics on principal component scores), while Manhattan is Ok for network data (how many edges I must traverse for going from A to B). In any case if you have a mixture of continuous and binary data (yes7no) you can use Euclidean as well (for binary data Euclidean Distance is equal to Hamming distance)...
I do not think there is much experimental or simulation-based investigation in this area. For the phenotype the fact that the analysis is robust to different distances is just fine. For the habitat, from my readings (and it seems a very logical approach) it is important to understand what is the focus of your interest: are you more interested in the absolute magnitude among the two data sets or just the general shape/tendency in fluctuations? Is the habitat data all continuous or are there also categorical data? If the interest is focused on the variation in absolute magnitude, euclidean or Manhattan are the most appropriate. If only interested in the general shape of the fluctuations, Bray-Curtis would probably be more appropriate. But I think the readers of your report/paper should also be informed that the results are sensitive to the distance metric used.
When in presence of very strong structures all the metrics will give very similar results (see attached)
Thank you very much to both of you ! I have only continuous data and I am interested in the general trend. I will try the Jaccard distance and try to transform my data to not have the issue of negative values with the Bray-Curtis distance. Thanks !
Hi @Lola Toomey, :-)
to the best of my knowledge, distance methods are applied according to factors such as dimensionality of the data, or, if the data represents a convex set, or not (image, if data is "convoluted" e.g., has many folds, the distance measures are not going to be meaningful if you "skip" fold, if, you "visualize" your data in 3D).
Also, the type of the probability distribution of data has a big impact on your selection of the type of "distance" measures that you want to use for classification (I assume you want to classify part of the data).
When straightforward distance measure doesn't work, one often uses the learning methods such as neural nets or Markov models.
Good luck with that. See also my further insights on the selection of distance methods and type of probability distribution.
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Statistical Software
- R