It might be helpful if you describe you data set a little more...including the type of distance measures used (your axis labels are vague).

The Mantel test of matrix correspondence is not exactly equivalent to OLS regression or a standard test of correlation. See Legendre et al. (2015) in MEE (and references therein).

There are lots of papers on the Mantel test. Here is some text from our web page (for PASSaGE 2) that might help. It's not focused on population genetic data, but it should help to explain the basic idea.

Best wishes, 

CDA

-----

The Mantel test (Mantel 1967; Mantel and Valand 1970) is an extremely versatile statistical test that has many uses, including spatial analysis. The Mantel test examines the relationship between two square matrices (often distance matrices) X and Y. The values within each matrix (Xij or Yij) represent a relationship between points i and j. The relationship represented by a matrix could be geographic distance, a data distance, an angle, a binary matrix, or almost any other conceivable data. Often one matrix is a binary matrix representing a hypothesis of relationships among the points or some other relationship (e.g., Xijmay equal 1 if points i and j are from the same country and 0 if they are not). By definition, the diagonals of both matrices are always filled with zeros.

The basic Mantel statistic is simply the sum of the products of the corresponding elements of the matrices, where is the double sum over all i and all j where i ≠ j. Because Z is can take on any value depending on the exact nature of X and Y, one usually uses a normalized Mantel coefficient, calculated as the correlation between the pair wise elements of X and Y. Like any product-moment coefficient, it ranges from –1 to 1.

While more or less interpretable as a correlation coefficient, the significance of the normalized Mantel coefficient cannot be found from standard statistical tests because the observations within the matrices are not independent. In general, the significance is tested through a randomization test by randomly permuting the order of the elements within one matrix (rows and columns are permuted in tandem). When the number of points is large (n > 40), it is possible to transform the Mantel statistic into a t statistic. The significance of t is obtained from an asymptotic approximation of the t-test, which is more reliable for larger n. See Fortin and Gurevitch (1993) and Dutilleul et al. (2000) for more information on the Mantel test.

The basic Mantel test allows for the comparison of two matrices. A number of extensions to include additional matrices have been developed (Dow and Cheverud 1985; Hubert 1987; Manly 1968, 1997; Smouse et al. 1986). One extension is to perform a partial Mantel test, where a third (or more) matrix is held constant while the relationship of the first two is determined (Smouse et al. 1986). This test is done by regressing the elements of X and Y onto the additional matrix (multiple regression for more than one additional matrix), and using the residuals from the regressions as the input for the standard Mantel test.

The traditional approach to estimating significance of a partial Mantel test is to permute the rows and columns of the residual matrices (e.g., this was the method used in PASSaGE 1). While computationally quick (relative to other approaches), this method has been found to have poor statistical properties (Legendre 2000). In this version of PASSaGE, one of the original matrices is permuted prior to the regression (or multiple regression), the regression for that matrix is repeated and then the partial Mantel correlation determined. While slower than the original approach, this should lead to more accurate estimates of the significance or partial Mantel statistics. See Oden and Sokal (1992) and Legendre (2000) for more information on partial Mantel tests.

The Mantel permutation tests are traditionally performed by comparing the magnitudes of Z, which is fine for one-tailed tests, but can give misleading results for two-tailed tests. For a two-tailed permutation test, it is important to transform each Z to an r, which may be more computationally intensive, particularly for partial Mantel tests (for simple pairwise tests it should make little difference).

It might be helpful if you describe you data set a little more...including the type of distance measures used (your axis labels are vague).

The Mantel test of matrix correspondence is not exactly equivalent to OLS regression or a standard test of correlation. See Legendre et al. (2015) in MEE (and references therein).

There are lots of papers on the Mantel test. Here is some text from our web page (for PASSaGE 2) that might help. It's not focused on population genetic data, but it should help to explain the basic idea.

Best wishes, 

CDA

-----

The Mantel test (Mantel 1967; Mantel and Valand 1970) is an extremely versatile statistical test that has many uses, including spatial analysis. The Mantel test examines the relationship between two square matrices (often distance matrices) X and Y. The values within each matrix (Xij or Yij) represent a relationship between points i and j. The relationship represented by a matrix could be geographic distance, a data distance, an angle, a binary matrix, or almost any other conceivable data. Often one matrix is a binary matrix representing a hypothesis of relationships among the points or some other relationship (e.g., Xijmay equal 1 if points i and j are from the same country and 0 if they are not). By definition, the diagonals of both matrices are always filled with zeros.

The basic Mantel statistic is simply the sum of the products of the corresponding elements of the matrices, where is the double sum over all i and all j where i ≠ j. Because Z is can take on any value depending on the exact nature of X and Y, one usually uses a normalized Mantel coefficient, calculated as the correlation between the pair wise elements of X and Y. Like any product-moment coefficient, it ranges from –1 to 1.

While more or less interpretable as a correlation coefficient, the significance of the normalized Mantel coefficient cannot be found from standard statistical tests because the observations within the matrices are not independent. In general, the significance is tested through a randomization test by randomly permuting the order of the elements within one matrix (rows and columns are permuted in tandem). When the number of points is large (n > 40), it is possible to transform the Mantel statistic into a t statistic. The significance of t is obtained from an asymptotic approximation of the t-test, which is more reliable for larger n. See Fortin and Gurevitch (1993) and Dutilleul et al. (2000) for more information on the Mantel test.

The basic Mantel test allows for the comparison of two matrices. A number of extensions to include additional matrices have been developed (Dow and Cheverud 1985; Hubert 1987; Manly 1968, 1997; Smouse et al. 1986). One extension is to perform a partial Mantel test, where a third (or more) matrix is held constant while the relationship of the first two is determined (Smouse et al. 1986). This test is done by regressing the elements of X and Y onto the additional matrix (multiple regression for more than one additional matrix), and using the residuals from the regressions as the input for the standard Mantel test.

The traditional approach to estimating significance of a partial Mantel test is to permute the rows and columns of the residual matrices (e.g., this was the method used in PASSaGE 1). While computationally quick (relative to other approaches), this method has been found to have poor statistical properties (Legendre 2000). In this version of PASSaGE, one of the original matrices is permuted prior to the regression (or multiple regression), the regression for that matrix is repeated and then the partial Mantel correlation determined. While slower than the original approach, this should lead to more accurate estimates of the significance or partial Mantel statistics. See Oden and Sokal (1992) and Legendre (2000) for more information on partial Mantel tests.

The Mantel permutation tests are traditionally performed by comparing the magnitudes of Z, which is fine for one-tailed tests, but can give misleading results for two-tailed tests. For a two-tailed permutation test, it is important to transform each Z to an r, which may be more computationally intensive, particularly for partial Mantel tests (for simple pairwise tests it should make little difference).

Simply by seeing your plot it reveals that with the increase of geographic distance the genetic distance of your sample is increasing very significantly. It may happen if the geography is such as it hinders the pollen movement in case of cross pollinated crops/species or the way  to stop random migration of population individuals from one area to other. FOr example if there is a large mountain and the samples are collected from both opposite side of that or a wider river etc. then the chance of free pollination or migration of samples from one side to other is rare and the species acquires mutation with a long lasting time frame and those are being inherited locally for the same species. The process of speciation will come into effect for such situations.

Besides other problems (see refs below), you have to be careful when using Mantel test and 'only' having two regions sampled. If you have differentiation between your two regions (and no isolation by distance within region), you will most likely find a significant P-value in your IBD (isolation by distance). Though it tells you very little by itself except that your regions are significantly differentiated. The latter would be best tested with a simple test for Fst or other index of genetic differentiation. What could be interesting is to run IBD within each of your 2 regions.

General interpretation of IBD:

Hutchison DW, Templeton AR (1999) Correlation of pairwise genetic and geographic distance measures: inferring the relative influences of gene flow and drift on the distribution of genetic variability. Evolution, 53, 1898-1914.

Issues with using Mantel tests:

Diniz-Filho JA, Soares TN, Lima JS, Dobrovolski R, Landeiro VL, de Campos Telles MP, Rangel TF, Bini LM (2013) Mantel test in population genetics. Genetics and Molecular Biology, 36, 475-485.

Guillot G, Rousset F (2013) Dismantling the Mantel tests. Methods in Ecology and Evolution, 4, 336-344.

Legendre P, Fortin M-J, Borcard D, Peres-Neto P (2015) Should the Mantel test be used in spatial analysis? Methods in Ecology and Evolution, 6, 1239-1247.

Many Thanks Dr. Sebastien Puechmaille

In fact I have got Fst/Gst value of 0.31 which shows the existence of significant differentiation among populations which was further supported by AMOVA test. 

I would pay close attention to what Sebastian has suggested. You cannot do a Mantel test with n=3....which explains why your scatter plot looks fishy. You would need more regular sampling.