it is not very clear to me what exactly you are asking. By "genotypic" correlations do you mean genetic correlations estimated from variance components? If this is the case, and some of the relevant variance components (e.g. genetic covariance between x and y, genetic variance for x and/or y) are not estimable, then you wouldn't get any sensible results for your genetic correlation between x and y. I am not familiar with PBtools, and maybe -1 is a way the software has to tell you that this correlation is not estimable or reliable (though it is odd, since -1 is in the range of possible values for genetic correlations, and I really think developers did not make this choice :-)).
If, on the other hand, you are estimating "genotypic correlations" between genotypes at some loci (similar to what you would do to estimate genomic kinship, or linkage disequilibrium, for instance), and the reason why you don't get a result for some traits can be that you have missing values somewhere, for instance. Still, -1 is a possible value.
Finally, I wouldn't report results without being sure of what they mean and how they were obtained. Therefore, check if -1 is a sensible results given your data (e.g. what about the raw "phenotypic" correlations between the traits?).
actually you are right. I have estimate the genotypic correlation manually; the result i have is absolute UN-estimable due to square root of negative value which is not possible. When i make use of SAS, it didn't run it at all but using Pb Tools, the results i have is -1, which is referring to unestimable value. with this cases, i believe that parameter can not be estimated. what can i do? Has there anyone facing this kind of challenges before? any way out
(linear) correlations can be positive or negative: I do not see the reason to take the square root of a (Pearson) correlation (which can be mathematically intractable, since the square root function is not defined for negative numbers).
Again, if these are "raw genotypic" correlations, just don't take the square root and present results as they are (maybe you may want to correct for the allele frequencies, but this is a different story).