I am working on phylogenetic studies where i have to select some outgroups for tree building. My question is how one can select the best outgroup for phylogenetic studiens
I think all options in the same order would be useful. I suggest the outgroup be the relative species to all other included in the tree further more should be older in the evolutionary time.
I agree with Mustafa! Usually I select (1) a species closely related to the group to be analyzed and (2) a species/species-group of the next superordinate unit closely related to the latter, i.e.(1), and possibly (3) so on!
Basically two groups of taxa: the first one (as has been said) taxa which are closely related to the study group. If you are studying species of one genus, you must include as outgroup species of other related genera. To determinate which are related genera, you can see published phylogenies in your group. Finally, you should include a group of species (1-2) not closely related to root the tree.
Include many taxa just outside your taxon of interest, and use only the most distant one of them (which may itself be a clade out of which you have perhaps sampled several subclades) as the outgroup in your analysis.
The choice of outgroups is a complicated issue as adding or removing some taxa can result in bold changes in the obtained trees and on characters´ optimization.
In principle, there is not such thing as a "best" outgroup. Must use as much outgroups as possible. You can think of it as a standar statistic analysis: data should not be discarded in the absence of strong evidence that justifies it. Also, the more data is included, the better is the character optimization during the analysis.
Really difficult especially if you are using the very best methods based on maximum likelihood models. If the outgroup is widely separated from the in-group, then you can be sure that really is an outgroup and doesn't belong somewhere within the in-groups. But if the chosen outgroup is to far away then the model of sequence evlolution is like to be poor for both the outgroup and the in-group because it is sort of trying to average organisms that probably follow different models of sequence evolution. Ideally you know enough about the biology of the in-group and the outgroup to choose a reasonable model for both..]
One thing I would do is to determine the best model In PAUP4 using the DT option and with and without the outgroup. Check to see if the respective models seem quite different from another (are the parameters radically diffenent?). If so try going with a somewhat more closely related outgroup if you can.