Outgroups often don't get the attention they merit. To my students I recommend two things. Use multiple outgroups, to test the monophyly of the ingroup (you assume it with the use of a single outgroup), and include among the outgroups all taxa thought to be related to the ingroup, i.e. exemplars from its sister group, or taxa thought to represent putative sister groups. Dense sampling of relevant outgroups will avoid assumptions of ingroup monophyly and improve the likelihood of your ingroup network being rooted at an appropriate branch

Outgroups often don't get the attention they merit. To my students I recommend two things. Use multiple outgroups, to test the monophyly of the ingroup (you assume it with the use of a single outgroup), and include among the outgroups all taxa thought to be related to the ingroup, i.e. exemplars from its sister group, or taxa thought to represent putative sister groups. Dense sampling of relevant outgroups will avoid assumptions of ingroup monophyly and improve the likelihood of your ingroup network being rooted at an appropriate branch

I agree with you on the gist of the approach, but like you said, it is based on certain assumptions which I have yet to see a rigorous test. What I envision is a model that examines aspects of the tree to determine whether the data "fits" the monophyletic assumption; It does not yet exist.

Outgroups are used to identify the position of the root on a phylogenetic tree. So in my view, the best available outgroup is the one that has diverged less from the root. A simple way of assessing this is by looking at the genetic distance separating each possible outgroup from your ingroup. Often, when assessing a set of outgroups, I find that the outgroup closest to the ingroup varies among loci. Note that I also definitely suggest the inclusion of multiple outgroups in a phylogenetic study, but looking at genetic distances can help, especially if different outgroups give conflicting results.

I've swapped various outgroups using the distance as a guide, however, the topology does not always remain fixed. What is bothersome is that it boils down to a subjective process of what may either "look good" and confirm pre-existing bias of how the tree should look. It is entirely typical that researchers fail to disclose how the outgroup was chosen, and what others may have been tested and rejected, and why. Looks like a doctoral thesis to make this happen.

I always state in papers why I have used a certain outgroup(s).

I usually use multiple outgroups to test the monophyly of the ingroup and include exemplars from its sister group, or taxa thought to represent putative sister groups.

You can also root the tree using the most basal of the sister group, assuming there has been a phylogenetic analysis to determine this.

Normally, outgroup selection relies on pre-existing knowledge. If it is not known whether an outgroup is *really* an outgroup then obviously one needs to be very careful. You will get hints that your outgroup selection is consistent with being a true outgroup, such as agreement with midpoint rooting, but (just as with every other aspect of phylogenetics) one should always be mindful of the assumptions being made when interpreting the results.

Obviously, it also depends how clear the signal in the data is - if the outgroup/midpoint root position is not close to the subsequent ingroup divergence, it is probably not a major concern. If, however, a small change to the outgroup and/or evolutionary/phylogenetic model used might change the ancestral point of the ingroup, a lot more care must be taken. Depending on the data, it might not be possible to confidently root the ingroup, however carefully the outgroup is chosen. If different outgroups give different results, I am not sure that I would trust any of them - the question becomes whether the specific conclusions being drawn are reliant on the outgroup choice, rather than the more general question of whether the root is "right". (It might not matter.)

I would also agree with the suggestion to use multiple outgroups when the signal is not clean, as imbalanced trees with one very long branch can sometimes behave badly. A balanced tree, with multiple sequences each side of the root, is more likely to behave consistently in my experience.