I am studying the new species of Modiolus (bivalve: Mutillidae). I am going to construct a phylogenetic tree for it. I am having trouble choosing a suitable outgroup for this genius.
In general, an outgroup should be as close as possible to the ingroup. If there is a risk that it is really part of the ingroup, then other outgroups are needed in addition, to help resolve those relationships. Multiple outgroups give a more reliable picture. So, in your case, if Modiolus is your focus, and the new species could fit anywhere, I would choose single representatives of a few genera in the Mytilidae (e.g. Benthomodiolus, Mytilus, Musculus, Homomya, Lithophaga, Leisosolenus, Fungiacava, Botula, Adula, Aulacomya, Perna etc).
In general, an outgroup should be as close as possible to the ingroup. If there is a risk that it is really part of the ingroup, then other outgroups are needed in addition, to help resolve those relationships. Multiple outgroups give a more reliable picture. So, in your case, if Modiolus is your focus, and the new species could fit anywhere, I would choose single representatives of a few genera in the Mytilidae (e.g. Benthomodiolus, Mytilus, Musculus, Homomya, Lithophaga, Leisosolenus, Fungiacava, Botula, Adula, Aulacomya, Perna etc).
I agree with Graham, that if the monophyly of your ingroup is doubtful, you should add several similar or "closely related" taxa (same tribe or subfamily if you are working with a species).
I would advise against adding taxa that are "too different" from your working group, they may have lots of uninformative characters (autapomorphies).
In my opinion, you can select outgroup from the closest relatives to your ingroup. For example representatives from a few genera. To be more precise, you should select one among the outgroup that having ancestors' characteristics and formed the earliest branching genus to root the tree.