Dear Lucio, perhaps I will not be of much help because I am working on a totally different group (vertebrate fossils). Also you might need to add a bit more information about your problem (e.g. software that you use; data/characters you analyze etc). In my case, working with vertebrate fossils, what we you can do in such cases is to enforce some constrains in the analysis, i.e. forcing (in an experimental way) the monophyly of two taxa, and then re-run the analysis with the constrains enforced (e.g. by using TNT software). Then you can see:

a) the number of additional steps needed to make this group monophyletic. If for example only few more characters are needed, then perhaps by adding more information in the analysis can recover this relationship. If there is a significant additional number of steps needed, then maybe indeed it is not "easy" to accept that the constrained taxa could form a monophyletic group.

b) more importantly, you can see the number of characters (and which characters) that support (or not) this grouping. This might reveal information on conflicting characters, missing information etc.

Hopefully, the above reasoning on totally different field could help you. Above all, I feel that you should not just look the topology (i.e. which groups are recovered together), but why these two families of yours are not grouping together. This might reveal possible changes and additions to be done in your matrix. Also try to see the support of this grouping that is "strange" for you. Hypothetically, a low support or conflicting evidence might give you confidence to search about possible other groupings. On the other hand, strong support of this "strange" clade is difficult to be ignored.

Hello Lucio,

The best way to pick an out group is to hit the literature and see what other researchers working with these enzymes have used in their analysis.

You can also BLAST your sequences and look for similar sequences on NCBI that can be used as out groups. Make sure to look up the literature behind any sequences you download, I have seen erroneous sequences that are obvious once you look at the literature that can skew your results.  

http://blast.ncbi.nlm.nih.gov/Blast.cgi

Thanks a lot!. I have some information from literature saying which are those enzyme families taht can be used as an out group. I just thought that maybe there's a way to do that totally blind. 

Best!!

Lucio

Choosing an outgroup is not absolutely required, but its useful when interpreting and comparing several trees (otherwise you can do it unrooted but this is when you just want to see similarities and not understand their evolution). Choosing an outgroup is not easy, it should not belong to the sample under study (the ingroup), requirying to have a good knowledge of the global phylogeny. Outgroups should not be too far away in evolution (otherwise the relationships with the ingroup might get lost), and not too close (as it could fall within the ingroup). I would recommend you to try looking for a common domain of these 3 families you are comparing (a quick CDD search would do fine). Then I would suggest you to try identify other distantly-related  proteins with the same domain but outside the 3 families, include one of these in your analysis as an outgroup. good luck! :) 

Thanks!!!