Dear Lucio, perhaps I will not be of much help because I am working on a totally different group (vertebrate fossils). Also you might need to add a bit more information about your problem (e.g. software that you use; data/characters you analyze etc). In my case, working with vertebrate fossils, what we you can do in such cases is to enforce some constrains in the analysis, i.e. forcing (in an experimental way) the monophyly of two taxa, and then re-run the analysis with the constrains enforced (e.g. by using TNT software). Then you can see:

a) the number of additional steps needed to make this group monophyletic. If for example only few more characters are needed, then perhaps by adding more information in the analysis can recover this relationship. If there is a significant additional number of steps needed, then maybe indeed it is not "easy" to accept that the constrained taxa could form a monophyletic group.

b) more importantly, you can see the number of characters (and which characters) that support (or not) this grouping. This might reveal information on conflicting characters, missing information etc.

Hopefully, the above reasoning on totally different field could help you. Above all, I feel that you should not just look the topology (i.e. which groups are recovered together), but why these two families of yours are not grouping together. This might reveal possible changes and additions to be done in your matrix. Also try to see the support of this grouping that is "strange" for you. Hypothetically, a low support or conflicting evidence might give you confidence to search about possible other groupings. On the other hand, strong support of this "strange" clade is difficult to be ignored.

Hello Lucio,

The best way to pick an out group is to hit the literature and see what other researchers working with these enzymes have used in their analysis.

You can also BLAST your sequences and look for similar sequences on NCBI that can be used as out groups. Make sure to look up the literature behind any sequences you download, I have seen erroneous sequences that are obvious once you look at the literature that can skew your results.  

http://blast.ncbi.nlm.nih.gov/Blast.cgi

Thanks a lot!. I have some information from literature saying which are those enzyme families taht can be used as an out group. I just thought that maybe there's a way to do that totally blind. 

Best!!

Lucio

Choosing an outgroup is not absolutely required, but its useful when interpreting and comparing several trees (otherwise you can do it unrooted but this is when you just want to see similarities and not understand their evolution). Choosing an outgroup is not easy, it should not belong to the sample under study (the ingroup), requirying to have a good knowledge of the global phylogeny. Outgroups should not be too far away in evolution (otherwise the relationships with the ingroup might get lost), and not too close (as it could fall within the ingroup). I would recommend you to try looking for a common domain of these 3 families you are comparing (a quick CDD search would do fine). Then I would suggest you to try identify other distantly-related  proteins with the same domain but outside the 3 families, include one of these in your analysis as an outgroup. good luck! :) 

Thanks!!!

I agree heavily with Jose Tomas Matus that using CDD to find families with matching domains is a very good idea. I would just caution that you may need to be careful about what you get from this search. If you pull a divergent protein where this domain is only a small part of the overall sequence you will wind up aligning only that domain and your tree will be a history of said region. This may not be a problem depending upon your circumstances, but it might be a major issue that gives you a false result.

One way of augmenting a CDD search is through a set of BLAST searches. I assume you already have accepted sequences for each of your families, plus your own sequences. Use a number of each from all three groups as seeds for BLAST searches and take a large number of hits (preferably, mostly those you know are not part of your groups). Ideally, these will coincide with your CDD searches and you can include a significant number of them to provide shape to your tree. A large and varied outgroup set can help ameliorate some of the problems of using a single too-distant or too-close taxon. If your new group falls in or sister to the two know groups with no outgroups among or between, then you have some good evidence that your 3rd group is closely related to the knowns.