Sequence logos have been one of the most widely used methods for the visualization of proteins or DNA motifs in the past 20 years. It has been used in many ways mainly to visually inspect the characteristics of predicted/discovered motifs and evaluate motif discovery tools (i.e, empirical study). I found it is very misleading to use sequence logos to compare computational motif prediction tools because it is a very subjective affair to say that a logo for example "resembles", "is close in appearance to", or "like" the known or annotated ones. My own small scale study found that there is a great mismatch between the "appearance" of the sequence logo and the actual performance of a tool. However, I am surprise this visualization method is still a widely accepted method for computational motif discovery tool prediction. My question is: why? What is your opinion of using this evaluation method?
I have used sequence logo for all the reasons you mentioned in your second sentence. I did not know it was being used to assess the performance of computational methods. I think the rationale is that the sequence logo is a representation of the frequencies of amino acid/nucleotides at each position, and so it is a representation of the gold standard. I imagine from the gold standard (set of annotated sequences) you can extract a meaningful sequence logo (indeed a PWM) and then you can compare the PWM you obtain from your method with the one from the gold standard. If your method performs well both PWMs should be similar, in terms of the matrices. However that also implies the logos will look alike.
Personally, I think that as far as the comparison between logos is based on a quantitative measure (e.g. alignment of PWMs and estimation of differences/similitudes) then there should not be a problem. Maybe I am missing some important details here though.
You are right on the statement "If your method performs well both PWMs should be similar, in terms of the matrices". In fact, this is the common assumption made when one use sequence logos for tools performance comparison. However, there are several reasons this assumption is flaw: a) many gold standard (PWM) is not available or partially available and therefore authors have to use only published logo in comparison but not PWM; b) let say a tool reported 100 sites and another tool reported 50sites and both PWMs matched the the gold standard, but we can't really tell which prediction tool is better since both give high matching scores. As a example, consider the sequence motifs predicted by three tools (see link). Which tool should the best, 2nd, and 3rd? if they all matches the annotated one. In fact, Tool1 has the highest precision/recall rate, but most users might choose tool2, or tool2 because they "looks" more similar to annotated one.
https://docs.google.com/file/d/0B0X0-eYUdq_TdWE5RU9EY3VTXzQ/edit?usp=sharing
"A sequence logo is created from a collection of aligned sequences and depicts the consensus sequence" but the "gold standard" or the consensus sequence or sequences may not exist in nature... it is just a tool no more...
You will have to follow up with other analysis tools to understand your output...
All the best
CE0702
I believe sequence logo is a nice and effective way of visualizing TFBS or protein motifs. If i am not wrong all agree on this as far as performance of a tool is concern, some more parameters are used like p-value of predicted sites & "gold standard", if you set a cut-offf for selecting individual sequences which give rise to PWM or/and sequence logo, you can then have more precise estimation of performance of a tool and i hope it is not that difficult. Popular tools like MEME provide this parameter.
About example discussed, tool predicting 100 sites (more inclusive) or 50 sites but highly reliable ones can be compared if we set filtering of sites give rise to PWM or Sequence logo correctly. And sequence logo then has nothing to do with performance of a tool.
more like E-value...not P-value!! and yes, you can derive a P-value from an E-value...
The formula p = 1-e(-E) allows one to derive a P-value from an Evalue. ref. Systems Biomedicine 1:1, 1–12; October/November/December 2012; © 2012 Landes Bioscience
Research paper research paper
*Correspondence to: Chai Erzerer; Email: [email protected]
Submitted: 01/12/12; Revised: 04/16/12; Accepted: 06/07/12
http://dx.doi.org/10.4161/sysbiomed.21734
Unfortunately, many published articles did not report the p- or e-value. I agree that, such score is useful but a recent article [Weirauch et al. (2013) Evaluation of methods for modeling transcription factor sequence specificity, Nature Biotechnology, vol 31 (http://www.nature.com/nbt/journal/v31/n2/full/nbt.2486.html BOX 1] suggested high-information content motif (can be converted to p-value) with "elegant logo appearance" does not necessary represent more accurate result.
on the 100 vs 50 sites, I mean both gives rise to logo that resemblance and matched to the annotated ones. Let say the filtering does not work. Both probably has almost the same p-value or motif score value because their pwm are almost the same. Since most papers did not report the "number of sites predicted", but logos, the "resemblance" of predicted to the "annotated one" can be misleading.
http://www.nature.com/nbt/journal/v31/n2/full/nbt.2486.html
- Badges
- Science method
- Similar topics
- Biological Science
- Genetics
- Genetic Analysis
- genetic techniques
- Sequencing