QTL will show you all genomic regions that affect the genotype. In general it will give you genes associated with the trait, but won't point to specific genomic loci i.e. SNPs.

GWAS is when you want to identify specific loci (SNPs) that particularly elevate the risk of the disease.

Studies that use both techniques generally start off with QTL to identify the genes that affect the phenotype, then sequence the genes in detail to identify SNPs that elevate risk.

A QTL is a genomic region with unknown effect and position that brings about variation in a phenotypic trait. A molecular marker with known position in the genome serves as a signpost to follow the QTL provided it is linked with the QTL. QTL mapping is a candidate  gene approach that involves exploiting this relationship to first detect it and then estimate its effect and position. However its ability to do so is limited in that it maps at a coarser level of the order of 10 to 20 cM of map distance which might contain more that one gene, thus precluding the possibility of cloning the gene responsible for the trait. Finer tune mapping is thus required for reducing the size of the region by various methods. GWAS, on the other hand, is a genome wide approach that involves considering hundreds and thousands of markers like SNPs to detect their association with the trait, either one at a time or considering them together by simple or sparse regression methods respectively. It is expected that in future due to reduced cost of sequencing, whole genome sequences (WGS) could be utilized to replace GWAS.    

QTL mapping give low resolution but give you high statistical power for detecting a QTL. The disadvantage is that you are limited to the genetic diversity present into the parents of your segregating population. You could think of using advanced intercrosses for increasing the resolution.

GWAS offers you very fine resolution, almost to the bp resolution, however, the power for detecting a QTL will be determined by the frequency of alleles, for instance, you may lose power for detecting rare alleles. Another disadvantage of GWAS is that is sensible to the population structure that may lead to many false positives. There are ways to correct for population structure, however, if your population structure contribute to the variation in your trait correcting for population structure may lead you to many false negatives.

Like you mention there are several methods that try to bridge the advantages of QTL mapping and GWAS. For examples, MAGIC and NAM populations they have a broader genetic diversity, and higher resolutions than bi-parental populations and the frequencies of alleles are quite well represented to reduce the problems of rare alleles. The development of these types of populations is labor and cost intensive

I think the decision between using one or the other depends on several factors:

So the question is what is the goal of your study:

1-To identify the causal gene or mutation (GWAS better than QTL mapping) or just looking for a linked marker (Probably a QTL mapping would do the job).

2- Are you interested in the whole genetic architecture of the trait, that is how many genes and their effect, (GWAS better over QTL mapping) or you are interested in major QTLs for practical purposes (QTL mapping  or GWAS would do the job).

3- Previous knowledge of the genetic architecture and heritability of the trait can help you to make a decision as well.

3.1 IF your trait or traits of interest is  underlined by few genes of large effect then GWAS will be suitable

3.2 If your trait is governed by rare alleles, then QTL mapping would be better or you would need large populations to detect them with GWAS.

Here I would like to suggest a few literature on the subject:

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3750305/

http://www.sciencedirect.com/science/article/pii/S1369526608000162

https://www.ncbi.nlm.nih.gov/pubmed/23462502

The main thing in my mind is the population structure specifically the LD level in your population.

If you have high LD (like in F2 mapping population) you’ll be doing QTL mapping regardless of the density of your markers or the analysis methods you use. The high LD won’t give you enough resolution to fine map the causative variation. The lower the LD in your population, the higher your chance to get closer to a specific gene assuming you have the needed density of SNPs to cover the genome (or the region of interest). I hope this helps.

Dear Aaron- Jay

The Genome-wide association studies (GWAS) is used as a powerful tool for investigating the genetic architecture of human disease. Please, read the attached file, I hope will be useful for you.

Good Luck

Dear Dr Aaron-Jay Alcala ,

Greetings and thank you for your question. Both methods could detect QTLs but for bi-parent you must have a wide parent in the traits under investigation and need time to develope these mapping population and this include F2:3, BC, DHLs or RILs mapping population. However for association mapping you must have a diverse collection from genotypes 100-200 genotypes.

Also, both need phenotyping and genotyping data

with best regards

Prof Dr Khaled F M Salem

The answers were good. Also, you can refer to the ANIMAL BREEDING AND GENETICS GROUP at the University of Georgia, Dr. Ignacy Misztal:

http://nce.ads.uga.edu/~ignacy/

Great article about GWAS:

Article A guide to genome-wide association analysis and post-analyti...

I'm interested to QTL mapping and GWAS.