There are quite a few approaches. E.g. you can try a cross-validation approach or partition your data into testing and training datasets.

There are quite a few approaches. E.g. you can try a cross-validation approach or partition your data into testing and training datasets.

Cross-validating results is always a very good thing to do, whenever possible. However, association studies are not predictions, and a standard k-fold cross-validation is less straightforward to apply. 

In such cases, I would suggest alternative resampling strategies like bootstrapping or jack-knife: this will allow you to replicate the analysis and isolate true signals from background noise (i.e. spurious associations)

Surely you don't expect me to run bootstrapping on a genome-wide scale... I can run GWAS first and than run permutations/bootstrapping for the interesting loci. Is that a good idea?

The total sample size 1,439. So I don't know about cross-validation (with a training and testing sample set), because than groups (of phenotypes etc) will become small and thus power (which is already on the [very]-low) will be very-very low in the training sample set. What do you think?

I made a general comment on resampling strategies for inferential problems. The computational feasibility is of course to be evaluated for each case. This naturally depends on the number of bootstrap (or jack-knife) replicates you intend to perform. Permutations are another resampling technique that would allow you to build a robust threshold for the identification of true signals. Again, the number of replicates is important and stems from a compromise between size of the problem and validity of results.

As for cross-validating results, you surely would incur in low power problems, and I also think the validation part risks to be rather weak: what would you be using the test set for?

Is your GWAS on complex, i.e. polygenic or mono-/oligogenic phenotypes, in humans or animals or plants?

Are you interested in single marker results per se and nothing else?

In addition to Sophie's suggestions the co-occurring association of other SNPs in high LD would at least provide a hint against genotyping errors.

Moreover for complex (polygenic) phenotypes you could try multimarker methods like pathway/gene set based analysis or polygenic score based analysis using CV.

Lab techniques could mean e.g. create and test a (conditional) knock-out animal (rodents, drosophila, ...) model for the respective genes/phenotypes (time consuming) or do research in corresponding cell cultures.

Many thanks for these useful answers. 

@Josef Frank: It is a complex, presumably polygenic trait GWAS in humans: several markers of atherosclerosis. I'm interested in single-marker results. Yes, I do check the imputation quality, HWE, allele frequency of the top variant and those close by - so as to make sure it's not a lone wolf although, that could still be a valid result. I'm actually also experimenting with gene-based association analyses (VEGAS) and I was thinking of using these results in pathway analysis. Indeed, I'm also running polygenic models using our GWAS results as training dataset, and a GWAS dataset of a closely related phenotype to validate.

@Sophie Limou: I shall check out the permutation testing in PLINK, but I'm actually running SNPTEST analyses as our dataset is imputed using IMPUTE2... Don't think there is a permutation testing method in that package... Or is there? In terms of functional in silico analyses, I do lookup whether our top-hits (or their proxies) have any other functional role (eQTL, GWAS hit) using for instance HaploReg, eQTL datasets, etc.

Thanks!

The bottom line is you probably need to replicate in an independent cohort unless there's hard biological evidence supporting a particular locus. Gene-based association tests and "pathway" analyses may help. However, the "right" gene sets are often difficult to define. Some, (KEGG) are better curated than others (e.g., gene ontology, protein-protein interaction datasets, etc.). However, e.g., KEGG tends to be less comprehensive because it indexes canonical pathways. Also, different gene set analyses will work better depending on the underlying genetic model. 

Cross-validation methods aren't designed to replicate results. They're mostly used to guard against overfitting, and to minimize the bias in parameter estimates. Bootstrapping is often used when the distribution of test statistic is complex or unknown. Since GWAS generally use methods (regression models) whose characteristics are well-known, bootstrapping seems of little value. Permutation testing could help in determining the empirical null distribution (and the empirical genomewide p-value) , but probably wouldn't yield a big difference with the standard (arbitrary) threshold for genomewide significance. Incidentally, plink's premutation routine uses an adaptive procedure which limits computational burden. Plink2.0 is extremely fast, and this should be feasible for a small GWAS of 1,400.

In my opinion, in-silico functional predictions are practically useless, especially if the signals are intergenic--Polyphen and SIFT predict for exonic variants. Besides, complex traits are not necessarily expected to have been subject to purifying selection. eQTLs may or may not be helpful. They have been well characterized in white blood cells and skin, but maybe not in your tissue of interest.