I found this paper that describes a statistical method that addresses your problem, I think: http://www.cbcb.umd.edu/~niranjan/papers/WhiteNagarajanPopPLOSCompBio09.pdf

The authors propose a method to identify species which are differently abundant in pairs of metagenomic samples, and they provide a particular treatment for cases where data are sparse.

It has been cited around 100 times, so if you trace the citations forward I think you would find some interesting more recent developments.

Thank you very much!

speaking of the DGGE bands analysis, i remember there is a gel analyser goes with the device. But the analyzer can only generate UPGMA clustering analysis based on the band pattern. If the analyzer is not available, you can make an OTU table after sequencing the bands. Then convert it to BIOM format, submitting to QIIME for further analysis.

If my guess is right, you may have sequenced the rRNA amplicons by 454 or MiSeq. To reduce the errors caused by sample size, one of the common methods is to rarefy all samples to uniform size. For instance, sample A has 3000 reads while sample B and sample C have 8000 and 2000 reads, respectively. Before comparison, you need to subsample the three samples using 2000 reads as the depth. Related command can be find in QIIME documents. Good luck

You need to rarefy your OTU table to an even depth comand in QIIME. Or you can use the vegan/phyloseq package in R ( rarefy_even_depth)  to rarefy otu table and do all further calculations. In case of DGGE bands, you can use NTsys software which is very basic, infact any kind of molecular marker (RFLP,RAPD,AFLP) based banding patterns with presence/absence data can be used to generate dissimilarity matrix and cluster generation.