John, 

 since I was curious  how well the above mentioned effects of reduction in variance and reduction in DF would cancel each other out, I did a little simulation in R:

P1=P2=NULL # vectors to store P-values from t-tests

for (i in 1:10000){  # we do this 10000 times with random values

s1=rnorm(30,10,2) # sample1: 30 individuals, mean of 10, SD of 2

s2=rnorm(30,12,2) # sample2: 30 individuals, mean of 12, SD of 2

tt=t.test(s1,s2) # t-test of sample1 vs sample 2 (n=30 each)

P1=c(P1,tt$p.value) # store P-value

pool=rep(c(1:10),3) # now we 'pool' data ..

s1p=tapply(s1,pool,mean) # by computing means of three animals

s2p=tapply(s2,pool,mean)

tt=t.test(s1p,s2p) # t-test of pooled samples (n=10 each)

P2=c(P2,tt$p.value) # store P-value

}

> mean(P1)

[1] 0.007467446

> mean(P2)

[1] 0.01023976

So, it turns out the P-values before and after pooling are indeed very similar. If anything, the punishment for reducing the sample size is greater than the benefit of having lower variance. Thus, the small error caused by pooling at least seems conservative.

Cheers,

Thomas

Hi John,

As long as the individuals you pool are independent and your sample size in general is not small (i.e. larger than 3 per group) you will be able to analyse these data with confidence. You will in my view not increase the likelihood of detecting between group differences, because your estimate of the single datapoint is just clouded by some extra error variance that should be independent of your treatment variable. This increased error variance is also averaged out because of the pooled sample analysis. So I do not think it is a problem per se. 

Am I missing your point maybe? Pop me an email otherwise. I can simulate some data to show you what I mean.

Cheers,

Mirre

Hi Mirre

Thanks for the swift response

The way I was looking at it was like this. If you have 30 measurements as a random sample of a population then you would have an unbiased estimate of the population variation. However if you grouped these 30 into 10 groups of 3, and calculated the mean of each group then calculated a variance for these means the variance of the grouped samples would be lower....(actually equal to original variance divided by root n). The grouping is like the tissue pooling process. so if you then were to compare the ten grouped samples to ten grouped samples from another population without accounting for the deflation of the variance you would be more likely to get a difference than if you just selected 10 individuals at random. but since the deflation of variance is a simple function of the sample sizes in the grouped samples then it would be theoretically possible to reconstruct the original individual variance.  Do you know if anyone actually done that? plus if they have how many pooled samples do you need to analyse to get a reasonable estimate for the original individual variance? 

cheers

john

John, I think your example is better suited to genetically heterogeneous populations.

As I understand it the high genetic homogeneity within any given mouse strain should provide a little more confidence in pooling samples without loosing much in the way of variation between mice of the same strain. In my previous work, I did a some mouse studies where I did not have sufficient material for microarray unless I pooled them. I pooled mice from the same litter, but can't imagine that it would make that much of a difference if I had pooled from different litters.

Personally, I think that with respect to mouse work analyses using either single samples or pooled samples will be dominated by analytical (technical?) variance. In other words differences between two mice of the same strain is small, while the technical variance (or error) introduced by your method of choice is likely to be large at least compared to the variance between individuals (pooling them will make little difference, but I have not tested this).

In most instances (and ideally), the phenomenon being studied manifests strongly under experimental manipulation so that any variance (technical or individual) represent only a fraction of your total variance, with most being associated with your treatment/manipulation.

Even if someone were to manage to identify statistically significant differences between mice of the same strain (within the same group) can't say that it would be of much biological significance as it would be difficult to distinguish between true individual variance and technical variance - this being the case only because you are working with a genetically homogenous model. Now, if you work involves human samples, then I think individual variance has to be taken into account.

I am now working with human samples and fortunately I have had enough material, but many studies have been published where pooling was unavoidable (I do not like that they did, but I understand their predicament). I avoid pooling human samples as much as possible. Because of the genetic heterogeneity between humans, I am likely to miss a lot with respect to individual variation when pooling.

The answer I almost always get with respect to the total number of n per group is as many as possible, but I have since learned that this question is better answered by performing statistical power analysis. Doing so will give you a statistically determined number to use for your n per group. Most people do not have the luxury of this type of statistical analysis (and limit their studies to n=3, some to even less) since it often will output a much larger number than can be practically obtained, but I suppose that is where experience and good scientific judgment come into play.

I look forward to reading about your findings.

Best.

Thanks for the message, and your interesting thoughts on this issue.

Do you have any data with respect to gene expression in inbred mice being less individually variable than in outbreds? I know it seems sensible that it should be this way but in fact for many other traits it is certainly not the case.

For example in my lab we have looked at variation in the the response to high fat diet feeding in both C57BL/6 (the classic inbred mouse strain) (Zhang, L.N., Morgan, D., Clapham, J.C. and Speakman, J.R. (2012)  Factors predicting non-genetic variability in body weight gain induced by a high fat diet in  inbred C57BL/6J mice. Obesity 20: 1179-88) and more recently in MF1 mice (an outbred strain) (Vaanholt, L.M., Sinclair, R., Mitchell, S.E and Speakman, J.R. (2015) Factors influencing individual variability in high fat diet induced weight gain in out-bred MF1 mice. Physiology and Behavior (online)) and the variation in the response is very similar (enormously high in both relative to any analytical error). Another trait we have measured in the same two species is litter size, and that is similarly all over the map - and here the analytical error is effectively zero. (for C57BL/6 the paper is Johnston, S., Grune, T., Bell, L., Murray, S., Souter, D., Erwin, S., Yearsley, J., Gordon, I., Illius, A., Kyriazakis, I., Speakman, J.R. (2006) Having it all - historical energy intakes do not generate the anticipated trade-offs in fecundity. Proceedings of the Royal Society of London B 273: 1369-1374 and for MF1 mice  Johnson, M.S., Thomson, S.C. and Speakman, J.R. (2001) Limits to sustained energy intake I. Lactation in the laboratory mouse Mus musculus. Journal of Experimental Biology 204: 1925-1935.)

The point is I don't think that we can assume that GE in inbred mice will be minimally variable between individuals simply on the basis that they are inbred - but if you have (or know of) data that shows that, I'd be very interested if you could direct me towards it.

best wishes 

John

Hi John and Ricardo,

First of all I think Ricardo is worried about Type II errors, yes you could asses that problem using prospective power-analysis, if you have a good estimate of the expected error variance (either between individual variation or measurement error).

As I understood you John, you are worried about type I errors because you average across individuals and this reduces the variance in your sample, through regression to the mean of the average mouse in your population. That the variance reduces is true (slightly depending on the averages and the amount of replicates you do and how many individual mice in each sample), but you do this at both sides of the comparison so there is no problem. You do not alter the means and the variance you compare it over is the same. You might run into some problems if the amount of mice in each sample vary and especially if you do not know how many mice are in each sample, but as I understand you can fully control this. I wrote some R code to explain this to you and sent it to you via email. 

So do not worry about the type I error for the means. If you would be interested in what the underlying original variance was amongst mice you could calculate that, if you know your measurement error. Probably not really your interest, you would only be able to say mice have X sd in X trait.. usually not something we are interested in. But if you want that let me know and I can probably help you. 

Sound like an interesting experiment, best of luck.

Cheers,

Mirre

Hi John,

I agree with Mirre. To use your example: By pooling samples from 3 mice you reduce the variance (which makes finding group differences more likely), but at the same time you reduce the sample size form 30 to 10,  and thus reduce the degrees of freedom (which makes finding group differences less likely). These two effects should pretty much cancel each other out. So I don't think there is a real problem, at least not statistically (but of course, you will need more experimental animals if you are forced to pool samples)

Cheers,

Thomas

John, 

 since I was curious  how well the above mentioned effects of reduction in variance and reduction in DF would cancel each other out, I did a little simulation in R:

P1=P2=NULL # vectors to store P-values from t-tests

for (i in 1:10000){  # we do this 10000 times with random values

s1=rnorm(30,10,2) # sample1: 30 individuals, mean of 10, SD of 2

s2=rnorm(30,12,2) # sample2: 30 individuals, mean of 12, SD of 2

tt=t.test(s1,s2) # t-test of sample1 vs sample 2 (n=30 each)

P1=c(P1,tt$p.value) # store P-value

pool=rep(c(1:10),3) # now we 'pool' data ..

s1p=tapply(s1,pool,mean) # by computing means of three animals

s2p=tapply(s2,pool,mean)

tt=t.test(s1p,s2p) # t-test of pooled samples (n=10 each)

P2=c(P2,tt$p.value) # store P-value

}

> mean(P1)

[1] 0.007467446

> mean(P2)

[1] 0.01023976

So, it turns out the P-values before and after pooling are indeed very similar. If anything, the punishment for reducing the sample size is greater than the benefit of having lower variance. Thus, the small error caused by pooling at least seems conservative.

Cheers,

Thomas

Hi Thomas. yes I also simulated the type I error to prove this to John. I'll forward you what I wrote, similar to yours. Anyhow statistically IF your samples are independent that you pool in the respective treatment comparisons there is no problem.

John, your previous studies focus on complex traits (e.g., weight) and though you have identified a significant correlation with baseline fat mass, I do not think you would find nearly as large a difference if any in gene expression between the mice that gained more versus the mice that gained less. I predict that that the simple feeding of a high-fat diet will have a greater impact on gene expression compared with low-fat diet, than between individuals who show differential weight gain from the same diet (where I would predict few if any statistically significant differences).

The way I understand it, for complex traits the dose differences in each gene between mice is likely to be small (i.e., not statistically significant - even if biologically significant), but the effect on phenotype, by the combination of expressed genes, can be large. I think this explains the large variation in mouse weight that you observed.

Your specific experimental design and the traits you are studying benefit from low technical (analytical) variation. You will find that the same is not true for transcriptomic (and perhaps any -omics) studies. In terms of the technology platform, microarrays are worse than RNA-seq (I have used both) with respect to technical variation. Some investigators have even gone as far as performing technical replicates expecting to minimize this variance (I suppose this is reasonable). With transcriptomics studies you also have the issue of Multiple hypothesis testing problem (Type I error), which is partly corrected with a false discovery rate (FDR)-adjusted p-value (Mirre and Thomas are probably better suited to speak at length about this if you are interested).

My experience is that unless you are able to reliably and consistently excise the same region of brain tissue, the greatest component of your variance will be technical (mostly due to operator "error" in excision, unless a precise method is used - e.g., laser capture) and sample preparation. Technical error will be high in this instance, so I would make certain that the effect you are studying is sufficiently large to overcome this. 

If you can be sufficiently confident that you have minimized technical variance, and the total number genes showing statistical significance is small, you may want to consider using other analytical tools to tease out biologically relevant changes. I have provided a link to one such tool.

Thus, although correlating non-genetic factors to some complex trait is fairly straightforward, I can't say the same when attempting to correlate with gene expression derived from transcriptomics technology.

Rambling a bit now, but I hope I have provided you sufficient useful information from the perspective of biology, at least as much as Mirre and Thomas have done from the perspective of statistics.

I am now following you and look forward to reading about how you ultimately have decided to tackle this problem.

Best.

http://labs.genetics.ucla.edu/horvath/CoexpressionNetwork/Rpackages/WGCNA/

Thomas/Mirre

Apologies for the delayed reply (travelling).

These calculations are really very interesting and I think that you have revealed an extremely important point. The first is the contrasting effects of reducing the variance and reducing the sample size which just about completely cancel each other out. This may not seem to be a big discovery but actually it is tremendously important. If you take a trait like body weight then actually it isn’t so important. that’s because in both the raw data and the pooling scenario you take 30 measurements. The question is just how you manipulate the data and the answer comes back that it pretty much doesn’t matter.

However, if we go back to my original issue, the question there was about pooling tissues because there is actually insufficient tissue to run individual analyses. In this scenario the pooling is not just an arithmetic exercise. In fact you pool the samples, and then you only make 10 measurements. The fact this comes out as being effectively the same as making the full 30 measurements therefore means that by pooling you have reduced the cost of the analysis by a factor of 3 with minimal loss in the resolution. So basically (if the key aim of the work is to compare groups rather than know the individual values) even if you do have enough tissue to run the individual samples you should always pool the samples because you get effectively the same answer from the pooled analysis as from running the individual samples – but it is substantially cheaper.

The key question now becomes what is the optimal pooling scenario? That is the models you ran were for a scenario of 30 original samples pooled in sets of 3 to give 10 final samples. Does this give the same as say pooling into sets of 6 to get 5 final samples, or pooling sets of 10 samples and doing 3 analyses or ultimately pooling into 2 groups of 15 individuals. If it turned out that the probability in this latter scenario was the same as that when doing the individual samples then the cost of the experiment could be reduced by a factor of 15 fold! I tried to rerun the R script you sent but it gave me some error messages that I couldn't debug - regarding the FOR statement and the }.

I hate to involve you in more work but maybe you could run the models with different pooling scenarios to see what happens to the p values!

Best wishes

John

 Ricardo - thanks for your thoughts and the link.

cheers john

Hi John,

So if measurement error would be 0 this would be true I guess. However this is not the case of course so you lose power to estimate that by pooling and hence if you could you would want to do each sample separately. Also me and Thomas both assumed the underlying data was normally distributed, but once you start pooling across other distributions such as Poisson (gene expression probably is Poisson) things are probably less uniform. Such pooling across a different than normal distribution will not inflate type I error, but will reduce power.  

So the actual decision if you can avoid pooling and choose to pool anyway would be that measurement error is substantially smaller than between-individual variance (which you reduce by pooling) and that the distributions are known, with the most easy option being that it is normal. So that you get the most statistical power out of your expensive method. 

Makes sense? I can change the code I sent you if you want. It already had a line in there for measurement error (but it said let's ignore this for now :D). 

Safe travels!

Mirre

I'm not sure that the parent distribution makes a big impact because of the central value theorum the sampling distribution for the mean is still normal even if the underlying population distribution is not.

Yes but depending on the distribution especially if it is very flat, and the normally distributed error this should reduce your power in my mind. Just because by accident you pool two mice with a very low and a very high value. Otherwise you would have been able to compare both measured at the individual level under a Poisson model with hence more power? See what I mean?

Usually what I do is to pool an equal part of my samples together, then after I get the results of the analysis, I confirm the variations of individual mRNA or proteins by qPCR or Western Blot, using the rest of the samples.

Thus, I don't perform statistical analysis on the pooled samples, I do it in qPCR and Western Blot confirmation.

Just some related code I quickly wrote for this averaging across these different distributions. Conclusion averaging across Poisson reduces power, compared that if you would measure with the same sample size a randomly drawn set of these pooled individuals:

#example for John#

#a_error=1 #analysis attributable error, measurement error say..

b_error=1 #biological variation b_etween individuals

meandifference=0.5 #difference between groups

mergeindividual=3 #merge 3 individuals

totalsample=10 #per experimental group

 #Poisson distributed

runs=100000

results=matrix(NA,nrow=runs)

results2=matrix(NA,nrow=runs)

q=1

while(q

Hi all

slightly tangential but just found this paper on pooling RNAseq samples to estimate alelle frequencies which suggests pooling is the way to go because it reduces costs at only a slight loss of information. i suspect that the same may be true for pooling to do transcriptomics, as per previous message

best

john 

Accuracy of allele frequency estimation using pooled RNA-Seq

Mateusz Konczal

Paweł Koteja

M T Stuglik

Jacek Radwan

W Babik

Institute of Environmental Sciences, Jagiellonian University, Gronostajowa 7, 30-387, Kraków, Poland.

Molecular Ecology Resources (Impact Factor: 5.63). 10/2013; 14(2). DOI: 10.1111/1755-0998.12186Source: PubMed

ABSTRACT For non-model organisms, genome-wide information that describes functionally relevant variation may be obtained by RNA-Seq following de novo transcriptome assembly. While sequencing has become relatively inexpensive, the preparation of a large number of sequencing libraries remains prohibitively expensive for population genetic analyses of non-model species. Pooling samples may be then an attractive alternative. To test whether pooled RNA-Seq accurately predicts true allele frequencies, we analyzed the liver transcriptomes of 10 bank voles. Each sample was sequenced both as an individually barcoded library and as a part of a pool. Equal amounts of total RNA from each vole were pooled prior to mRNA selection and library construction. Reads were mapped onto the de novo assembled reference transcriptome. High-quality genotypes for individual voles, determined for 23,682 SNPs, provided information on "true" allele frequencies; allele frequencies estimated from the pool were then compared to these values. "True" frequencies and those estimated from the pool were highly correlated. Mean relative estimation error was 21% and did not depend on expression level. However, we also observed a minor effects of inter-individual variation in gene expression and allele specific gene expression influencing allele frequency estimation accuracy. Moreover we observed strong negative relationship between minor allele frequency and relative estimation error. Our results indicate that pooled RNA-Seq exhibits accuracy comparable to pooled genome resequencing, but variation in expression level between individuals should be assessed and accounted for. This should help in taking account the difference in accuracy between conservatively expressed transcripts and these which are variable in expression level.