The authors of the following paper make some excellent points about the use of and difference between repeats and replicates:
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3321166/
They are very clear about what to incorporate in figures. However, I've always been aiming to learn whether and how uncertainty in experimental results should be taken account of in statistical comparisons.
Say, an experiment, such as qPCR, compares the normalised expression of a gene under two different conditions. The test and control condition is performed on during the same experiment (e.g. on the same plate). Good practice would dictate that this experiment is repeated at least three times, preferably more, to see if the results are reproducible. Each of these experiments can contain more than one technical replicate (e.g. multiple qPCR reactions). These technical replicates provide a degree of uncertainty about each mean value. How is this uncertainty best incorporated in the statistical comparison between the two conditions across the experimental repeats?
Perhaps in the above example the technical error is very constant. Perhaps in this case the predicted technical uncertainty does not have to be incorporated into final analyses. However, how does one incorporate technical or 'pseudobiological' uncertainty that is more variable?
Another example:
Three experimental repeats are conducted with a cell culture plate that measures the concentration of a particular cytokine between two conditions. For each experimental repeat, each condition is tested with three wells of cells (three pseudobiological replicates, as they are three wells, but are on the same plate, contain the same passage no.etc). To measure the cytokine concentration, an ELISA is performed with two technical replicates per cell culture well.
Each step returns an uncertain value (technical rep and pseudobiological reps). The pseudobiological replicates could be removed, but even among these significant variance could exist due to varying conditions on the culture plate. The reps could provide a better estimate of the culture plate mean.
Again, what would be the best approach to incorporate the uncertainties of technical and perhaps unknown origins in the final comparison between experimental repeats.
Bit of a story. There may be many differing opinions on this, but I would like to hear them.
There is an entire area of statistics called Design of Experiments.One of the many topics that comes up in this area and class(es) deals with how reproducible and robust the response is.
One of the issues is that there is a Day-to-Day variance in all systems. To get more robust, reproducible results, you need to run replicates over several days. Not just one.
When you design an experiment, you should look at the power of your results too. T-tests are the least powerful you can find. If you have a very weak test, you will miss a lot and commit many Type 2 errors, (failure to find a significant difference). This will make your results very unreliable.
When you test more than one factor, you need to use a design from the Design of Experiments section of Statistics. T-tests are about the worst thing you can use for experiments. Using Factorial Designs, Response Surfaces, Split Plots, etc are far better options.
Having spent years studying and working in the chemical analysis industry, I know that many scientists don't know about these methods. It is much to the detriment of scientists that they don't know about it.
If you use Design of Experiments, and control for day-to-day variability and other factors that can affect your results, you will have a smaller variability than t-tests. You will get a standard error for each thing you test for. This gives you the variability for each factor you test. You will also have good power.
I'm not quiet certain, but I would be comfortable with treating the uncertainty just as you would error. Then use the same rules for error propagation through out your data set to determine your final uncertainty.
Also, I vaguely remember doing an ANOVA on large complex data sets to compare variance to determine statistical significance. This might work here.
Fortunately, we have a paper for that.
Have a read through and feel free to ask more questions.
Article Quantitative polymerase chain reaction: A framework for impr...
There is an entire area of statistics called Design of Experiments.One of the many topics that comes up in this area and class(es) deals with how reproducible and robust the response is.
One of the issues is that there is a Day-to-Day variance in all systems. To get more robust, reproducible results, you need to run replicates over several days. Not just one.
When you design an experiment, you should look at the power of your results too. T-tests are the least powerful you can find. If you have a very weak test, you will miss a lot and commit many Type 2 errors, (failure to find a significant difference). This will make your results very unreliable.
When you test more than one factor, you need to use a design from the Design of Experiments section of Statistics. T-tests are about the worst thing you can use for experiments. Using Factorial Designs, Response Surfaces, Split Plots, etc are far better options.
Having spent years studying and working in the chemical analysis industry, I know that many scientists don't know about these methods. It is much to the detriment of scientists that they don't know about it.
If you use Design of Experiments, and control for day-to-day variability and other factors that can affect your results, you will have a smaller variability than t-tests. You will get a standard error for each thing you test for. This gives you the variability for each factor you test. You will also have good power.
I can't comment on any particular method of how to conduct an appropriate mathematical consideration of experimental variance (or "error" or "uncertainty", and different from biological variance) across experiments. I certainly sympathize, and I've asked similar questions regarding the handling of variance across pseudo-reps and techno-reps! However, one technique that helped me track the precision/accuracy change across "real" replicates in my own work was to track the coefficient of variance (CV) within an experimental "read" or plating or observation (CV = stdev relative to the mean, typically expressed as a precent = [stdev(set a)/mean(set a)] x 100%). So if your daily CV is ~20%, and then you note a spike to, say, 60% or so after one particular replication, then at the very least you know that there was rather high intra-replicate variance across your duplicate measurements, and you could try and sort out if it arose from the pseuobio or the techno reps...more of a troubleshooting technique to identify and limit artificial variation as early as possible, which could result from technique, reagents, co-workers, instrument wear, etc.
At the end of the day, there are several types of "uncertainty". Primarily I think that biological variability is what we should be trying to capture, and this is the variability to be used in biological hypothesis-testing. The other source of variability which you describe as uncertainty that which results from experimental measurements, and I would argue that this type of uncertainty should perhaps be handled separately, as it may bear no relation to the underlying biology. While I cannot answer the question of "how" to incorporate these different kinds of variance, I can pose the related question: "Should we be combining variance from all sources, or would it be preferable to attempt to discriminate between (likely) biological from (primarily) technical variation?".
I really agree with Jason's comments and will add my quick capsule version. I take it as a given to always have multiple tech reps of each bio rep, and the tech reps get checked before being compiled into a mean value by analyzing % CV. Typically if there is a method problem with one rep (something as simple as a bubble in a well on a 96-well plate for instance), the CV of the tech reps will go way up and you can identify the bad actor. Obviously 3 tech reps is the bare minimum for % CV to be of any use or else you can't identify the bad actor when two values don't match up. Otherwise decent replicate values tend to have a CV
Thanks for all the great answers. I would like to ask, if the assay has a particular technical CV, should this not determine the sensitivity of the used technique and should this not be propagated through the experiment into the final results? I can see arguments on both sides of this, but I find it difficult justifying one way over the other.
Keeping in mind what Jason wrote (which represent my opinion too), the CV is normally calculated using QCs (when it is possible to create QCs) . Therefore, a certain batch will have a certain CV. Alternatively, you create technical and biological replicates.
Method optimization will tell you what length (n. of samples) your batch can be extended to or how many replicates you will need to estimate the method uncertainty.
Therefore, my answer to your question is yes, it should be incorporated.
As Jason pointed out, biological uncertainty (which is often the question your experiment is trying to answer to) is one thing, method uncertainty is another thing.
The technical CV will tell you to what degree you can trust your analysis. It will tell you that a biological variation below the tech CV cannot be regarded as significant.
Hence, and addendum to my final answer is the following: if you want to be sure that your technical uncertainty can be included into the final result, make sure your method is optimized and robust enough.
SV said: "If the assay has a particular technical CV, should this not determine the sensitivity of the used technique and should this not be propagated through the experiment into the final results?"
The answer to the first part is the basis of the lower limit of detection (mean result of zero concentration of analyte + 3*SD). In other words, researchers have addressed this topic repeatedly in analytical methods. It's not based directly on CV, but it's another measure of variability that's setting the sensitivity.
As to the second part I am curious how you would propose propagating the method error through the final analysis while maintaining the power of the data set strictly within the number of biological replicates. I don't think there is any real way of doing it in common programs like Prism (or if there is, I have not learned how to do so). Simple workaround approaches to attempt to include all measured values would falsely inflates the n beyond the number of bio reps. So the cure is far worse than the disease, at least in that case.
Regarding the propagation of technical uncertainty through to some final estimate: if the majority of error or noise in the data is random, and could be reasonably expected to be normally or log-normally distributed about the mean of a set of technical replicate measurements, then you can expect that the distribution of the means of independent samples from that biological population to be normally/log-normally distributed as well. If the samples from a biological population are truly random and independent, and the distribution of replicate measurements is approximately normal, then your standard deviation will remain the same whether you performed 3 or 300 samples, as the stdev is mathematically independent of the number of samples, and given the assumptions above, should resemble the "real", biological variation. Precision, or SEM, on the other hand, is directly related to number of biological observations. Perhaps reporting the stdev of the biological replicates with some kind of 95% confidence interval of the technical replicates reported in parallel would be a way to address your concern...I'm unsure. I still think that handling the different sources of variability in different manners may be the most appropriate, i.e. not combining every measurement into a single pot.
On the other hand, if the error/noise is NOT random, as it may not be if you approach a "hard" limit such as a technical LOD/LOQ as Joshua described above, you may wish to use some kind of non-parametric means to describe and compare your biological measurements, as this situation would violate one of the required parameters for commonly used comparison tests (i.e. T-tests, ANOVA, etc). Or separate out all the "too low to quantify accurately" measurements into a separate pot, and describe them qualitatively as possible detects, but at unquantifiable levels.
A side note: is it possible to "trick" some programs like GraphPad into using the variance from "all" measurements without artificially inflating the biological "n", if you were to calculate the SEM or StDev separately (i.e. in Excel or similar), and then input only means, SEM/STDEV and "n", you could designate the "n" to be the number of biological samples, and the SEM/STDEV could be calculated from whatever measurements you choose. I'm by no means recommending this, and it severely limits the analytic and output options of GraphPad when you enter data in this manner.
What Joshua wrote lead me to write the following.
"including the method error" does not mean adding up the error to the result in order to get the biological uncertainty sorted out. Basically, I would not "propagate" this error in my calculations. I would use it as a filter.
This way, the method uncertainty is measured and evaluated in order to filter the final results, diving what can potentially be a hit from what falls into the random event category. Than comes all of the biological probability calculations.
About the use of CV to evaluate sensitivity, I agree with Joshua. CV does not tell me how low I can go. It tells me how precise I am when I measure something. Or, to phrase it better, it should tell me how precise I am at a particular concentration.
Finally, Jason introduced something that, in my opinion, is fundamental when evaluating the finals results. The CI (confidence intervals). CIs can help described both quality and reliability of an experiment.
https://www.researchgate.net/publication/258835771_Decision_making_under_uncertainty_in_energy_systems_State_of_the_art
Article Decision making under uncertainty in energy systems: State of the art
- Badges
- Science topic
- Similar topics
- Medicine
- Physiology
- Biosignals
- Biomedical Signal Processing
- Statistical Analysis