Example:

Say that you actually want to know the effect of a drug on the blood pressure of patients.

You can measure the change in blood pressure in a given patient. Fine. But what about other patients? Will the drug have the very same effect? Surely not. So we have a problem. Statistics provides a solution: we can determine the drug-effect for a larger sample of patients and model our knowledge about the drug-effect based on a probability model. We can then use that data to estimate a plausible (credible) range for the effect, or determine if the data is sufficient to conclude the direction of the effect (increase or decrease). The statistical model is based on the number of patients. The more patients, the more information you have about the variability of the drug-effect and the more precise can be the estimate and the more confidently you can conclude the direction of the effect.

The variability of the measurements you have is attributable to the variability between patients (biological variance), but also to the variability of the measurement process (technical variance). This means, even if you would measure the drug-effect several times in the same patient, the imperfect measurement process would give you different results each time. Statistics tells us that the average of the replicate measurements will vary less than the individual measurements. Averaging will reduce the influence of the technical variation. Therefore we can use the average as a better estimate of the drug-effect of an individual patient.

But what remains important for your scientific question is, how variable the drug-effects are between different patients. Even if you measured the same patient 1000 times, you know nothing about the biological variance. A statistical analysis on such 1000 values takes these values as if they were from 1000 patients, so it assumes a huge amount of information about the biological variance - that is not there! The results still refer to a single unique, possibly special patient and it is not at all clear if and how this result may be generalized to another patient (or to "patient" in general).

Only measureing different patients will give you that required information, and the amount of information does increase only with the number of patients in your study, but not with the number of replicate measurements you take from each patient. Taking several replicate measurements is useful/advantagous/rewarding only when the technical variance is large compared to the biological variance and when the sample size is rather small (the fewer patients you have, the more important are good estimates for each patient whereas in large samples the technical variance part does not have any great impact on the result anyway).

You can use hierarchical (multilevel) models to model the technical variance separately from the biological variance, you can also model the replication structure as fixed effects ("ANCOVA" designs), but often it is simpler to average the replicate measurements per patient and use a single (average) value per patient in the analysis.

Hi Jochen,

Many thanks for your answer. My sample size for this study is relatively small, however I am confident we got the most amount of samples that we could during fieldwork; and trying to get more samples is now impossible at this stage. Therefore the results of the analysis will probably not give the best indication of biological variance; however due to my sample size being small, would I be able to use repeated measures in this instance? It is rather frustrating, as an analysis of repeated measures shows a clear significant effect, whereas taking an average of each individual in control and disease groups returns a P-value which is only slightly too high (e.g. 0.06, 0,07).

Once again, many thanks for your detailed answer. 

You can always take as many replicate measurements on each individual as you want. It may just not be beneficial when the technical variance is rather small and when each measurement is costly. If getting more samples is possible, then you should do so.

Regarding your p-values: you can have a single individual in each of two groups, say, and 1000 replicate measurements for these two. If you run a t-test on these 2x1000 values you may find a tiny p-value. This p-values tells you that 1000 replicate measurements are clearly sufficient to conclude the direction of the mean difference between these two individuals. It does not tell you anything about how the difference may look like in other individuals. It can be anything. You have no information on that! So you cannot say if or how this difference -between these two individuals you measured - can be attributed to the fact that they do belong to different groups, or if it is just a meaningless difference that would likely be observed for any two individuals, no matter to which group they belong.

If you average the 1000 replicate measurements, you end up with n=1 averages per group, these are as many values as you have individuals. Running a t-test with these averaged values will give you no result at all. The test won't even work here. This makes sense, because now the test should would be about the difference between groups, but there is no information about the variability (within and between the groups), so the test must fail.

If you had a few individuals per group, the test can calculate a p-value, as now there is some information on the variability of the individuals. The p-values will become smaller with increasing sample size. Again, if you would NOT average the replicate measurements, the p-value is likely quite small, but it is actually NOT interpretable, as the test would mix the variability between individuals with the variability between replicate measurements (the test cannot "see" if a value is from a new individual or just a replicate). So your small p-value may be small, but it is meaningless, because it is not interpretable.

Please let me know if these references/sites are helpful to you:

1.  Why small sample size undermines the reliability of neuroscience

https://brain.mpg.de/fileadmin/user_upload/images/IMPRS/Master_Reading_List/small_samples_Nature_Reviews.pdf

Apr 10, 2013 ... Low statistical power (because of low sample size of studies, small effects or both ) nega- tively affects the likelihood that a nominally statistically.

2.  Small sample size generalization - CiteSeerX

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.132.2874&rep=rep1&type=pdf  Abstract. The generalization of linear classifiers is considered for training sample sizes smaller than the feature size. It is shown that there exists a good linear ...

3.  SAMPLE SIZE IN PSYCHOLOGICAL RESEARCH OVER THE PAST ...

http://emilkirkegaard.dk/en/wp-content/uploads/Sample-size-in-psychological-research-over-the-past-30-years.pdf

Sample sizes have not been integrated in core psychological research, although re- ... found power for studies reporting small effect sizes to be .36—much less.

4.  Sample Size - Gerald van Belle

http://www.vanbelle.org/chapters/webchapter2.pdfThe ingredients in a sample size calculation, for one or two groups, are: ..... It is frequently helpful to be able to estimate quickly the variability in a small data set.

Dennis

Dennis Mazur

Dear All,

Many thanks for your detailed answers. The sample size is a disadvantage, however we are going to try and use other tests to measure correlations and to try some regression analysis. 

Thanks once more for your replies.

Adam, if your p-values are "too high" -- e.g., 0.06 or 0.07, this simply means that the results are significant---at that p-value. i.e., six times out of 100 you could expect those results by chance alone.  Should you discard your results as "non-significant" because they could occur by chance six times out of 100 instead of only five times out of 100?  There is still a bias out there about the p-value having to be .05 or less to be meaningful, but this value is really arbitrary and came about through a historical fluke as much as anything.  In our papers I like to write the results as a compromise, saying for example "the differences between subjects and controls was marginally significant at p = .07" and let the reader decide on the importance of that difference.  Many journal editors are shifting to that practice now.  The p-value is but one indicator of the "significance" or importance of a research finding  -- there are effect sizes to consider, as well as sample size and the statistical power of the study.  Some have argued that in studies with small sample sizes, a p-value of 0.10 or lower should be considered significant.  

@Jerry,

although I understand your motivation, I think it is not a good idea to give such advice generally, i.e. without referring to the particular case. If the response is easy to measure, with little variance, and the sample size is okayish, I wouldn't even consider 0.04 as "significant". I agree that there are studies where 0.1 might be considered "significant", but nut only because of a low sample size - it would require good arguments why it is so difficult to measure the effect and to get a larger sample.