Neither t-test nor the signed-rank test nor any other test cares about where the data comes from and what we think the data represents. So there is no reason to believe that this is incorrect on that basis.

The key point here is the experimental design and the sources of variation.

If you reapeat a measurement you will get a (more or less) different result. If you repeat a measurement on a new preparation (e.g. of a neuron) or on a new individual (e.g. a neuron isolated from another animal) you will get even more variable results.

Knowing the variability between individuals (aka "biological variance") is important when we want to generalize our results to indviduals that we have not included in our study. This is the usual aim of such research.

If we measure our attribute of interest in several individuals (even under best-controlled lab-conditions) the data will be "noisy". This noise is a result of at least two sources of variance: the technical variance (the process of measurement is not perfect and will given more or less different results each time, even on the same individual), and the biological variance (the individuals are different for reasons we can not control and that we often do not even know). We can directly measure (or estimate) the technical variance, simply by replicating measurements on the same individual under the same conditions, but we can not estimate the "pure" biological variance (if we measure different individuals the noise is always the sum of both, technical and biological variance).

Often, the technical variance is negligibly small compared to the biological variance. So there is no need to consider it and the variance of the data is taken as the biological variance. But sometimes the technical variance is very large and by far not negligible. In these cases we need to somehow estimate this technical variance and remove it from the total variance in order to get an estimate for the "pure" biological variance. As noise will cancel out from the average when the number of measurements increases, one can solve this problem simply by doing a number of technical replicates and average their results. These averages can be taken as measurements that have a smaller technical variance (so small that it might be negligible again). The biological variance (between averages obtained from different individuals) is not affected.

This is one reason. Another reason is that the procedures calculating the statistical significance (or, equivalently, standard errors or confidence intervals) refer to a particular level of replication, what is usually the individual (it is about replicating the study, and if it is to be replicated, new individuals would have to be used/measured). These procedures assume that any value stems from a different individual. Including several technical replicates per individual will produce a result under the *wrong* assumption that the data is from many more individuals than have actually been observed.

So, in contrast to your impression, is is usually correct using the averages in such cases, and it would be a mistake not doing so (what is called pseudo-replication; studies using pseudo-replication are usually considered fraud).

I am not sure I understand your thinking. Let's say we have three neurons and we present a stimulus. If we present a stimulus three times and measure the response before and after the stimulus, taking all trials into consideration when doing a test vs taking only the means for each neuron as a single data point can give a different statistical result. Taking the means may be significant if the means are let's say 10, 11 and 12 vs 11, 12 and 13, but taking the individual measurements may not lead to a statistically significant result because of trial to trial variability (large variance). I cannot see how taking the means is a better test than considering the trial to trial variance. It seems equivalent to saying that the size of the variance is irrelevant in making statistical comparisons.

No, the p-value from the means is larger than the p-value of the "per-trial data", because the standard error is larger and the test operates on less degrees of freedom, at least when the ratio of biological to technical variance is not extremely close to zero. So your example is wrong for almost any practical case. The opposite would be right.

I still wonder if you  understand my question. The standard error could be much larger for the per trial observations, than for the means. There is no guarantee it would be smaller. Let's take three neurons with per trial data: First neuron, three trials: 12, 22, 32 spikes, 22±10 (mean±SD). Second neuron: 13, 23, 33 spikes, 23±10 (mean±SD). Third neuron: 14, 24, 34, spikes, 24±10 (mean±SD). The standard deviation of the mean of the three means (22, 23, 24) would only be 1, very much smaller than 10. So by picking a random other triplet of neurons with means let's say of 23, 24 and 25 one could argue that the second triplet has significantly greater spike rates than the first triplet because of the low variance of the mean of means calculated in this way (if the per trial variance is not taken into consideration).You seem to argue that the variance of the mean of means would be greater when it could be smaller as I have just described and as for the number of neurons it remains 3 always, so the degrees of freedom do not change as far as the number of neurons is concerned. In any case a published reference discussing pitfalls in calculations of means of means or medians would be useful.

If the SD in one neuron is 10, the SD between neurons can't be smaller than 10 (it may accidentally happen, though, but statistically the expected SD between neurons includes the variance of the replicate measurements AND the biological variance between different neurons.

Now I think we begin to understand each other. Occasionally you see analyses where variance in the replicates is not taken into consideration. You may include the variance of the replicate measurements when you do the calculations but not everyone does. Mean spike rates are compared in two different situations, using e.g. a t-test, as if the means were single observation data (disregarding their variance). This type of analysis is never mentioned in statistical tests and my feeling and my memory from my student days is that it is not correct. However, I have not found a reference that says that disregarding the variance of the means is an issue. It is likely an issue, even almost certainly an issue, I have just not found a reference dealing with this topic.

Actually, the experiment should be described in the methods section. If each of 5 neurons per group is measured 3 times, and a t-test is performed, then this test is understood being performed on the 2 x 5 averages (of the triplicates per neuron) with (5-1)+(5-1) = 8 degrees of freedom. There is no problem with that.

A problem occurs when the test is in fact performed using the 2 x 5 x 3 = 30 measured values with 18 degrees of freedom. The usual interpretation assumes that this result refers to the knowledge of 30 values from 30 independent neurons. It will vastly overstate the statistical significance of the observed group-difference.

This can be made clear with an extreme case:

I take two single neurons, one treated and one control. I measure each neuron 100 times and observe 23±10 for the control neuron and 26±10 for the treated neuron. The mean difference is 3, and the standard error of this difference, assuming we had 100 independent measurements, is sqrt( 10²/100 + 10²/100 ) = sqrt(2) = 1.14. The t-value is 3/1.14 = 2.12, and the two-sided p-value from the t-distribution with 200-2 d.f. is 0.035. This result tells us that 2 x 100 measurements are enough to sufficiently well recognize the difference between these two neurons. But is tells us NOTHING about what to expect if we would take other neurons. This experiment does not allow us to generalize to other neurons because it gives no information about the variability between neurons. A test considering this fact would be performed on the means per neuron, and here we have only n=1 per group, resulting in (1-1)+(1-1) = 0 degrees of freedom for the test (and such a test is impossible).

Or take n=5 neurons per group, each measured 1000 times, so that the resulting 5 averages per group had a negligible variance. Let the SD between neurons be 1 and the SD between mesurements on the same neuron 10. Analyzed correctly, the technical variance can be ignored when we use the 5 averages per group. The SE of the treatment effect (let it be 1, too) is sqrt(1²/5 + 1²/5) = sqrt(2/5) = 0.63 and the t-value is 1.58 what results in p = 0.15 (with 8 d.f.). If you wrongly use the 2000 values for the test, the treatment effect remains 1, the calculated SE is sqrt(11²/1000+11²/1000) = 0.49, the t-value is 1/0.49 = 2.03 and p = 0.044 (with 1998 d.f.). Note that increasing the number of technical replicates will NOT affect the p-value when the averages are used, but it will result in lower and lower p-values when all the original values are used. That should not be the case, as the additional technical replicates do not provide any more information on the "population of neurons".

Thanks for the illustration. It is of course a statistical pitfall, so it is not completely unrelated to my question. It is not the problem I am concerned with at the moment but of course treating the individual trials from the various neurons as independent observations would artificially increase the degrees of freedom - and decrease the standard error as you have just explained so well. Some researchers sometimes like to use the standard error as an index of variance, and this would make things even worse. Your illustration here can be helpful in case something like that turns up. If you have a reference for that it would also be useful. I think I have something on pseudoreplication but more would not hurt.

http://www.nature.com/nmeth/journal/v11/n9/full/nmeth.3091.html

http://www.stats4biol.info/

You may alo have a look at multilevel-modelling that allows to acknowledge different levels of replication in one experiment (like animal -> cell stock -> cell culture -> culture vial -> measurement; or animal -> organ -> tissue section -> cells).

Excellent! Thank you very much!