I need to assess the validity of an approach used by a third party.
First, we did paired t-test on each time point across the whole time course of [variable X] to determine statistically significant differences between conditions A and B [for our group of subjects]. Shapiro-Francia normality test has been performed before the paired t-test for each time point and outliers have been removed to ensure the normal distribution.
In other words, they have subjects 1..n in conditions A and B, who are measured and generate time series data in each condition. They isolate each timepoint 1..k and test difference using paired t-test, so obtaining:
t1 = t-test([s1A ... snA], [s1B ... snB])
...
tk = t-test([s1A ... snA], [s1B ... snB])
They then correct for FWE.
So, they have 2 time series from 2 conditions, of the same variable. They want to show statistical difference of conditions. My intuition would say that they should use something like confidence bands: i.e. a principled test that two time series are drawn from separate distributions can be obtained by confidence band methods (e.g. Korpela, etal, 2014 - https://dl.acm.org/citation.cfm?id=2664081).
I would intuit that the t-test approach is undermined by non-iid data, even though they are testing each point separately. But I'm not certain about this. My intuition, I think, stems from the fact that the error bars calculated independently for each time point would be narrower than the confidence band calculated across the whole time series. But I would wish to make this case more clearly, and without possibility of error!
All thoughts appreciated.
" Shapiro-Francia normality test has been performed before the paired t-test for each time point and outliers have been removed to ensure the normal distribution. " - Outch! Two no-go's in a row :)
" I would intuit that the t-test approach is undermined by non-iid data, even though they are testing each point separately " - they test the data of sample 1 against that of sample 2. If the samples are independent, this is fine (although I would use a pooled variance estimate to get the standard errors [it's what post-hoc tests usually do]). The problem then is that data from the same sample but at different time-points is not independent, and this will affect the FWER (or the correction of individual p-values to keep a desired FWER).
If the time points are relatively far apart from each other (a lot can happen in-between), and if these are few (2-4, say) time-points, the approach might be ok (I would say "tolerable"). If these are more time-points and the time spans between are so short that the data from t[n] provifdes considerable information about the data at t[n+1], then one should better use an appropriate regression model (including "sampleID" as a random factor to account for the non-independence). There are very flexible models, like GAMs (e.g. spline regression). These models should ideally be build based on some reasonable theory, so that resonable hypotheses can be tested. Right now the entire approach sounds quite blurry, and it is not very clear to me what valuable insight (scientifically) the rejection of one of these tested hypotheses (from the t-tests) would give us ("hey, these series are not really identical!" is not a very smart conclusion, imho).
You should use time series model describing your data, then you can may estimate model coefficients that would tell you something about the general behaviour of the series and uncertainty regarding these estimations.
For other information, have a look to these links:
https://www.researchgate.net/post/How_to_compare_two_time_series
https://www.researchgate.net/post/How_do_I_statistically_compare_multiple_temperature_time-series_data
start with Cross-correlation ...you will need to look at auto-correlation: partial-correlation to define Time series model...long time back I did arelevant study....
>>>
Dhanoa, M. S., Deriaz, R. E. (1984). Variability of the in vitro digestibility of standard herbage samples Grass and Forage Science, 39, 17-25
Thank you all for responses. Jochen Wilhelm I used your response to deal with the original analysts, so additional thanks to you. It seems a tricky problem, but in the end I think the principle of common sense should apply: the time points are not independent, therefore the tests are not, and therefore we should prefer to use a technique which designed to handle that scenario, rather than patching one which is not.
In case you want to read bit more on time series anysis, I include Reprint of my paper...
- Badges
- Science topic
- Similar topics
- Cognitive Science
- Thinking