Dear all,
I am currently processing the data for one of our study concerning intracranial stimulation. As we had obtained a large number of data points (about 30k) and a lot of our comparisons ended up highly statistical significant I was wondering if I had to add a correction for the large data set, supplementary to the correction for multiple hypothesis testing (for excluding family wise error rate) and reporting the effect size (for relying the discussions upon the most relevant differences).
My gratitude for considering this question,
Mihai
You seem to confuse hypothesis testing and significance testing. If you were really doing hypothesis testing then you had to desing the whole study (including the amount of data you will consider!) to achieve the goals of the test. Hypothesis tests are performed to decide between two given substantial alternatives (A and B). If you have enough data to estimate the response with a precision that easily will separate A from B, the formal test procedure won't give you a rational decision. You would instead simply look if the estimate is close to A or if it is close to B.
I suppose you actually did a significance test: find the expectation of "more extreme data" under some given null hypothesis (H0). This expectation is used to judge the "discredit of H0 by the data", and when this discredit is considered "significant" you would feel inclined to take the observed difference to H0 seriousely. This judgement depends on many factors, and one of these factors surely is the sample size.
If you did significance tests there is nothing like an "error rate" to be controlled. Controlling error rates is a feature of hypothesis tests, and this is not applicable to and not possible with significance tests, at least not in any sensible and rational way. The usual statements in papers that "p
With a large data set and many variables you have another problem. The more darts you throw blindly at a dartboard the greater your chance of finding outcomes that deviate significantly from reality. I typically have 85 univariate models that I work with. The system is not well enough known that I can make rational choices to reduce this number to 2 or 3 most important variables. So I am faced with the prospect that it is very likely that I will find significant differences that have nothing to do with the system I study. I thought that the following article might be useful.
http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124
For effect size in a multivariate analysis try looking up Mahalanobis distance.
- Badges
- Science topic
- Similar topics
- Correction