This question is concerned with assessing the statistical significance of a result, and is often addressed by calculating a p-value or confidence interval to estimate the probability of obtaining the observed result by chance.
In general, describing the p-value as the probability of obtaining the result "by chance" isn't quite right.
The problem is that when you ask for the chance of an event, that depends on the actual distribution. You can say, given a particular distribution, A, the chance of the event, or something more extreme, is a%, but given the distribution B, the corresponding chance for comparison is b%. That is why looking at different hypotheses and such results, and considering sample size, can give you an idea of effect size. But it is also why interpretation can be tricky. Both p-values and standard errors are functions of sample size, and are related, but stopping at standard error and considering that is more easily interpreted meaningfully.
- Badges
- Science topic
- Similar topics
- Methodology
- Research Methodology