Dear Selina

you must specify your question, about p- value it's depends on statistical test and your research questions, so if you use statistics such as t or f or chi square for example

you use p- value, which is varies depends on your sample size and your purpose, and also you must determine whether your assumption is one or two tailed. 

Some other questions might need another criteria as in factor analysis and how to determine no. Of factors so we use parallel analysis and eigen value above 1 for example.

ABOUT ANOVA : you use it when you try to test the hypothesis about if there's a significant differences between the means of more than 2 groups (3 or more) and whether at least one of them differs from another.

ANCOVA is the same case of ANOVA but in this case you have a covariate variable you want to exclude its effect.

after all there's so many books you can get easily in Google, and it's a wide topic to discuss it. 

Good luck

Rana

Dear Selina, 


in principle, it is the same rationale for all tests. You want to to test if your effect is bigger than the error. The simplest example is a two group t-test. You want to know if the difference between two means (effect) is bigger than the error (a measure of the variability of your data). The ratio of this two values,i.e. effect/error, gives you an estimate for the effect size, and how this values distribute determines the according test: t, F, chi etc.

You need the statistical test (i.e. t-test in this case) to estimate how likely your effect size is under the assumption that the difference is zero (which is called null hypothesis testing NHT; H0: there is no difference, H1: the means differ). To put it differently, you would say the difference is statistically significant, if such an effect size appears in less than 5% of all cases, if H0 is correct. BUT, the critical p-value of 5% / p

The p value you obtained is statistically significant or not depends upon the level of significance (α)   you had fixed in your material and method section. The level of significance (α) is usually fixed at 0.05 or 0.01 before start of the study. Based on your level of significance , p value is stated as significant or not. If you had fixed level of significance α =0.05, a p value < 0.05 is statistically significant and if the level of significance α =0.01, a p value < 0.01 is statistically significant.

I recommend the book

"What is a p value anyway?" by Andrew J. Vickers.

Selina -

Some tests make better use of information than others. Consider a sign test and a rank test. Both are distribution-free, but the sign test uses the least of the available information, and is generally less "powerful," meaning that if you look at a type II error analysis to go with a given p-value, the less powerful test would be somewhat analogous to less "coverage" in a confidence interval.

If you use a parametric test, you generally have even more 'power,' although some of it is artificial in that the real distribution will not be exactly as assumed.

A p-value itself, being a function of sample size, means that if you set a given level (often 0.05, as you did, which has unfortunately become a 'standard' of sorts, though that is not good - it should depend on sample size, type of test, and what you are seeking to learn), all you have to do to 'reject' a null hypothesis, even if almost exactly correct, is to increase your sample size. To 'accept' it, just decrease your sample size. I have seen huge purchases justified by saying a hardware test met specifications because a null hypothesis was "accepted," when in reality, the sample size was too small to say much of anything with reasonable certainty. See letter at the first attached link, and if you follow the second link, you can find a detailed example.

When you can use a confidence interval with continuous data, you are in a much better position to draw informed conclusions.

Cheers - Jim

Article Practical Interpretation of Hypothesis Tests - letter to the...

Conference Paper On the Lehmann power analysis for the Wilcoxon rank sum test

Yes, the principle is the same for every test. Every statistical test is a signal-to-noise test, where the signal is the effect size and the 'noise' is a combination of a) the variability of the data and b) the amount of data.

When you think about it, the ability of data to produce chance patterns (noise) is due to two things

a) the variability of the data: data that are very variable can more easily throw up chance patterns

b) the amount of data: these chance patterns tend to be damped out by large amounts of data. Small datasets are more likely to throw up chance patterns.

So we need to calculate the 'noise potential' of the data by combining these two:

Variability of data ÷ amount of data (this is the standard error)

Now we can relate the size of the effect to the pattern-producing potential of the data

Size of effect ÷ standard error

And this is the basis of all tests

The t-test, for example, measures the size of the effect as the difference between the two means, and measures the standard error using the standard deviation to measure variability and the square root of sample size to measure amount of data. 

But all tests work the same way.

It is not only when p

You should also note that failing to reject the null hypothesis is not the same as proving that the null hypothesis is true.

add-on to Timothy:

...and that being able to reject the null hypothesis is not the same as proving that the alternative hypothesis is true!

.

Tests can guide decisions. Tests are not about proofs. And p-values are not  statements about the probability which which a hypothesis is true or false. Tests allow the control of the rate of false decisions among decisions, not among hypotheses. What is generally overseen or ignored is that sensible desired rates can only be determined based on a loss-function, and that "fail to reject H0" is also a decision (since it leads to a particular factual behaviour) that is associated with (often considerable) expected loss. Testing makes no sense when I can't quantify the benefits and losses. Instead the estimated effect size should be substantially interpreted, and if a decision is to be made it should be based on this estimate, not on a p-value.

A fundamental principle of research is that it should not only be rigorously conducted but it should also be presented in such a way that allows readers to judge its value for themselves. This is why not only the methodology should be described but also p values published. The levels of 0.05 0.01 and 0.001 are simply a set of conventions about how to summarise the p value in words. Although 0.05 is the least good of these you can think of it as an absolute threshold for quoting any success. We talk of type 1 and type 2 errors. One involves missing a good result that was in the data. The other involves falsely claiming a result. For you reputation you should always avoid the latter and be conservative in your claims.

" it should also be presented in such a way that allows readers to judge its value for themselves. " - YES!

"but also p values published" - Although is is per se not wrong to show p-values, I really wonder what the benefit of "showing exact p-values" should be. The p-values are as ranom as the data. Even worst! The statistical distribution of the p-values depends on the actual ("true") effect size and on the sample size (besides everything else like the data, the used model, the kind of test, etc).

Error rates are meaingless when you cannot quantify the losses that are associated with a particular wrong decision. Nobody does this in practice (and I say this is usually impossible to do in basic research).

The problem is the use of p-values for interpreting results. The argumentation in most papers is more philosophical ("is there an effect?") or even religious ("what is the true effect?")  than scientifical ("what are plausible estimates of the effect size?"). Such "whether-questions" are not in the domain of empirical science, and we do us no favour thinking in these lines. We should answer "how much-questions" and (report and) interpret the estimated effect sizes.

Substituting p-values by confidence intervals was intended as a step towards this direction, but it failed because the researchers kept their habit of seeking answers to rather religious kinds of questions. At the (sad) end the only thing they were able to do is again to claim "there is an effect" when the CI did not include zero...

How conservative one should be in his claims depends very much on many many things and circumstances. Beeing too conservative can be much more harmful then beeing to liberal. Is depends on so many things. Obnly one thing is clear: any "common rule of thumb" is the worst solution.

I agree that the data should be presented so that it can be used by others, which

alas is not done by p-values or significance tests, contrary to the indoctrination

you seem to have been given.  It would be a miracle if any of the point null

hypotheses you are testing are true; however, there are reasons to act as if

they were, indeed.  I believe it was Box who said that "all models are false,

but some are useful."

The ideal thing to do is to balance ALL the consequences of an action in

ALL states of nature.  It is necessary to approximate.  If you wish for more

information, contact me directly.

i agree with Bryan

The only thing of lasting value that is generated is the raw data (assuming that the experimental design was sound, and the data gathered with care). Yet, how many people take the time to recover the raw data and spend hours re-analyzing it? If the raw data are provided, how many would re-analyze the data just to see if they come to the same conclusions as the authors? My brain is just too small and my time too limited to take in all the raw data ever generated. It is a bad deal, but I am kind of stuck with needing some way to condense thousands of data values into a few numbers. Knowing that treatment 1 is 2.834592 units different from treatment 2 is no more useful than knowing that the p-value for a t-test for treatment differences was 0.059832.

Maybe what would work is to insist that all experiments be replicated 5 or more times. One could then analyze each experiment and develop a 95% CI for the estimated p-value. If you have enough data, you could do something similar using a jackknife/bootstrap method. The problem then becomes that half the manuscript is devoted to describing the analysis methods, and a 15 page MS becomes a 30 page MS and reviewers and readers get grouchy. It also makes the manuscript look unfocused. Also there is the standard funding issues.

Timothy, I think I see you point. But we are living in the time of internet and web-servers. The raw data does not need to be given in the manuscript but it must be made available as supplementary data. It is not a substitute for the analysis made by the authors, it is an addition, to provide the possibility (for those who care) to re-analyzed the data under different aspects, using different models, making different assumptions. It also makes it possible to re-use data in different contexts. Although this is in deatil a difficult thing, but I think it is very valuable to have data from different studies that one can use as reference or control (I think of all the animals that are killed just to get control values over and over again; given the experimental conditions are strictly clear, a collection of published data should allow to estimate the expected outcome for the controls with sufficient precision - even accounting for lab-to-lab differences). This would save money work and time.