CLARIFICATION OF WHAT TEST TO USE

The common confusion is the use of various statistical tests in which situation: (i) one sample with normal distribution, (ii) two or more sample with normal distribution, (iii) small single sample which is non-normally distributed; (iv) two samples which are non-normally distributed. The followings are various test commonly used for continuous data.

F-STATISTICS

F-test is used only for group comparison. When there is more than one group of population sample, then F-test is used. A second situation where the F-test is use is where there are two small samples which display chi-square distribution. The test statistic is given by:

(1) F = (S1^2) / (S2^2)

… where S1^2 = variance of sample 1 and S2^2 = variance of sample 2. The critical value is read off the F-table where S1 has degree of freedom for the numerator and S2 has the degrees of freedom for the denominator. Where the two degree of freedoms (v1 : v2) intersects, the “critical value is found”----compare this value to the F(observed) from equation (1) above. Of F(observed) is smaller than F(standard), then the two populations or samples are relatively the same or similar, i.e. there is no statistically significant difference between the two samples.

Generally, when there is a replication study, i.e. trying to confirm prior studies, researcher would use the original study as sample-1 and treat the current study as sample-2. If F(observed) < F(standard), it is fair to say that the prior study has been replicated, i.e. confirmed. If series of experiments in subsequent studies shows F(observed) > F(standard), then there is an issue: whether the prior study is wrong or the subsequent study did not follow exactly the same procedures of the prior study.

T-TEST

T-test is use when there is one sample involved. The confidence interval is generally set at 0.95. The 0.95 confidence interval represents the mean +/- 2 units of standard deviation. The t-equation is given by:

(2) T = (x^ - mu) / (S/sqrt n)

Z-TEST

The Z-test is used for comparing the sample to the assumed population which is assumed to be normally distributed.

(3) Z = (x^ - mu) / (sigma / sqrt n)

CHI SQUARE TEST

If the sample is small and is non-normal, a chi-square test is used. The chi square test statistic is given by:

(4) Chi square = (n – 1)S^2 / sigma^2

(n – 1) is the degrees of freedom. S^2 = sample variance, and sigma^2 = population variance. Sigma^2 may be derived from Z-equation. Elsewhere, chi square is commonly given as:

(5) Chi square = (O – E)^2 / E

… where O = observed value, E = expected value.

The purpose of chi square test is to determine best fit of the data to the expected distribution. Since the data is continuous, the expected distribution is “normal distribution.” Thus, statement (4) above is more of a common test statistic because it involves the use of standard scores via t-test and Z-test.

BETA

Beta is used to determine the probability of Type II error. If there is a significant beta, it means that the alternative hypothesis has been wrongly rejected, i.e. the confidence defined by mean + units of standard deviation intrudes into +alpha region (actual means that there is a significance finding), BUT the null hypothesis is not rejected or the alternatively hypothesis has been rejected. This test is used in all hypothesis tests involving:

T-test

Z-test

F-test

Chi square test

These statistical tests are used for hypothesis testing. Beta test is used to determine whether the conclusion reached in the hypothesis testing was correct. Thus, beta is a test of a test statistic’s conclusion reached by the researcher.

VALIDITY & RELIABILITY OF MODEL

F-test does not answer the question of validity or reliability of the model, i.e. F-test does not say any thing about the quality of the model. It is used for the comparison of observations from two data arrays. Separate tests for validity and reliability of the model may be accomplished separately. Generally, that is done through correlation analysis. Although in some circumstances, F-test may be involved, but it is used in a different context than comparison studies.

Impressive...thanks a lot Paul for the wonderful piece of clarification on the issue...Your points are well noted and very much appreciated

One cannot rely on F-test only. For instance, the F-statistic could be impressive but this is not guarantee that the regression is OK, because even in that case the regression may be spurious. In fact, F-statistic imply only that all regressors are not zeros, which is not very informative, i.e. in most cases one will find that ALL regressors are not zero (even if some of them are zero).

thanks a lot Goran for the opinion shared. Your point is well noted and very much appreciated

"Thus, must a significant F-statistic always imply good, perfect, valid or best model?"

The answer is simple: NO

There are two reasons:

1) the p-value is the probability of the data, given the null hypothesis AND the correct model AND all assumptions being correct. As the p-value does not contain any *direct* information about the "correctnes" of the null hypothesis it neither contains any direct information about the correctness about the entire model.

2) the habit of rejecting some null hypothesis based *solely* on a test statistc is the Neymanian, purely decision-theoretic approach. It is only about decisions and holding defined error-reates in the long run of makinig many similar decisions. It is not in any way related to assertions about the "correctnes", "goodness", "validity", "reliability" or "usefulness" of a model.

The value of a model is not realted to test statistics (or p-values). A good model is relatively (as compared to other, possibly competing models) simple, instructive, useful and reliable. Reliability can never be judged based on data that was used to set up the model. Only the concordance of new data from other, independent studies can show how reliable a model is.

thanks a lot Jochen for the wonderful opinion shared...Your suggestion is well noted and very much appreciated