Why do wou want to explain these terms?

In a manuscript you either give the observed significance ("p value") or report a common level of significance (that is needed to interpret the data w.r.t. the tested hypotheses; "alpha"), and you five interval estimates of a specified levels of confidence (usually 95%). It should be expected that the reader knows what "(statistical) significance" and "(statistical) confidence" mean (although, sadly, this is actually not the case -- otherwise there weren't so many misinterpretations).

If your paper is about statistics, the RG forum might not be a good place to provide you sufficient expertise in these things...

You can consult good econometrics text books for guidance.

All the best

I agree with Samuel Tawiah Baidoo. It can be found in textbooks. However, there are common mistakes. Just by chance I have discussed some of them in my lecture this week, see included material. There are other mistakes which cannot be explained in a short note. I am planning to write a paper about it, but it may take a year. If you wish I keep you posted.

Best regards,

Michael Grabinski

Finding a good textbook about statistics, especially at an introductory level, is really hard! Most textbooks explain NHST (which is itself a faulty and illogical concept), and they give plain wrong explanations about significance and confidence. I have a collection of such (bad) books in my shelf.

Very often "made simple" essentially turns out to be "made wrong".

I don't have an overview of books addressing econometrics, but there is apaper reporting an overview about textbooks for phychology students. I don't assume that this is different in any other area:

Article Failing Grade: 89% of Introduction-to-Psychology Textbooks T...

In the book "A Guide to What's Wrong with Economics" (ISBN1843311488) I found on page 231 following paragraph:

"We looked at all the textbooks, from advanced econometrics books to elementary statistics for business and economics and found that, with few exceptions, professors propagate the very error we're preaching against (Ziliak and McCloseky 2004; McClosky and Ziliak 1996). Dear students: show your teachers how to correct the error, politely, and open handedly, and very patiently (remember: Professor's supposed to know). Dont fear. Today you're at worst a victim, tomorrow a saint."

To build on Jochen Wilhelm, there are indeed many textbooks giving wrong answers. The maybe worst mistake is to assume a Gaussian distribution in almost all cases. Surprisingly this mistake is already described in a textbook of 1924 (The Calculus Of Observations by Whittaker and Robinson) On page 179 one can read: "Everybody believes in the exponential law of errors: the experimenters, because they think it can be proved by mathematics; and the mathematicians, because they believe it has been established by observation".

First of all, explain yourself the words and ask & describe what they mean in the original language - not in latin:

latin = confidence: e.g. truth

latin = significance: e.g. meaning

what does they mean in your language?

In other words: If analizing numbers you should answer the questions:

Are the numbers of a true source? What does the frequency mean?

Second: try to find out if the well known density distributions are valid for your work, perhaps you can use the Bayesian concept for your work.

Use also popular publications to find out what is useful.

I recommend the book of Andrew Jawlik "Statistics from A to Z" Confusing concepts are clarified. It includes your questions.

The content of the book is great in terms of explaining the meaning of each confusing statistical concepts.

Level of significance is the probability of Type I error (alpha) in testing statistical hypotheses. In testing of hypothesis, we fix alpha and try to minimize the probability of type II error (Beta). Alpha is also associated with p-value.

Confidence level is in reference to "Interval Estimation". A confidence level of alpha will mean the interval estimate (confidence interval) will include the true value of the parameter with alpha confidence. For example 95% confidence interval based on sample size of 50, say can be interpreted as follows: If we draw 100 samples,each of size 50, then 95 samples will cover the true value of the parameter.

Jochen Wilhelm , I think he has asked about on writing the CI and p-value of the statistical model in details rather than just reporting it in the manuscript.

Most journals accept extremely bad reporting styles, so looking at some example publications offers you all kinds of crap. This mean, in practice, you can (almost) do what you want. However, a good way of reporting is to write the conclusion, the test, the test statistic, the number of degreesof freedom, the observed p-value and, optionally, the limits of the confidence interval of aspecified level of confidence. If the test is obvious from the context and the given test statistic, it's sufficient to give the statistic.

Here is an example for the test of a mean difference between two groups (treated and control) where the observed estimate (treated-control) was positive:

The treatment effect was positive (Welsh-t, t=3.49, d.f.=7.161, p=0.01, 95%CI=0.52...2.86).

Here an example for an effect considered"not significant":

The treatment effect was not conclusive (Welsh-t, t=1.68, d.f.=7.8, p=0.13, 95%CI=-0.44...2.84).

For experiments with several (or many) pair-wise comparisons between groups, I would describe the model and the multiple testing procedure (e.g. Dunnett or Tukey etc.) together with the residual degrees of freedom once and the family-wise error rate that is accepted. The results are usually summarized in a diagram indicating which comparisons are considered significant. A catepillar plot can be used to show all the (adjusted) confidence intervals.

In these examples it is understood that the CI refers to the mean difference. For other tests it might be neccesary to specify what is meant, for instance if the CI is given for the odds ratio or for the log odds ratio.

More complicated statistical models (e.g. including covariables, repeated measures, random factors etc) are best presented using a formalized way to describe the model and complete coefficients table plus some relevant statistics about the overall model.

Thank you for your valuable answer Girdhar Agarwal ji

Girdhar Agarwal gave an neat, compact formulation of the confidence interval:

"For example 95% confidence interval based on sample size of 50, say can be interpreted as follows: If we draw 100 samples, each of size 50, then 95 samples will cover the true value of the parameter."

The next remark has little practical value, but it might result in better understanding of the mechanism: the percentage of the 100 samples which cover the true value of the parameter is unobservable. Also, this percentage is a random number, with a distribution around 95% and zooming in on 95% if the number of drawn samples increases.

Here is a simple piece of R code for having fun, experimenting with the concept of confidence intervals:

pop=1:100; smplsize=10; replications=25

coef

Hi hope this paper helps

Article Choosing the Level of Significance: A Decision‐theoretic Approach

I also attach my working paper, which uses my R package OptSig

https://cran.r-project.org/web/packages/OptSig/index.html

.

@Jae-Hoon Kim

nice paper ; i used to understand the choice of the significance level along those lines but then i stumbled on the following and maybe you can help me out of my perplexity

.

the expected risk is written

P*L1*a + (1-P)*L2*b

with P my prior probability on H0 to be true

a, b the first and second type error rates and L1, L2 the associated costs

.

so far so good ... but b, the second error rate (not rejecting the H0 when it is indeed true) is computed under the H0 hypothesis and ...

... it seems odd to evaluate the risk, when i do not believe H0 to be true (the second term (1-P)*L2*b) with a second error rate b computed under the H0 hypothesis

.

can you help me out of my perplexity ?

.

Hi Fabrice,

b (Prob Type II error) should be calculated under H1, not H0.

So you need to specify a meaningful value under H1: a value of economic significance when H0 is violated. It is calculated in the same way as power since power = 1 -b,

In my OptSig package, it can be done by spcifiying a value for Cohen's d or ncp (non-centrality parameter). Cohen'd = (mu0 - mu1)/s, where mu0 is the value of parameter under H0, mu1 is a value under H1, and s is the standard deviation.

The choice of mu1 is importance and it should make sense economically and practically. My examples in the attached paper (Kim2020.pdf) should help with simple examples.

JHK