The dependency of the significance level and sample size is well recognised in medical research, where the sample size is often decided by a researcher. In economic research the sample size is often given and cannot be changed. It is well noted that testing the same hypothesis with different sample sizes leads to different results, i.e. in cases of small sample size (e.g. below 100) p-values indicate lower significance. If we use the same cutoff for p-values, then the estimation with small samples yields fewer rejections of the Ho (no relationship between independent and dependent variables) than the same estimation with big samples. Thus, small sample may mean often no relationship, whereas big sample is associated with better chances to define relationship with acceptable level of significance. This sounds not consistent. P-value cutoff should take into account the sample size and keep the balance between type 1 and type 2 errors.
How to do this? Please share respective literature and your opinion.
See this blog post by Dan Lakens and take a look at the references:
http://daniellakens.blogspot.com/2018/12/testing-whether-observed-data-should.html
Thank you much, very useful links. I am deeply impressed by Good's insights. However, the suggested is more relevant for big samples and does not work for small samples as in my case. The choice of 100 is a question? Anyway, these are very good texts to learn from.
Type I and Type II errors are interdependent. You cannot adjust one without affecting the other. For instance, when alpha is set to .05, beta is .20. If you decrease the alpha level this will simultaneously increase beta and vice versa. Adjustments should be based not on trying to balance Type I/Type II errors, but based on which type of error is more problematic in the particular situation (in my opinion). In short, there is no 'optimal' p-value cutoff.
Better yet: forget p-values and use a Bayesian approach.
"There's nothing inherently special about setting alpha to .05" - a former professor of mine
in med research it is possible to choose one type of error as priority and focus on it. In economics my perception is that both errors are important, and one cannot be preferred over another. And because of interdependency fixing alpha may mean very high type II error, which is not good and leads to conclusions pre-defined depending on what was chosen to put in null and alternative hypothesis. My goal in finding an optimal alpha is about minimising two interconnected errors.
I will learn the Bayesian approach, thanks for the recommendation
this article seems to answer the question
Article Setting an Optimal α That Minimizes Errors in Null Hypothesi...
I plan to use in STATA power analysis to estimate type II error given my sample and different levels of type I error. Then as the article recommends to choose an optimal combination of two errors
Technically they alpha and beta apply only to the the long-run. To find out beta you need to know what you consider as an "effect" or what the "effect" might be. Alpha is fixed beforehand based on expertise on perhaps known noise and variance in the data as in Thom Baguley reference.
For example in the figure below I fixed the "true effect" as being 0.1 (the vertical dark line). Each of the horizontal points with the sticks represents the point estimate ("effect size") and confidence/coverage intervals of an experiment (100 repetitions). I fixed alpha at 0.05 and confidence intervals are at 95%, indicating 95% cover the "true effect" (under ideal conditions and exact repetition). However only 70% of experiments are significant, indicating that the confidence intervals do not cover the NULL (0) (the vertical dashed red line). Thus our beta would 70%. However, I only know this by fixing the "effect" at 0.1, in reality we do not know the "effect" would be 0.1, we would try to find this out. Hereby repetition would be the ideal way. Only by repetition we would approximate the "true" effect and our error (under ideal conditions and exact repetition).
To me it seems "difficult" to deterimne beta (and alpha) after the experiment. As they are usefull metaphysics (epistemic though) as it comes to fix a baseline before the experiment, but not after when everything has been observed. When determining beta afterwards you suggest the observed "effect" would be the actual effect (I think?), but we want to find this out with the long-run, not the statistic perse.
Iam rather idealistic (and why I am possibly wrong) so if we did not think about the experimental setup (e.g., sample size, exluding confouders etc.) and we do not repeat things. Perhaps it is beter to drop the idealistic point of errors as being "real" and focus more on the idea of directionality and vauge undestanding of under/overestimation (not sure). I am still not sure if it is possible there is a methodologized logic of inference in which we can say something is "true" or "correct" and it bothers me (at least something in the trend of inductive validity would be a start).
Best,
I did estimation for sample n=85 of 8 financial consumer protection policies on one economic indicator. The sample cannot be increased, these are 85 countries in the WB survey. The hull hypothesis in the testing is that the coefficient is about 0, i.e. the policy does not have association with the indicator. In such cases of policy reforms focus on alpha and ignoring beta may lead to anti-reform perception. It is hard to beat the null hypothesis and beta being high represents the risk of not doing effective reform. In economics there are cases when samples are relatively small (number of countries), surveys for panel data are costly, or the relationships like responses to rare crisis are unique and appear with some lag.
Added p-values for the independent variables (8 financial consumer protection policies). With alpha level that minimizes both errors (type I + II) three policies are significant to have association with the indicator, one policy is on margin (the orange), other four policies are not significant with the balanced alpha. If risk tolerance allows alpha at 15% level, then three policies are considered significant. Decision about policy #2 with marginal significance (p-value 0.20) with balanced alpha (25%) may resolve with risk tolerance consideration, i.e. is it acceptable to have alpha 25% and beta 26%?
Would you agree with this logic? Any other considerations?
Sayed Ibrahim Ali
Yes, your recommendation is very valuable and in line with the other findings. In the case of my research question about certain reform choice in financial consumer protection area the sample cannot be increased, it is determined by the number of countries that participated in WB FICP survey. Also for a reform consideration, particularly when the target indicator is a "burning" issue, statistical probability of reform impact may go well beyond 5% to 20-30% in order to balance the probability of rejecting an efficient reform, in my opinion. And then it can be reconciled with case by case and context analysis. Not balancing alpha and beta and focusing only at alpha may lead to anti-reform conclusions, i.e. high probability of rejecting efficient reforms. The point on large samples and effect size are also very supported. Many thanks for your input.
Other useful links
Article Power and Sample Size Calculations. A Review and Computer Program
Article Power and Sample Size Calculations for Studies Involving Lin...
Article Power Analyses for Detecting Effects for Multiple Coefficien...
Article Power Analysis for Regression Coefficients: The Role of Mult...
blog with short explanation and formula on slope coefficient
https://stats.stackexchange.com/questions/86871/r-power-calculation-for-beta-coefficient
- Badges
- Science topic
- Similar topics
- Philosophy
- Esthetics
- Interpretation