It is a common concern to hold alpha values at 0.05 and that it is completely arbitrary. I have been interested in using other methods, but I know that there may be pushback from editors to use standard statistics rather than weighing type I and type II errors and choosing an alpha value from there. I've seen it done before, but how much is the real drawback from doing that in publishing?
The alpha level is the least of your concerns in my opinion. Of much greater importance is working hard at the experimental design phase to identify key research questions. Follow up by identifying the experimental design statistical analysis methods and adequate level of data to satisfy the identified data objectives. To avoid the alpha issue, make sure that your approach emphasizes estimation and confidence intervals as the primary way of summarizing results.
It is also critical to conduct a power analysis to insure that the experimental design is adequate to estimate parameters with adequate levels of precision to answer research questions. Funding decisions should be based on the adequacy of the design and number of samples to answer questions with a high likelihood of unambiguous results..
You can easily justify your selection by backing up your decisions via a power analysis software, e.g. G-power. What is important is to adequately select your statistical test according to how your data is, e.g. if response/predictor variables are continous or categorical, if they behave well, etc
It is not necessary to set alpha level at 0.05. But it will be very dificult to justify a larger value... If you want to set it at 0.01 or 0.001, you just need to explain why to be more conservative about it.
About beta level, it is always welcome when authors demonstrate their beta levels, especially if it was calculated a priori
You should use the methods and statistics that are consistent with your research question, purpose & rationale. Most probably if it is post-positive epistemology then ensure your methods align with that paradigm.
The alpha level is the least of your concerns in my opinion. Of much greater importance is working hard at the experimental design phase to identify key research questions. Follow up by identifying the experimental design statistical analysis methods and adequate level of data to satisfy the identified data objectives. To avoid the alpha issue, make sure that your approach emphasizes estimation and confidence intervals as the primary way of summarizing results.
It is also critical to conduct a power analysis to insure that the experimental design is adequate to estimate parameters with adequate levels of precision to answer research questions. Funding decisions should be based on the adequacy of the design and number of samples to answer questions with a high likelihood of unambiguous results..
Forget alpha values, significance and so forth and concentrate on the clarity of results. A good statistical analysis is an analysis that allows to immediately grasp what happened NOT ONLY in terms of p but, much more important, in terms of the order of magnitude of the observed effect by means of descriptors that link in a natural way to the phenomenon you are describing..see the interesting paper attached..
Thank you for all the suggestions. I definitely agree that choosing the right alpha levels depend on the experimental design and we should start analyzing our data critically, rather than making it fit a specific analysis in order to get our p
- Badges
- Science topic
- Similar topics
- Publishing