There are several issues related to your excellent observation/question, and while people have been talking about this problem for a long time (e.g., Meehl on differences between physics and psychology, Rosenthal on the file-drawer problem), there has more recently been proposals that will likely increase the likelihood of non-significant results being published. This include pre-registration and editors telling reviewers not just to look for significance. A problem though, with both significant and non-significant results, is that many studies are under-powered. If researchers have not collected enough data to provide measurement that are precise enough for the purpose, reviewers should recommend rejection.

There are several issues related to your excellent observation/question, and while people have been talking about this problem for a long time (e.g., Meehl on differences between physics and psychology, Rosenthal on the file-drawer problem), there has more recently been proposals that will likely increase the likelihood of non-significant results being published. This include pre-registration and editors telling reviewers not just to look for significance. A problem though, with both significant and non-significant results, is that many studies are under-powered. If researchers have not collected enough data to provide measurement that are precise enough for the purpose, reviewers should recommend rejection.

Thank you Dr. Daniel

This an interesting read

'The garden of forking paths: Why multiple comparisons can be a problem, even when there is no "fishing expedition" or "p-hacking" and the research hypothesis was posited ahead of time' by Andrew Gelmany and Eric Lokenz

http://www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdf

It shows the problems of finding 'significant' results even whene there is not multiple testing sensu stricto ... "we discussed several prominent research papers in which statistical significance was attained via a sort of invisible multiplicity: data-dependent analysis choices"

Thank you so much Prof. Kelvyn Jones.

Dear Muhammad,

Very interesting question there. The issues as regards significant and insignificant levels within social science research have lingered for long. In some cases, some insignificant findings are very unique and berg for explanation particularly when it digress from a well established theory but means something else meaningful.

However, most reviewers and editors discredit insignificant results because it is difficult to ascribe unique reasons for these results. For, example, insignificant findings mostly have strong linkages with (1) small sample size (2) poor sampling efforts and strategies (3) application of unsuitable modeling techniques, etc. In this regards, it is really hard to relate insignificant results with the perfect situation and, therefore, often tend to be disregarded.

As you know a p-value of 0.05 is often chosen a priori. It has been used as a kind of standard. Having said this there have been people I have encountered in my life who will carry out an analysis and make the data confess, i.e., they get a p-value of 0.10 and they start boasting that the "improved" method they suggested is statstically significant. As pointed out correctly by Dr. Wright a large enough sample size should have been chosen. The size is determined often by the statistical power beta. Once one has done that and is willing to accept a p-significance of 0.05 or better, stick with it and do not keep monkeying around with the alpha value.

Hello Muhammad Usman

Thanks for raising an interesting issue. I have few observations about this issue. Recently, editors and reviewers started to accept the links which are insignificant given that the ration is not large in number compare to total hypotheses. It is quite embarrassing if all the proposed relationships in the model are insignificant. One logic behind that the model (or relationships) that you have developed is expected to be based on a theory. Testing a relationship without grounded on any theory is a risky one which may lead to unexpected outcome. In social science (unlike natural science) nothing is perfect even a particular construct which has been well accepted and statically tested has become eventually questionable (at least in terms of measure). in this connection, context and culture are important to play the role in determining the assumed relationship. Finally, sample size does matter for testing the assumed relationship because we test the hypothesis based on sample information to infer about entire population. From my side, having one or few insignificant relationship is normal and accepted (if you have proper justification) but not all.

Thank you very much all of you for your time and response. Very much appreciated.

"This creates a perception that those studies with non-significant results are not accepted for publication." Well, it depends on your hypotheses. Sometimes, insignificance may be very valuable result. But this problem is more intricate: https://www.nature.com/news/big-names-in-statistics-want-to-shake-up-much-maligned-p-value-1.22375

Dear Researcher,

Generally hypothesis are generated from experience, observations and literatures i.e. the things that are already existing in the society. Thus it is more likely to show statistical significance, but not always. However, It is a old concept that only those articles having lots of statistical significance are or should be published in a journal. If research is carried out scientifically but still does not show any significance, it is itself a king finding that research community should be aware about and thus carry the equal weightage.

On the Basant Kumar Karn comment, here is a relevant quote from a great paper by Bill McGuire: " pretty much have to follow. Experiments on such hypotheses naturally turn out to be more like demonstrations than tests. If the experiment does not come out "right," then the researcher does not say that the hypothesis is wrong but rather that something was wrong with the experiment, and he corrects and revises it, perhaps by using more appropriate subjects, by strengthening the independent variable manipulation, by blocking off extraneous response possibilities, or by setting up a more appropriate context, etc. " Paper is at https://pdfs.semanticscholar.org/8473/c6ca382305d80a7e6a284d49d753d5062d30.pdf. A common response is to a "significant" finding is that it is either obvious or wrong. Of course that is what we as scientists should be doing, and the problems aren't across all disciplines and all labs.

The discussion is a little misleading. Usually a null hypothesis of the form 'no effect' is tested, but that need no be the case. When 'parameter = 0' can be rejected, it is assumed a parameter has been measured/exists and it is relevant to tell. It does not seem to be important to let scientist know that some parameters appear to be 0. But a significance test can be any test of the form "parameter = c" or (p1 =c1, p2 =c2) (c can be any constant including 0). Most scientist do not use this form, but I do as I follow the betting model for science (by W.K.B Hofstee, De empirische discussie (the empirical discussion)). Am i willing to bet money on the results of a research I am performing? Am I willing to suggest values for all parameters in my model (maybe including confidence interval as with Bayesian statistics)? Reading publications should help me in 'best guest estimates. Image you are given N variables from any sciences and are requested to built the best model for these N variables. Some of the N variables might be unrelated and can or will not be within the best model for these N variables. In the end sciences needs to be able to tell for all and any variable imaginable if they are related or not. And if they are related how strongly the relation will be. This also includes knowledge about how variables are not related.

"why is it necessary to have statistically significant results?" In a short, because this is the only way to make sure if your results is TRUE or NOT, and does not really mean it is important! If it’s significant, then is a true value, and if it is not statistically significant, then is not true value. Either one is vital as a research outcome because it’s emerging out of the research process.

@Bahram, statistics can never make results true or not. Popper dismissed the idea that statistics could be used as a demarcation criteria. You may have been unlucky in having drawn the wrong sample and have insignificant results. It does not make it NOT TRUE. It does not provide sufficient corroboration. When parameter values are close to zero in the population, you might not be able to find these values with the sample you have, but if you have a lot of samples they should be significant in most of them.

Dear Muhammad

One of the biggest problems of our contemporary scientific community is that it has biased its perspective towards positivism and the indiscriminate use of statistics as a tool for analysis. I think that in the area of ​​social sciences we should not focus exclusively on statistical significance, a concept that is more applicable in hard sciences and it is supported in assumptions that are hardly fulfilled when we work with humans or society. Gauss himself, never imagined that his model of the normal distribution and subsequent applications were used to model social behavior and now we are using it indiscriminately. I believe that the social sciences should be better supported in qualitative approaches and historical - hermeneutic perspectives, as Weber said, than to try to copy or transpose methods that hardly explain human and social behavior.

Very well explained one of the key issues in social science research.Thank you Dr. Leon DARIO

León Darío Parra Bernal ,

I do not see a big distinction between quantitative and qualitative. There are a lot of statistical methods that can work on different types of data. As long as you can categorize information and make it ‘data’ one can find a suitable statistical procedure. In a Generalized Linear Model (GLM, but still a simple statistical model) one can model a regression analysis with a chi-square distribution. It would not be simple to explain why one would do that, but it is possible. Assuming a normal distribution is not necessary.

If you can not categorize data, you might not able to convey ideas or concept to others and you can not have ‘science’. Even narratives can be analyzed using text-mining techniques. I do not believe the social sciences would be ‘better served’ by exclusively using one or a few possibilities.

I do not think ‘ …we are using it (the normal distribution) indiscriminately’. Since the development of the GLM, we have included many distributions that are not a ‘normal distribution’. See https://en.wikipedia.org/wiki/Exponential_family to show you which distributions can be used in a GLM.

I'm agree with you Peter. There are a lot of models that do not consider the normal distribution as statistical assumption, such as the case of the GML as you mentioned, or non-parametric models. However, I believe that in social sciences we should take up more emphasis in the use of approaches such as historical, hermeneutic or anthropological that could provide us a better understand of social reality in front of mathematical or statistical models which, in the best of cases, explains the 30 percent of a phenomenon. Anyway, both approaches will always be complementary and not mutually exclusive.

This is an interesting discussion.

I believe that studies with non-significant results have a place. A fellow psychology student asked the other day what to do as her findings did not align with previous research or the hypothesis. I advised her to still draft up the findings and then come up with some ideas why here results didn't match the current theory as well as look at limitations of study as mentioned by someone further up.

The big criticism of social sciences especially psychology is that the sample population is not representative of the general population. A lot of the studies are done in Western Universities so the population is mostly European College Students.

Studies that turn up unexpected results, even if those results are not significant may create a new aspect to study and as such do need publication.

Interesting question Dr Muhammad Usman ...FOLLOWING

I submitted in a good impact factor journal. One reviewer suggested a minor revision but the other recommended a rejection. The associate editor rejected it saying that on the basis of the second reviewer's comment, I reject the paper. In fact, there was only one comment. Why two out of your five hypotheses got rejected? The study failed to contribute what it intended to. It happened three days ago. So again thinking of that why three accepted main hypotheses did not matter but two rejected hypotheses mattered and the paper got rejected. Your comments please

Maybe we should review the reason why you test for significance with a hypothesis parameter = 0. It seems to me, you really want to know: does the parameter exist (at all). The purpose of science is to understand which parameters do and do not exist. The exact magnitude is of secondary importance. One of my professors told us that 'everything is related to everything else' but I do not think this means 'everything is related to everything else directly'. Science can only advance if we know which parameters do and do not exist. Journals do not seem to provide us with the correct information at all. @Usman, I think your article should have been published because of the two hypothesis that were rejected. (Assuming the rest would have the necessary qualifications).

Thank you so much Peter Moorer .