@Balázs: I think we are both running out of new things to say here. Sort of a "quibble-off".... I should just let you have the last word, but I can't help one last quibble: it seems to me that an individual theorist can never use your version of The Scientific Method, because the required experiments or fresh observations may not be possible for that one person to perform; in that case a theory becomes "scientific" only after its predictions are thoroughly tested by others, sometimes after the original theorist is deceased. This seems awfully retroactive for a "Method".

I regard my original Question as "answered"; RG should revise its criteria in the Open Review tool! Meanwhile we have raised a number of more philosophically interesting questions; anyone feel like enshrining them in new Question threads?

Well, dear Jess, it can be formulated as a kind of axiom: if an experiment cannot be reproduced it is untenable. This is exactly one of the most solid criteria for distinguishing between science and pseudo-science.

I understand your reaction, but I think comparing reproducibility with "likes" in Facebook is a little bit exaggerate.

Carlos: what do you mean by "untenable" in this context? I agree that this axiom is widely employed, but it is possible for an axiom to be silly, or even wrong (internally inconsistent). I assert that NO experiment can ever be truly reproducible, because the statistical significance of raw data depends, formally and quantitatively, on the assumptions and uncertainties that the experimenter started with. The second time around (or when another experimenter tries to repeat the same experiment), those assumptions and uncertainties are erased, or at least modified dramatically. One can do a second experiment that is (or isn't) CONSISTENT with [the conclusions of the analysis of] the first experiment, but that's not the same thing as reproducing the same experiment -- that's impossible.

Basically an experiment cannot be reproduced, but that is not the issue. If scientific conclusions cannot be confirmed, then science would not exist.The point is that a workable definition of repeatability and of reproducibility needs to be applied. If an experiment or an analysis is repeated, then the results of that second attempt should be within a-priori set limits of the result of the first attempt, e.g. +/- 3%. If an experiment consists of a treated group and a control group, then all experiments should give a significant difference between the groups at an a-priori set limit, e.g. p

I am inclined to think that reproducibility is an attribute of research that involves experimental design. This often reflects the ability to isolate a phenomenon and treat it as a closed system by establishing boundary conditions (initial conditions). Ideally, such conventional practice enables the generalisation of research results.

One important point to remember is that reproducibility means while we hold certain things constant (ceteris paribus), there must be no qualitative variation which mean that a regular conjunction of events can be observed and reproduced.

Dear Colleagues, let me remind you that in some disciplines like astronomy (supernovae), solar-terrestrial science (solar flares, total eclipse), geology, geophysics, natural hazards (landslides, tsunamis, earthquakes, volcanic eruptions, practically anything related), palaeontology, archaeology, etc. reproducibility cannot be provided in the sense as it is used in e.g., physics, chemistry, or biology. In some cases even the statistics cannot be applied (low number of observations, fossils found, etc.). Despite that circumstance the contribution can be still scientific, if the scientific method is applied. Kind regards, Balázs

@Balazs: I was holding back Astronomy as a trump card, but now you've played it, and the discussion stopped abruptly.  I assume that means everyone else has seen the error of their ways, and ResearchGate will soon revise its criteria in the Open Review terms of reference... so now I have only you to quibble with.  You implicitly inserted a new criterion for "Science" -- would you care to define what you mean by "the scientific method"?  :-)

@Balázs

I would like to add meteorology to your list of discipline to emphasise that, indeed reproducibility is rarely possible in these domains specifically because inherent phenomena to these areas cannot be isolated. Thus, systems remain open making their behaviour almost impossible to predict. I do no think that reproducibility can be dissociated from prediction.  

I would also like to add that defining the  "scientific method" is a very challenging task. I do not think anyone knows what is the scientific method!

Dear Jess, dear Mohamed,

When I have written my contribution above I stopped typing for a moment asking myself what would happen if someone asked what scientific method was meant there. It is certainly challenging to define it in short. In spite of that practically everyone believes that he/knows that answer. Certainly if I try to give an educated answer its length would be that of a monograph.

Anyhow, I make a try. In my opinion scientific method is a way to study the environment (the observable world/universe) of the individual (the experimenter) so that the experimenter selects something (certain area/volume/site/period of time and/or a selected phenomenon/context), describes the environmental/experimental setting as good as it is possible (and thought to be relevant at the current level of knowledge) and then he/she describes the outcome of the experiment (here that is the difference between reproducible and non-reproducible experiments, see below) and then we try to analyze the outcome in order to make implications for (a) outcome of other experiments (b) derive a somewhat general rule that the results of the experiment (and the previous experiments of anyone) do not contradict with and useful to foretell in (a) type situations (c) to design and build anything useful (bridge, mobile phone, cooking device) in which we make use the outcome of the experiment(s) that would work as we expect from our experiments and implications (types (a)+(b)) based on them (will hold the traffic crossing through, will transmit our messages/voice, will cook the dish faster/better).

The difference between the reproducible and non-reproducible experiments is IMHO that the experimenter can influence the setting of the experiment or not, or whether the experimenter can trigger the event/phenomenon to be studied or not. Otherwise the mandatory description of the experimental (observational) environment, the description of the observations (following the standards of the scientific branch at the current level of knowledge and "practice") and the way we imply using certain logic/mathematics is the same.

A consequence of this line of thought is that what is meant scientific, will be redefined time by time (example: study of magnetism

Dear Balázs, I never meant to imply that you were trying to shut down the discussion, only that almost everyone else seems to have withdrawn following your irrefutable argument about Astronomy.  Perhaps I was a little jealous.  :-) 

Now, as to "The Scientific Method", you have voiced most of my potential criticisms yourself (upstaging me once again!  ;-)  so I won't go picking it apart, but I will remark that by your definition Einstein did not use The Scientific Method in his 1905 paper on the Special Theory of Relativity; in fact I would say theoreticians are universally disqualified by the terms of reference you describe.  Since I'm an experimentalist, I don't mind a bit; but I don't think it works as a criterion for Science. 

In fact, nothing does!  Every time someone tries to pin down a tidy definition of Science, they do themselves and others a great disservice.  Defining Science is no easier than defining Art; in fact, when you look at the similar problems encountered by such attempts, the wisdom of separating the two in the first place becomes questionable. 

I recently wrote an inflammatory little essay (it probably really should be classified as a Rant) that you might find relevant:  http://jick.net/wordpress/?p=18

In spite of the ultimate futility of trying to explain what Science is, we are all drawn into the discussion because it is so entertaining and illuminating.  Sort of like Physics itself: every theory is ultimately wrong (with the possible exception of Statistical Mechanics)  but some are wrong sooner than others. 

Cheers -- Jess

Dear Jess,

Nice comment, thanks. I would object only one statement:

"but I will remark that by your definition Einstein did not use The Scientific Method in his 1905 paper on the Special Theory of Relativity; in fact I would say theoreticians are universally disqualified by the terms of reference you describe."

I disagree. The "scientific" theories have typically corollaries, logical consequences or even predictions that can be verified by observations (typically later). If I state that the Earth stands on a big turtle, and we launch satellites and they observe the big turtle then my theory was proven scientific. If I assume that sailing westward from Europe I can reach India and I find it, it is a proof of the theory. If later it turns out that I reached not India but something else then the verification was not successful.

In case of Einstein, he formulated a theory as a scientific theory to be proved/disproved later. If I establish a religion that says the Sun is in the center of the Universe that can be later considered as a scientific theory as well, though my religion originally did not claim/need to be scientifically verified. (That anyhow would become obsolete as the next religion suggests that the center of the universe is the center of our galaxy. At this point Isaac Asimov starts to type more quickly :-))

I hope I could explain why theoretic scientists can be considered scientists as well. Phew... :-) That saves my life probably :-)

Thank you again for the nice discussion and please keep posting... Regards, Balázs

Actually Einstein used "thought experiments" to explain previous observations that had until then seemed mutually contradictory when subjected to logic and common sense.  He showed that common sense might be completely wrong.  It was.  I concede that his STR had a number of now-familiar implications that stood the test of experimental verification, but he didn't do those experiments; he did Gedankenexperimenten.  Anyway I should also back off from my hard line on theorists, or my theorist friends will be annoyed....

@Jess: I do not want to stick too much to my text and theory above, but IMHO a Gedankenexperiment is the same as any religion from this point of view (of course it intended to be scientific, no question about): until its verification it is as much as speculative as anything else like UFOs, cold fusion, etc. In science there were many cases when "nonsensical" theory has been presented that even was based on observations.

Let me cite here only one brilliant scientist (just to draw the attention to him), namely Alfred Wegener, the forerunner of the plate tectonics theory (BTW: 2015 is the centennial of the first publication of his seminal book). He put together a puzzle of observations very tediously that showed that the continents had to be arranged very differently than today. At the time of its publication and in the next decades this theory was considered speculative (despite the observations that it could explain and the predictions he made) and most of the scientist were convinced that it is essentially wrong because it is impossible. Until ca. the 1950s it was not possible to prove it. However, the times changed gradually even though, as Rudolf Trümpy (a great alpine geologist) noticed in 1999 in his talk, in the 1940s and 1950s three manuscripts stating very similar theories to plate tectonics still had been rejected by scientific journals.

Of course, if you figure out something that does not fit to the current canonical theories, even it explains the observations better than the previous ones, you will have a hard time to achieve its acceptance. I still think that these concepts, like Wegener's work cited above were and remain scientific because they were based on observations and some solid theoretical background (well, retrospectively it is easy to judge). Regards, Balázs

@Balázs: I think we are both running out of new things to say here. Sort of a "quibble-off".... I should just let you have the last word, but I can't help one last quibble: it seems to me that an individual theorist can never use your version of The Scientific Method, because the required experiments or fresh observations may not be possible for that one person to perform; in that case a theory becomes "scientific" only after its predictions are thoroughly tested by others, sometimes after the original theorist is deceased. This seems awfully retroactive for a "Method".

I regard my original Question as "answered"; RG should revise its criteria in the Open Review tool! Meanwhile we have raised a number of more philosophically interesting questions; anyone feel like enshrining them in new Question threads?