Dear colleague

I just read your question. The situation is not such the one you describe. Always such a shadow of things is preceded by the principle-philosophical commitment of the author, as an it what should he do and what he expects.

Experimenters should always be designed in advance. The scientific contribution consists in the design of the specificity of the experiment, the examination of the results of the experiment and the quality interpretation of the obtained results. The quantitative results of the experiment are almost useless without a creative quality analysis of the outcome. Establishing connections between used parameters, understanding and accepting detected links should be a significant shift in the perception of the problem being investigated. The use of statistical tools should not be a purpose for itself, nor is it evidence of anything. The results obtained just justify the researcher's expectations that he is on the right track of solving the problem

The image processing and computer vision community has had this problem almost from its beginning. Many papers could not be reproduced by other teams which made comparison very difficult. The Image Processing On Line (IPOL) journal addressed this issue head on in the following paper:

http://mcolom.perso.math.cnrs.fr/download/articles/RCBR2015.pdf

If your question is more philosophical in the sense "can we truly trust software", there is always a piece of software on which you rely that you haven't written/tested/understood (just think of your OS, mathematical functions in your language of choice, etc.). Also, know that except for VERY simple systems, you cannot guarantee that a running software is bug free (just think that you also rely on the OS which isn't bug free).

In that sense, computer vision is as much an applied mathematics as an experimental science like biochemistry where very small differences in pH, temperature, etc, could sometimes change the experiment result radically! And yet, we are making drugs to fight cancer! The same is true about much of the scientific research using software...

To cite one instance, I got the opportunity to listen to a research presentation. The significance of one of his major results was tested statistically using a software application. I had to ask if how I can rely that the assumptions required for using the test was really embedded in the source code of the software. I only got an answer that in many literature, the statistical test he used through the software was also performed. Do you think that answer is justifiable enough?

Hello Eugenio,

I think many scientists are not familiar enough with the underlying assumptions on which their tests are based and many just "do as others do" and that the answer you got just reflect this...

When you say "I had to ask if how I can rely that the assumptions required for using the test was really embedded in the source code of the software"

I would rather say that the assumptions required for using the test must be satisfied by the data rather than the software. This needs to be checked by the scientist using other statistical tests implemented by other algorithms in the software.

Regards,

Bruno

Hi Bruno!

Well, yes, I agree. Primarily, it should be assured that the assumptions for using the test must be satisfied by the data. Then, if so, there should be some means to make sure that the scientist knows what the software really does to his data in detail so as to make sure that he gets the desired result with reliability.

Cheers,

Cedric

Yeah those research articles normally contains the algorithms on how the software works, and also the type of software used and why the software is needed for the research results.

Regards

" Then, if so, there should be some means to make sure that the scientist knows what the software really does to his data in detail so as to make sure that he gets the desired result with reliability."

On the other hand, besides torturing and inquiring what the scientist really knows, at one point, you just have to trust people or show them wrong!