I recently, with co-authors David Bowes and Tracy Hall, conducted a meta-analysis of 600 results from a number of software defect prediction studies:
https://www.researchgate.net/publication/262003721_Researcher_Bias_The_Use_of_Machine_Learning_in_Software_Defect_Prediction
Which is published in IEEE TSE 40(6) pp603-616.
We tried to understand what factors impact prediction performance. Worryingly the dominant factor is neither the choice of algorithm (eg random forest, etc) nor the data set but actually the research group that conducts the study.
I do not believe that researchers are intentionally distorting their results but it's well known that scientists have a preference for some results over others [1-3].
One remedy we suggest is blind analysis. This entails masking the treatments from the analyst, usually by re-labelling to something uniformative.
So my question is:
Should blind analysis be a norm in empirical software engineering?
Thanks for your ideas and thoughts
Martin Shepperd
References:
[1] R. Rosenthal, "On the social psychology of the psychological experiment: the experimenter's hypothesis as unintended determinant of experimental results," American Scientist, vol. 51, pp268-283, 1963.
[2] J. Sherwood and M. Nataupsky, " Predicting the Conclusions of Negro-White Intelligence from Biographical Characteristics of the Investigator," J. of Personality & Social Psychology, vol. 8, pp53-58, 1968.
[3] K. Dickersin, "The existence of publication bias and risk factors for its occurrence," J. Am. Med. Assoc., vol. 263, pp1385-1389, 1990.
Article Researcher Bias: The Use of Machine Learning in Software Def...
The question is rather specific, but the problem is very interesting.
To make a point:
1) The problem is related to fault detection based on features of software components. It can be considered a binary classification problem (a fault is present or not)
2) The paper analyzes the work performed by 23 groups of researchers. They applied various classifiers and compared their results (probably to show that one of the choices outperforms the others)
3) A number of datasets describing features of software components were used, however, most of them (80%) originated from NASA and Eclipse.
4) The conclusion is that the quality of results (detection capabilities) depends the most on a researcher group, whereas the influence of the classifier or a dataset used is weak.
Datasets from NASA provide features that can be calculated automatically from the code, there is little room for human intervention. I haven't seen the other.
Personally, I think that this somehow surprising result is not caused by biases of researchers. I would rather relate it to preprocessing steps, dimensionality reduction and maybe various software packages applied in analyses.
Blind analysis or relabeling can be used to avoid dimensionality reduction based on hunch, i.e. to manually remove features, which seem redundant (but they aren't). Are there any other applications?
Hi Piotr,
Thanks for your thoughtful response.
I agree pre-processing can be rather important and it's often not well documented. If research groups have particular expertise or preferred pre-processing methods then it is quite possible research group will act as a proxy for this. Moreover this effect may not be limited to just pre-processing. Many of the machine learning techniques are quite sophisticated and there are many subtleties. Consequently some groups may simply be more skilled at applying some methods than others.
However I'm not sure I fully agree with your statement that "there is little room for room for human intervention". In my experience these experiments generate many results and only some are reported. Also there are a wide range of statistical procedures for evaluating the meaning of the results (in terms of statistical significance and/or effect size).
My thinking is blind analysis cannot do any harm and it may militate against conscious / unconscious preference for particular results eg that the researchers new technique for learning a classifier is 'better' than competing approaches.
Best wishes - Martin
Hello Martin,
I would rather look for a business use case and focus on an organization developing software, which may wish to apply machine learning techniques to predict defects or assess risk of not meeting quality criteria in its products.
I feel that in large organizations, running sometimes hundreds of projects per year, such process should be managed. Its activities comprise data collection, building models, performing automated analyses, validating, reporting, benchmarking, etc. It is a general idea, the whole process should be detailed and divided into design, implementation and maintenance (inluding regulatr evaluation) phases.
Blind analyses can be considered a good practice within this process (especially related to the design and evaluation). However, it seems to me that ongoing prediction or assessment should be fully automated.
Best regards
Piotr
Hi Piotr,
I think your response teases out an issue that I should have made explicit (so thank you). I was really only addressing researchers (in a lab or university setting) rather than practitioners who in some sense might be considered to be doing research.
Generally the goal of software engineering research is to discover new knowledge to help the practice of software engineering. So in the setting of defect prediction it might be to provide guidance as to which learning techniques, for a particular context, might be most appropriate for finding prediction systems. As you put it, which procedures can/should be automated follow from the results of the research. I'm concerned about bias in the initial research which lead to confusing, contradictory or plain misleading guidance to practitioners.
Best wishes - Martin
- Badges
- Science topic