We know different random seeds lead to big classifier performance difference. Let us learn from RG colleagues

Take note of the random seed to avoid variation in your results and during the process of reproducing the results of your experiments. You should also take note of the parameters configuration of your machine learning algorithm if you have replicability of your results in mind either by you or other researchers who may be interested in your results. All the best.

Interesting question. I would recommend recording the random seed for future reference just in case someone else would like to replicate the results. Or, if I get a very different results from another experiment after I change the value of one of the parameters (cluster centres, dilation width, etc), I can then set the random seed of that experiment to the same value, and determine if the difference in result is caused by the change in the value of the parameter, or change in the value of the random seed.

Another aspect that you should consider is that once you change the random seed, if you are using the splitting method (70% training set and 30% test set), the results should not  be different significantly. In the case of splitting, you can change the seed randomly, and see how your methods operate in terms of different training set and test set. If the results are quite similar, it show that your model is prone to changes in test set and it can still operate well if test set would change.

Thank you all for the answers.

I'm following your recommendations and setting specific random seeds for replication, but always testing the model's robustness by exposing it to complete randomness.

Disclaimer: I am user and main developer of the framework pySPACE and a big fan of fixing the random seed. I might be biased. Here, are my reasons why I think, fixing the random seed is important:

1. We often do the splitting of the data into training, validation, and test data at run time. If you compare different hyperparameters or algorithms, which is the main task of the software, data splitting should be the same among all parallel executed evaluations because we want to measure the differences between the parameters of interest and not the different data splittings. Note that in some cases you have time dependencies in the data. In that case you shouldn't have a random splitting of the data but a time-dependent splitting.

2. For controlling the effect of randomness, we repeat the experiment several times and use the run number as a random seed. This gives you a controlled way of sufficient randomization. The more random behavior and components you have, the more runs you should test. Nevertheless, each run with the same run number will have the same pseudo-random behaviour.

3. A main goal of the software is to provide reproducible results. When my colleagues use the same data and processing configuration, I want them to get the same results. When I repeat my experiment, I also want to get the same result.

4. When working with deep learning, the control of the randomness is crucial because it has a large effect.

http://pyspace.github.io/pyspace/