If we train a data model once on a dataset using a machine learning algorithm, save the model, and then train it again using the same algorithm and the same dataset and data ordering, will the first model be the same as the second?
I would propose a classification of ml algorithms based on their "determinism"
in this respect. On the one extreme we would have:
(i) those which always produce an identical model when trained from the same dataset with the records presented in the same order and on the other end we would have:
(ii) those which produce a different model each time with a very high variability.
Two reasons for why a resulting model varies could be (a) in the machine learning algorithm itself there could be a random walk somewhere, or (b) a sampling of a probability distribution to assign a component of an optimization function. More examples would be welcome !
Also, it would be great to do an inventory of the main ML algorithms based on their "stability" with respect to retraining under the same conditions (i.e. same data in same order). E.g. decision tree induction vs support vector vs neural networks. Any suggestions of an initial list and ranking would be great !
See. http://machinelearningmastery.com/a-tour-of-machine-learning-algorithms/
for quite a comprehensive list of methods.
There is an element of chance in the training process. In some software, you can get reproducible answers by using something like set.seed( ) in the R language. Using the seed number again with the same data will then give the same result. Then you can report the software you used with the seed. However in general the different outcomes will be close together, but as with sampling, you will occasionally get outliers (depending on the seed you choose).
There is an element of chance in the training process. In some software, you can get reproducible answers by using something like set.seed( ) in the R language. Using the seed number again with the same data will then give the same result. Then you can report the software you used with the seed. However in general the different outcomes will be close together, but as with sampling, you will occasionally get outliers (depending on the seed you choose).
A believe that depends on the inference algorithm. If you are using some convex objective function with a unique maximum, probably you are going to always have the same results. However, if you are using some MCMC or variational algorithm for Bayesian optimization with a highly constrained support on the model parameters, probably this may lead to small difference if the model is well specified, or big differences if not. Random forest or any other model selection technique tend to always lead to different results, given the nature of the curse of dimensionality under a wide model space.
If the answer is helpful, please recommend for others,
Best,
The stability criterion is not relevant (at all) to the ability of an ML technique to produce predictive model. The reasons of the apparent model instability are in the data. In particular, insufficient sampling of latent data structures and intercorrelations among independent variables are most probable causes. When one deals with such imperfect data sets (and data sets are imperfect in most areas of human activity that really matter), the deterministic ML techniques do not solve the instability problem. They just "sweep it under the rug". In fact, even if, for some reasons, one is about to use such a "stable" ML algorithm, he/she must be looking for ways to perturb it (by, for example, randomly removing a small number of samples and rebuilding the model) in order to test how bad the data are. There are established and well-documented ways of such model perturbation and using it to build more stable and predictive models, such as cross-validation, bootstrapping or y-randomization.
Thanks to all for your replies. Actually I was looking for aspects in the algorithm itself, rather than the data, but Dmitri has a valid point that the data complexity can of course influence the result if it has multiple local solutions. Also, Jhonathan is right in mentioning the convexity (or not) of the solution space.
- Badges
- Science topic
- Similar topics
- Computer Science
- Data Mining