Hello,
I need to perform a One-Class Classification (OCC) task, but the training data does not fit on my memory. I was using sklearn's OneClassSVM and when I realized that I would need online training, I noticed that this class does not offer a partial_fit method. Looking for an alternative, I found SGDClassifier, which offers such method, but cannot be used for OCC tasks, as far as I know. I also found that sklearn does not support HDF5, which could be an alternative to this problem.
Which Python's lib should I use for OCC with support to HDF5 would you most recommend? If there is no such alternative, which lib should I use to perform OCC tasks with online training, not necessarily supporting HDF5?
Thank you!
Marcelo Gomes Pereira de Lacerda Firstly, have you sampled the data and built a model? what was its accuracy? is its low then, resample, train and calculate accuracy.
If this modeling is just a onetime thing, then I would suggest you to use google cloud computing (GCP) or amazon AWS. They give you free credits and can use it for training your model few time so that you can overcome your memory issue. If this is an ongoing process then I would suggest read this
https://machinelearningmastery.com/save-load-machine-learning-models-python-scikit-learn/
https://files.pythonhosted.org/packages/9c/d4/edfeb1dde0b1e5d465afa38724359a61d425aacb5929dfbf075d24ada85a/h5py-2.8.0-cp27-cp27m-win_amd64.whl
http://scikit-learn.org/stable/auto_examples/svm/plot_oneclass.html
My suggestion is to get a random subset of samples that fit in memory and apply one-class SVM on top of that. If you repeat the process for several times, you will obtain multiple one-class SVMs. Then, you can apply a voting scheme to determine if a sample is outlier or not.
Good idea Radu Tudor Ionescu . Thanks a lot! I am gonna test it and give some feedback later.
- Badges
- Science topic