I want to train a machine learning algorithm able to learn from very few number of examples (less than 100). This is a regression problem with 10 categoric inputs (each between 2 and 10 categories). Any suggestion aside the classical dimensionality reduction approaches?
You may want to try tree-based approaches. You can simply start with Classification and Regression Trees (CART), however there are certain problems associated with some of the tree-based approaches like your predictions are restricted with the range of your input data. This is not the case for all the tree learners.
I would suggest gradient boosted trees (especially stochastic gradient boosted trees -> http://astro.temple.edu/~msobel/courses_files/StochasticBoosting(gradient).pdf). You can handle categorical predictors with tree-based learners. Missing values are allowed. As a side product, a feature importance value is evaluated for each feature. They are good at handling interaction effects and so on.
Be careful about the number of levels of your categorical predictors. Some tree-based approaches are biased towards the features with larger number of levels when building the trees.
You may want to try tree-based approaches. You can simply start with Classification and Regression Trees (CART), however there are certain problems associated with some of the tree-based approaches like your predictions are restricted with the range of your input data. This is not the case for all the tree learners.
I would suggest gradient boosted trees (especially stochastic gradient boosted trees -> http://astro.temple.edu/~msobel/courses_files/StochasticBoosting(gradient).pdf). You can handle categorical predictors with tree-based learners. Missing values are allowed. As a side product, a feature importance value is evaluated for each feature. They are good at handling interaction effects and so on.
Be careful about the number of levels of your categorical predictors. Some tree-based approaches are biased towards the features with larger number of levels when building the trees.
I think decision tree approach will be reasonable. You can use demo version of See5 from www.rulequest.com
Border pairs method can analyze data before learning proces to find out how difficult they are. This method always found a solution and is resistant against the overfitting and is working well with small number of learning patterns. Paper with description is on my RG page.
If you can choose which training data to learn, try active learning techniques to get the most out of your
If existing techniques to learn from your small dataset dose not work, you can use semi supervised approaches. First over sample your data and then use semi supervised approaches.
My experience is that there is not much you can do without a feature selection (dimensionality reduction) phase. Data are too sparse and any result would be too noisy... it however depends on the detailed nature of the data and of the problem.
You need to use a very simple model, such as the best univariate model, or a mixture of univariate models.
You can try the MODL algorithm, which regroup modalities of categorical attributes (and discretize continuous attributes) using a MDL criteria, and then mix the obtained best univariate predictors using a selective naive bayes.
You can try the software here:
www.khiops.com
Search for one-sample problems and small sample problems.. Its a topic of interest to many people like me :) .. More than dimensionality reduction, probably the question is how robust the set of features you are able to obtain from a given training examplers are to natural inter-sample variability.
For training, decision trees may be the best solution, but all other proposed algorithms could equally be applied.
What you really need is to test all these techniques(or any other), and estimate the generalization error. For validating different algorithms and estimating the generalization error, consider the following procedure:
A) generating bootstrap samples, that is one of the best approaches when only a few samples is available for training.
B) estimating the error using some validation technique in a loop and repeat many times, as follows:
for i=1 to 1000 (the higher, the best)
shuffle dataset
divide dataset into training and test sets (90-10% for example)
train your model with some technique with training set
determine mean error in test set
accumulate error
end
error = error/1000
C) Apply this technique to any technique, to estimate its generalization error.
D) Select the best technique, depending on this value.
- Badges
- Science topic