Don't know about machine learning. If you're looking for patterns with not too many attributes (but theoretically you can use automatic nested permutated low-D patterns) you should look for patterns at data acquisition. Many if not most of the burning questions of NPsy are of very low dimension, but at the same time are dimensional, i.e. tehy are "tangible objects", not "clay" - and therefore can expected to require a comprehensive, clean direct approach, where alternative concatenated paths among attributes do not neutralize each other. After a scare for the "cures of dimensionality" - a combinatorial dichotomic 12-D-grid will have 2exp12 (= 4096) hyperboxes. ML will only bring you back some surface patterns, it assumes relevant data to coalesce within islands of low-D, and that there are local phenomena of smoothness (Bishop, 2006, p.37).. ... buona continuazione! goffredo

PS my propaganda-leaflet for configural frequency analysis and some pertaining literature are attached.

With your data set size your choices are somewhat limited. I would investigate bootstrap cross validation. See Harrell, Regression Modelling Strategies, Springer. Harrell has some nice R packages that do this. See the chapters on Logistic Regression. Best wishes.

Hello, Massimiliano,

That sounds like a "Who was first?" problem :-)

First of all: Indeed, 50 cases in the training set seems a bit less to train a ml model on. Whatever learning method you are using, the risk of overfitting is major.

Having that said: In my opinion the best practice is always build the "best" model first, then use it however you like, not inverse. In the end, you want to filter on the best predictions possible, not on poor results.

Model first - find the best predictors. Clean the data, prepare the variables (select only those worth keeping, drop the useless, drop the bias-potential ones, normalize - if the case -, build some new ones - if needed etc. Choose an algorithm and experiment: Features selection, Train, Test, Evaluate. Repeat.

Cross-validation is the best choice in your case (hold-out is out of discussion) - maybe you may want to try stratified k-fold cv instead? I recommend it, as simpler, more intuitive and better implemented in many packages. (cv can be used, in some cases, for parameter tuning as well - it's useful for trees -, see caret package in R, look for the 'cp' - complexity parameter - value).

For logistic regression and trees AUC is a good tool in search for a good generalized model. Use the confusion matrix as well to look for the errors.

Once you have a good model, you may play however you like with different trade-offs between error types. The ROC curve is the weapon of choice for picking/tuning the threshold value to  of a (probabilistic) prediction. As a rule of thumb, a low threshold will ensure a high "hit rate" (i.e. sensitivity), however it comes together with a low specificity. Keeping a recall of 1 or close means a really low specificity, but once you have that already fixed, as you decided, looks simpler: just apply the threshold to filter on the predictions.

Heads-up:

- Using cv or random split to evaluate the feature selection or parameter tuning effects on a model, keep in mind to always set/keep the random seed constant!

- Evaluation only on cv can be misleading on small datasets. On the other hand, in time, new data fed can introduce systematic differences between new and original data, so the model should be continuously updated.

- do not forget to always feed the model with exactly the same type of variables! ;-)

Good luck!

ps I don't find recommendable/useful any meta-algorithm (boosting, stacking, whatever) for small datasets.

Thank you Cristian for you advices! I'm right on track with what you wrote!

I know that 50 cases is a very little sample, but unfortunately that's what in the medical field it's usually available as a first step. If the proof-of-concepts is good, new data will become the test set first and new training cases later.

With this little sample and the computational resources I have, LPOCV is more than feasible. That's why I went for it.

After a careful phase of feature extraction, engineering and selection, I've almost closed the model with nearly ten predictors. And results are very promising (AUC> 0.9; if recall=1, I get spec =0.67). I do know that the only the test set will give a true performance metric but I think that with such results it's worth to keep on.

I actually achieved these results with extreme gradient boosting (xgbTree package in R), slighty better than with random forest and SVM. Do you think that meta-algorithms may be biased (aka will poorly generalized) if trained with such little sample?

Thank you also Gottfried and David for your suggestions!

I am very glad to see your results are so promising! 67% specificity at 100% sensitivity with 90% AUC?! Wow! That ROC curve should not look very curvy at all. :-) You have excellent separation there and solid ground to move on, good luck with this! And with the new data, either.

Regarding the ensemble learning in general and boosting in particular. It's my opinion that training on the edge (ie really small datasets) is already risky, 'boosting' on the risk might be riskier :-D

xgboost in R is basically a regression based algorithm, using linear /logistic regression as a solver/classifier (tree learning) respectively. It iteratively optimize ('boost') on the gradient descent function used to find the global minimum of the cost function on the regression parameters.

As I can see it's highly customizable - which is good, but the quality of the model might be sensitive to internal tuning - and one can monitor the learning progress - which is even better.

Btw, I'm just curious, how logistic regression performed on your set?! It's a simple yet very powerful and trustful tool. (Coming to regression, always normalize for fast convergence - not a problem with "normal" size datasets, a headache with really large ones).

best,

C.

Yes, I tried that and the AUC gets round 0.79 (with a subsample of the selected predictors gets 0.81). To reduce the variance of the model, I think I will choose the best model according to logistic regression (I don't think I can get very low bias with such a little sample...) and than try to push just this model at the maximum with bagging/boosting/etc. Sounds fair enough?

Many many thanks!

If you want to diagnose it's much better to test configurations of symptoms (and/or markers) for their over- or underoccurence comparing between already diagnosed groups (e.g. unipolar depression prototypes).

See www.biposuisse.ch/hd18 (Heidelberg 2018) or specifically slides 221 to 228 (talk si in German, but tabels are not).

www.biposuisse/hd18/konfigurations-analyse-g-lienert/view

P.S. Emerging developments in ML allowing for the integration of patterns are complex-variable- or event quaternion (in R) variable-ML (Igor and N. Aizenberg 1971)