How do we evaluate the importance of individual features for a specific property using ML algorithms (say using GBR) and construct an optimal features set for our problem.
image taken from: 10.1038/s41467-018-05761-w
It all depends on what the endpoint is, the one you wish to analyze or evaluate. Linear regression analysis is nice, if you have a linear process.
This classical model-structure estimation problem has been solved, for instance, in
@incollection{KarKul:88,
author={M. K\'{a}rn\'{y} and R. Kulhav\'{y}},
title={Structure determination of regression-type models for adaptive
prediction and control},
year=1988,
booktitle={{B}ayesian Analysis of Time Series and Dynamic
Models},
editor={J.C. Spall},
publisher={Marcel Dekker},
address={N.Y. },
}
You can do it in many ways. PCA is a nice way to gather important parameters. Another way would be to train multiple models with and without specific features and see how that will influence error. Correlations can also help. However, in most cases you need to use your head and see what parameters, why and how are effecting your results. In some cases ANOVA is a nice technique but only if you think and not blindly thrust in results. For example, speed in metres and speed in centimetres are both just speed so using one of them is enough. I know that was stupid example, but it shows the point. Know your data, analyse what impacts results and you will do great. Good luck, hope it will help even a bit.
Andrius Ambrutis Indeed it is helpful. Thank you for the insights you provided on the topic.