p-value can be used to compare various machine/deep learning models against a standard to see which one(s) have statistically similar predictive performance at a given significance level (usually 5%), especially if you are using n-fold cross-validation (CV). Let's say you are comparing the performance of SVM and Näive Bayes classifiers against the XGBoost classifier. You already determined the AUCs of each model via a 10-fold CV. Instead of just getting the average of each AUC, you can use the values for each model to obtain the model's respective p-value against the standard in a pair-wise manner. That way you can check for statistical significance.

For more clarification, look up my article at https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9641908/

Superb Emmanuel Onah . I like this use-case of p-value.

Though my question was more tailored to a situation where my machine learning model is already decided, and I just want to understand which features to keep and which ones to reject as I have a huge number of features.

P-value is a certain type of a significance result, from some statistical test. So, can you use a p-value for feature selection depends on what statistical test are you going to use for feature selection.

You should select the features that are causally linked to the desired output. Statistical tests cannot achieve this understanding for you.

Also, p-value assumes a certain statistical distribution; if that assumption is invalid then the p-value can be meaningless. P-values are very useful when you know the causal feature to desired output relationship and you're just trying to discriminate between signal and noise.

I like this answer Dan Harris . How do we understand causal linkage of features to desired output ? Please name some known methods.

The only valid causal discovery method I am aware of is captured here (I really enjoyed reading this book): https://www.amazon.com/Logical-Leap-Induction-Physics/dp/0451230051

PS: oh and you can use ML techniques to narrow down the potentially causally informative features (the one's that do not correlate to desired output can be ruled out as non-causal or non-informative), which saves you a ton of time.

cool thanks Dan Harris