hi
you can measure the correlation between the feature-set selected or even each one of features with the corresponding label of each instance in dataset. some methods available are gini-score, fisher discriminant ratio, relief, ...
you can use each of these measures or other ones based on your requirement. it's important to make a reliable model with the selected feature. if you want to write a paper you should search for evaluation metrics used by other researchers. good luck
Waqar Khan ,
First of all, you do not Need to carry out a feature sensitivity analysis. It is not a requirement, but just an added bonus.
Secondly, in most applications, Feature Selection (via feature importances) and Feature Sensitivity are similar and complementary. Adding feature sensitivity after feature selection seems to be overkill. (More on this later)
While there are many different algorithms for estimating the sensitivity of a model on a set of features, I believe that you may be relatively new to Machine learning and suggest that you start with a couple of simple approaches:
a) How sensitive is the model to changes in the value of a feature? The classical way to carry this out is to shuffle or perturb the values of feature and see the magnitude of change in the model prediction.
b) How is the model affected if a certain feature's measurement is missing?
These 2 approaches would be a good point to start with sensitivity analysis. In case you are short on time and need to get results quickly, there are good libraries that can carry out sensitivity analysis for you. For instance, the pytolemaic package can do sensitivity analysis for most python ML models and generates pretty good reports, along with figures automatically (https://pypi.org/project/pytolemaic/).
Aashwin Mishra Dear Sir,
So many thanks to you. I will try the library as you mentioned in python and try it. second my code is in Matlab, so any suggestions for Matlab too?
Waqar Khan,
Let me divide my response into 2 parts:
a) Feature Sensitivity in Matlab: The Machine Learning toolbox in Matlab has a Sensitivity Analyzer feature that you can use. However, it has limited functionality and works on a limited set of models. (https://www.mathworks.com/help/sldo/ug/what-is-sensitivity-analysis.html)
b) Recommendation: May I humbly recommend you to consider moving your workflow over to python, etc if you plan to perform Machine Learning research for a few years? Matlab is run by the MathWorks corporation, and its development etc is centrally planned. So most of the packages/toolboxes that you will find in Matlab will be limited based on how profitable developing them may be. This will limit your choices. In an open source environment, like while using python ML tools, individual users can develop their own packages and share them. Thus, there is a better chance of finding good packages that suit your individual needs. This is just a friendly suggestion, think about it.
What ever choice you make, I wish you the best of success.
PS- If you found my answers to be relevant and helpful, may I ask you to recommend them? That way, other researchers will be able to find them easily in the future.
Aashwin Mishra Thank you so much. Let me try harder on this. I will be keep in touch for further assistance.
Waqar Khan
No problem, Feel free to send me a direct message any time you need help.
Saeed Sarbaziazad Thank you so much sir for your nice answer. I will try on it one by one.
Aashwin Mishra Sir I have one question, as per your first answer
"Adding feature sensitivity after feature selection seems to be overkill"
I have already got selected features from the model. For this if I do sensitivity analysis, Its not good way right? or if yes then follow the way what you suggested in python right Sir? Please guide me about this.
Saeed Sarbaziazad Sir, Is Gini-score is the sensitivity analysis right? Could you please answer me. Thank you in advance.
I dont know how you carried out your feature selection. Bereft of that knowledge, I'd recommend you use the sensitivity analysis to select the feature so that you end up with a minimal, robust model.
Waqar Khan hi. unfortunately I cant understand what do you mean. sorry
but gini index is one of the feature ranking methods. you can rank features based on the score retrived by gini score. you can finally evaluate the selected features performance by creating a model with new features and calculating some evaluation metrics such as accuracy, precision, recall, sensitivity, specificity and any other metric that other papers have measured and you want to compare your results with them.
good luck
Hello Waqar Khan
I think it is a good idea to apply a sensitivity analysis after feature selection to measure the quality of the selected features.
Pytolemaic is a library that you can easily use to report the sensitivity analysis.
Here is the link: https://pypi.org/project/pytolemaic/
Meriem Ferdjouni Thank you so much for sharing your suggestions and Link. I will following your suggestions. Thank you once again.
- Badges
- Science topic