In one hand, there are limited number of data-points and so many features for the analysis, let’s say 50 data-points and 60 features. On the other hand, the unusual data-points are more important than the usual-points and you want to know the underline facts of happening those unusual data-points. How do you approach the analysis to tackle these two issues at the same time?
Any help would be appreciated.
Thank you David. Don't the robust techniques including M-estimator down-weight the outliers/unusual data-points. However, those points are really important In my case and I don't want to down-weight those (Even, I'm more interested in out-weight those outliers because I'm more concerned about their underline facts about those data-points).
Yes, the M-estimator down weights the influence of the outliers on the results.
To answer your question some more information is needed.
What do you mean by important?
Is the research task to identify unusual points?
Or are these unusual points hindering your analysis?
How big is the share of these points in relation to the total data set?
Hi Jan,
Those are really good questions. Here are my answers,
What do you mean by important? I meant defining the unusual points are really critical for me. The idea is using those features and develop some kind of predictive model to predict those unusual points before happening and try to come up with some contingency plans accordingly.
Is the research task to identify unusual points?
Or are these unusual points hindering your analysis?
The research task is defining the environment that those unusual points are happening and try to predict those.
How big is the share of these points in relation to the total data set? The percentage of unusual points are around 20% (10) and the rest (40) are usual points.
Please let me know if you have any other questions.
Thank you.
Well, that narrows the task very much down.
I would say you could have a look at methods that are applied in predictive maintenance
https://en.wikipedia.org/wiki/Predictive_maintenance
This is a very vivid application field in Data Science. I guess you'll find a lot of information on the web.
- Badges
- Science topic
- Similar topics
- Medicine
- Physiology
- Biosignals
- Biomedical Signal Processing
- Statistical Analysis