I am new in data analysis and trying to identify and delete outliers from original data when doing T-squares analysis.
I have 150 data in a vector(150 rows and 1 col) which have 10 outliers
I would like to remove the outliers from these data in order to prepare that
for ANN learning,
So, I analyze this data by T-squares method and plot of this method show outliers in the normalized data obviously,
but I would like to have a way to mathematically delete these detected outliers from original data (not normalized data)
any suggestion or ideas to aid me in doing that is greatly appreciated.
Thanks in advance for your help.
Dear Nefas,
If you select the boxplots you will find the outliers. In the figure (graph) the numbers identified with * correspond to the cases with severe outliers and you can delete them from your database. In addiction you can also eliminate the cases identified with o, which represent moderate outliers.
good luck with your research!
Dear Nefas,
Actually, identifying outliers is easy. May be using three sigma rules is the simplest procedure, but I think deleting outliers is not practical option, The best way is wieghted your data by using bisquare function to give down weights to outliers. In this case the outliers are multiplied by zero weight. Deletion observations may lead to bias estimation, but weighting the outlying observations by zero can not lead to bias . Good luck
@Hassan Uraibi
Many thanks for your idea, down-weighting outliers can help me
You welcome. Please, let me see your data first. Then I will give you my suggestion.
@Hassan Uraibi
Sure, If it is possible, give me your email, I will send my data and T-square analysis result for you.
Thank you.
The attached paper may be helpful. The psi functions in this paper could be replaced by the bisquare as Hassan suggests. To use the program with your data simply use a regression model that only has a constant term in it. For the analysis using R, see the link below. Best wishes, David
https://www.amazon.com/Modern-Applied-Statistics-S-4TH/dp/B0042NJY80/ref=sr_1_1?ie=UTF8&qid=1478464377&sr=8-1&keywords=modern+applied+statistics+with+s.+fourth+edition
- Badges
- Science topic
- Similar topics
- Mathematics
- Algebra
- Matrix