I have couple of datasets wherein each dataset has 10000 rows to 40000 rows. All format, columns and data in these dataset are different. Each of these dataset have anomalies (less than 2%). For example, a 24000 element data set has 172 anomalies. While building models for predicting the anomalies (binary classification), we are seeing a lot of errors which may be due to skewness in datasets. Looking for pointers/ approach or methods to handle datasets with skewed data. Thanks in advance for help.
The accurate prediction for the large population of the datasets may greatly provide and offer a number of validations (vsp) and inference statistical problems (isp) and questions. In the onset the accuracy question is the main problem in statistics and indeed a statistical problem. The question which may arise is how are the vsp and isp in hypothesis reveal. The related solution is the samples accuracy to the question. The accuracy may be seasonal, outliers, marginals sample, population distributional. The skewness will centrally for the large numbers anomaly, indicating the data sample or confidence of the statistic used. The shaping of the problem is necessarily not a statistical problem which is data perfectly supported by the statistics. However, the statistical question is the inference to the problem. The formulation of the hypothesis is the complete statistical problem solution. By the virtue of the accuracy of the samples in the population. Hence fourth the unbased/baised samples are contributing to statistical problems with do not have clear hypothesis solutions. In machine learning this problem is commonly related to the multi-variety optimization for the averages and operationally the Lagrange multiplier at various levels of deviations to sample statistics. As the symmetric approach the skewness can validates the anomalies questions relating to the shaping of the question & samples.
The question needs a specification of what algorithms can be considered a machine learning one and what you call skewness in the data.
I would call a "well-designed" machine learning "algorithm", as a set of instructions where the machine can write its own algorithm based on its learning experience. While the word "machine learning" can be broadly defined and sometimes "abused" because it sounds "trendy", I assume that we have the same understanding what specifies as a machine learning "algorithm".
Additionally, I understand skewness as asymmetry in your data. Mathematically speaking, skew is just the centralised third moment of given samples. If this is non-zero, then there is positive or negative skew in your data. It is possible that you probably meant outliers or probably "fat-tailedness".
P.S: Samples as binary classifications are always skewed when the success rate p!=0.5, which is practically almost always the case.
Given these definitions, such algorithm should have no problem dealing with skewed data. For neural networks, I suggest a minimum of three layers (expecting that the machine would analyse the first moment in the first layer, second moment in the second layer, and third moment in the third layer). Other methods should have similar concepts, just allowing the machine to think for another "layer" would probably help.
If the purpose of your algorithm is to find anomalies given a dataset, then the training data is quite small.
- Badges
- Science topic