Text classification task, if data quantity is low but data quality is not low. We could use data augment methods for improvement.
But the situation is that data quantity is not low and data quality is low. (noise in the labels, or training data accuracy low)
The way I get the low quality data is by unsupervised methods or rule-based methods. In detail, I deal with a multi-label classification task. First I crawl web page such as wiki and use regex-based rule to mark the label. The model input is the wiki title and the model output is the rule-matched labels from wiki content.
The low quality data may be noisy and untrained as you have written such data could be reprocessed but if data is irrelevant , out of context and not explained the problem and requirement then the data would be futile . The unsupervised method or rule based method could help by the understanding the new problem and the rules those would help to take decisions .
The best practice is data cleaning, consider for example PCA, LDA, CNN etc.
First thing is need to choose the proper dateset.If the data quality is low ,data p reprocessing is required . Otherwise you can go through the Reinforcement learning if it is suitable for your idea.
Dear Tong Guo, before trying to apply any kind of technique, you should address a full exploratory data analysis to identify the four data fundamental problems (Noise, Consistency, Outlier, Missing Values).
Be careful with any intent to fill missing values because it could produce a bias.
Try to identify the white noise to discriminate noise from Causal-based mistakes and other variations (for example, possibly derived from human mistakes during the data collecting).
To analyze the experimental design associated with the process in which the data are coming from. The use of multivariate techniques to analyze data and transform them avoiding bias is expected (e.g. Correspondence analysis).
Data quality is the key to a good model. Even when you have an excellent algorithm, when garbage in, garbage out. Be careful in this aspect.
Remember that the data is not important themselves, the data is important based on their meaning. When data are not representative of a given situation, the problem possibly would be associated with the data origin, how they have been obtained, or how that measured aspects were understood.
As you said that data quality is low, you need to do data preprocessing to improve the quality of dataset.
The ensemble methods are used to improve the accuracy, but they will not help to improve the quality of dataset.
We need to conduct a pilot study. In this study, it is necessary to synthesize models, and then identify the objects of the training sample in the most reliable of these models. After that, you need to delete the objects in the training sample that resulted in false negative decisions from the training sample or create your own classes for them. Continue working with the training sample that was obtained. there are other options. For example, you can remove from the training sample those objects for which the confidence of identification with the classes to which they belong was below a certain threshold.
Data is what it is. I have seen many biased "cleaning' approaches that do not really work, except for favoring good results. If you have missing values or inconsistent values for functional features, for example, try to use embeddings as a pre-processing step in order to have a uniform representation for each case. Otherwise, I would say try to use your data as it is as much as possible (doing the basic pre-processing only).
The low quality data may be noisy and untrained as you have written such data could be reprocessed but if data is irrelevant , out of context and not explained the problem and requirement then the data would be futile . The unsupervised method or rule based method could help by the understanding the new problem and the rules those would help to take decisions .
If your data quantity is low ..then there is a possibility of over fitting your training data. ..so there are different possible solutions to get rid from it. .
1) increase the training data. ..
2) apply regularization ( best approach)...but it may induce little bias to yr model but it can be resolved using larger network model.
3) try to apply different neural network architecture. ..
Among the above three I prefer regularisation to overcome the problem.
Dear Tong,
Guaranteeing the quality of data is indeed fundamental, given that it may compromise algorithms’ performance and reliability. Frequent data-quality problems are:
- Inconsistency or Duplicates
- Noise or Outliers
- False Predictors or Information Leakers
- Lack of Density/Small Data
- Dataset Shift
- Missing Data
- Imbalanced Data
- Small Disjuncts
- Class Overlap
You may investigate on these topics. An interesting line of research is the characterization of data characteristics, which might give you an idea on what type of preprocessing your dataset needs! Since you mention data augmentation, there are also certain methods (oversampling) that take some of these issues into account. Some helpful references:
Lorena, A. C., Garcia, L. P., Lehmann, J., Souto, M. C., & Ho, T. K. (2019). How Complex is your classification problem? A survey on measuring classification complexity. ACM Computing Surveys (CSUR), 52(5), 1-34.
Das, S., Datta, S., & Chaudhuri, B. B. (2018). Handling data irregularities in classification: Foundations, trends, and future challenges. Pattern Recognition, 81, 674-693.
Santos, M. S., Soares, J. P., Abreu, P. H., Araujo, H., & Santos, J. (2018). Cross-validation for imbalanced datasets: Avoiding overoptimistic and overfitting approaches [research frontier]. ieee ComputatioNal iNtelligeNCe magaziNe, 13(4), 59-76.
Article Cross-Validation for Imbalanced Datasets: Avoiding Overoptim...
Best of luck!
I create a model on all the data and solve the identification problem on a training sample. Then there are two options. The first is to simply delete the data that caused the identification errors from the training sample. The second more advanced option is to create separate classes for identifying the data that led to errors. Both of these options are implemented in the Eidos system of my development. There is a third option: describe the training data more fully, so that there are fewer poorly identified ones left.
Я создаю модель на всех данных и решаю задачу идентификации на тренировочной выборке. Потом есть два вариант. Первый просто удалить из обучающей выборки те данные, которые вызвали ошибки идентификации. Второй более развитый вариант - создать для отдельные классы для идентификации тех данных, которые привели к ошибкам. В системе Эйдос моей разработки реализованы оба эти варианта. Есть еще третий вариант: более полно описать данные обучающей , чтобы плохо идентифицируемых осталось меньше.
first define what does it mean by low or high quality data. In an argument mining exercise a low quality text may be a text with few number of argumentation discourse units;or,it may be a data set made up of texts replete with ADUs but different disciplines(which will decrease the quality of your data set, because different genres have different argumentative structures);your low quality data set may be due to your annotation(Tagging);accordingly first define the reason then go for compensation as a case in point for the last example you can read paper bellow.
( Conference Paper Will it Blend? Blending Weak and Strong Labeled Data in a Ne...
)the best
- Badges
- Science topic