using k-fold cross validation you can select the best discriminating feature set....

Dear friend

There are different kind of tests on the data that you can perform before you use it for the training purpose:

1. reliability test.

2. There is some test known as Alpha test- this is one of the most widely test used.

I have used this method to validate the data sets in one of my papers. You may have a look at them.

3. You can also do some other statistical tests like info gain, correlation among the variables and others like that.

But the selection of the test that you do depends on the type of task that you want to perform.

I hope this helps you.

All the best.

Best regards

Dr.Indrajit Mandal

I hope this helps.

By 'validate the quality of a training dataset', I am assuming you are referring to minimizing the possibility of class label "A" being mistakenly labelled as "B"? If I am right, maybe you can start with this publication (mind you, it is not a new publication).

http://dl.acm.org/citation.cfm?id=641658

The first point to verify would be, your training set is a sample of your population therefore you could have the same distribution of "errors" in your population and therefore your training set as the expected quality.  If you could have multiple training set out of the population you can do few inferences to verify.

@Dr. Indrajit Mandal, could you please show me the title of the paper, and if possible the link to download it?

one easy way is that, subset your training data to 20 percent to train classifier and 80 percent to test the result , if a class labeled in wrong change the class label and try again.

Here is my research paper dedicated to tackle this question. 

An Approach to Evaluate the Fitness of One Class Structure via Dynamic Centroids

Jing-Doo Wang and Hsiang-Chuan Liu

Expert Systems with Applications, October 2011, Volume 38, Issue 11, pp. 13764-13772.

I think you must divided your data set into three  category for Training,validation and testing .You can read my research paper ,may be find your answer .

thanks

In order to estimate the quality of your training set labels (part (1) of your question), you will need to obtain accurate labels for a subset of your dataset first and then estimate the quality of your old labels based on these.

I would subsample the dataset randomly and annotate it redundantly (i.e. multiple judgments per instance), either using experts (you and colleagues) or crowdsourcing. A common scheme used in crowdsourcing would be to obtain a certain number of annotations for each instance (e.g. 3) and then compute how well these match (inter-rater agreement). In case of doubt, you obtain additional annotations until sufficient agreement is reached (e.g. a certain threshold for the agreement measure of your choice is reached).

Regarding (2), there already have been interesting pointers, which might improve classification performance. If these prove to be insufficient (or inapplicable), there are always two courses of action open regarding the data set itself:

1. obtain more data (only if quality is not too poor)

2. obtain better quality data by collecting additional annotations for your existing data

Article oro.open.ac.uk How Reliable are Annotations via Crowdsourcin...

Really, the one way only exists - to call expert or several experts.

All statistics approaches have assumption which cannot be usually checked. For example, suggestion that your set of input features CAN discriminate classes. Nfold cross validation mentioned by several authors above used this assumption.

Well, an interesting problem and here are some thoughts. 1) Maybe you could analyze it from another angle by using a different machine learning algorithm to try to gain some insight on the dataset.  For example, while you look at the misclassification rate in a decision tree to identify the rules for a target value, a neural network identifies a good model by using the square error between the model and the runs... Cart's iris.csv is a good one to analyze from these two different perspectives.  Or, maybe try a clustering algorithm to see what is grouped together to look for anything odd when compared to the other models. 2) If you can identify or suspect which data is bad then maybe you could replace those particular  fields in the records with null values... which would provide you with information. 3) Lastly, have a huge amount of data so that these outliers don't make much difference.

Thank you very much to all of you for your great comments/potential sulitions. To some extent, I agree with the solution on using another ML technique such as clustering (without labels) to detect the ouliers in the "low qualified" training dataset and remove them from that training dataset before training the classifier.

However, considering the 2nd question and observing the process of train/education that human being does. I have this idea:

(1) In education, the learner/student is taugh by a teacher/professor. But the knowledge that the professor teaches may NOT be always correct. In ML: "student" is the classifier, the knowledge comes from training dataset with labels, and incorrect knowledge are incorrect labeled instances.

(2) If the student has some aditional knowlede, independent with knowledge he was taught by the professor to detect the "incorrect" points and correct them, then he can make a better decision compared to merly/blindly applying 100% knowledge the profesor taugh.

(3) The additional knowledge that a student/classifier can gain would be also from the training dataset but without the attached labels.

How about your opinions on this direction or further advice?

Best regards,

Yes, it sounds like you are talking about surrogates? These are fields that can act as a substitute for deciding on a split in a decision tree when there are missing values.  

I am not sure about the separate fields that play as surrogates in a complicated dataset (with large number of fields). There may be some and each of them may be attached with some weight (in the correlion with other fields). However, the data themselve will have some correlation with each other regarless of what the label people makes for each instance. This is the rationale behind clustering technique. My original opinion is: 

(1) Using the low qualified, but adequate, labled dataset to train the classifier which will be used to classify the new data. The rationale is that, although the qualify of the labels in the training dataset is low but the correct lables whould be valuable for training the model (in a direct way). Imagine that 80% of the labels are correct then they would be very useful to gain the knowledge (features-> lables) from the data .

(2) Use the correlation of features themselves (without label) to analyze the homogeneity of the resulted classes to remove the miss-classified instances.

I wonder this approach make some sense in the practice.

Diffgram https://diffgram.com (where I work).

We have an integrated SDK to validate data that helps with the "did the human label it correctly" problem https://medium.com/diffgram/how-to-validate-your-deep-learning-model-with-the-diffgram-sdk-tutorial-

and: https://medium.com/diffgram/scale-annotation-quality-7dd285a22d01

• Workflow. Manage tasks at scale with Jobs.
• Visually see history of annotations over time so you can control quality
• Video support. Video frames with interpolation can save a ton of time!

Diffgram is web based:

• Fast to get started - create an account and start in moments.
• Share work with teammates for review

Interesting observations

Excellent question

https://www.researchgate.net/post/How-to-verify-a-good-training-set

Step 1: Determine data sample. Determine the data to sample. ...

Step 2: Validate the database. Before you move your data, you need to ensure that all the required data is present in your existing database. ...

Step 3: Validate the data format.

https://www.alooma.com/blog/what-is-data-validation

In any data set, there is a useful signal and there is noise. Accordingly, there is a signal-to-noise ratio. The higher this ratio, the higher the quality of the data set. To assess the quality of a real data set, it is necessary to generate a random data set similar in dimensions (the number and type of scales and gradations, the training sample) and link the models created on the basis of these sets. The difference between these models is due to the presence of patterns (signal) in the real data set. There are several methods for suppressing noise in the data. For example, you can use a non-metric integral criterion (inter-vector distance), which is a filter that suppresses white noise. You can also use the division of classes into typical and atypical parts. This can be done by performing divisive (dividing) clustering. All this is implemented in the Eidos system of my development. In my publications in RG, you can see examples of the application of these approaches, for example, in the works:

Lutsenko E.V. ASC-analysis of the impact of the space environment on seismic activity on Earth (earthquakes of the world with a magnitude > 6 for 1900-2019), January 2020, DOI: 10.13140/RG.2.2.21913.16481, https://www.researchgate.net/publication/338541064.

Lutsenko E.V. Forecasting the risks of loan non-repayment using an intelligent iterative algorithm for accounting for atypical cases // May 2021, DOI: 10.13140/RG.2.2.32991.38560, https://www.researchgate.net/publication/351924470.

Lutsenko E.V. INVARIANT TO VOLUMES OF DATA, A FUZZY MULTICLASS GENERALIZATION OF F-MEASURE OF PLAUSIBILITY IN VAN RIJSBERGEN MODELS IN ASC-ANALYSIS AND IN THE "EIDOS" SYSTEM, June 2019, https://www.researchgate.net/publication/333641576