Could you suggest some suitable tests with references for conducting reliability and validity test of a categorical variable? The study used factorial ANOVA.
Thanks in advance.
Gobinda
Hi, Gobinda,
Sure, here is two article that show how to use a machine-learning method (no distributional assumptions, exact p-values, and cross-generalizability analyses are standard). The examples use ordinal data, but the procedure when applied to categorical data is identical (except exact p is determined for a categorical attribute, rather than for an ordered attribute).
Inter-rater:
- https://odajournal.com/2014/04/13/how-to-assess-inter-observer-reliability-of-ratings-made-on-ordinal-scales-evaluating-and-comparing-the-emergency-severity-index-version-3-and-canadian-triage-acuity-scale
Inter-method:
- https://odajournal.com/2014/04/14/how-to-assess-the-inter-method-parallel-forms-reliability-of-ratings-made-on-ordinal-scales-emergency-severity-index-version-3-and-canadian-triage-acuity-scale
The best way is to redo the study. You build the model with the first set of data and then see how well the model performs with the second set of data.
The second best option is to build models wherein you leave out a subset of your data. The model is rebuilt with the reduced data set and tested on the data that was left out. The subset may be one or more values.
You could take your data as accurate, and build a simulation. In treatment A=1,B=1 the mean was 2.3 with a standard deviation of 0.9 while in treatment A=1, B=2 the mean was 2.8 with a standard deviation of 0.4. Take these values and put them into a random number generator, from which you then collect a sample size of n and run the analysis. Do this many times (100,000 or so) and figure out how frequently one could get a significant result given your sample size and the underlying distribution of your data.
In all cases, the accuracy of the method is largely an effect of your sample size. None of these approaches will work well with 4 replicates in each treatment because it is unlikely that one sample of size 4 will accurately estimate the true population mean and standard deviation. If you had 1,000,000 samples of size 4 and took the average then you would have a good estimate of the population values, but that does not apply to a lone sample.
- Badges
- Science method
- Similar topics
- Mathematics
- Statistics
- Statistical Modeling
- Analyses of Variance
- ANOVA