There is a gold standard method (G) to measure a continuous variable. But it is costly. A new method (N) was developped and we want to compare with G. We used MAE (Mean Absolute Error), BA plot in addition to error (|N-G|)and error proportion (|N-G| / G) more than certain %.
Now I need help in assessing its repeatability. We have an option of three readings from both N and G. How will we deduce which method has better repeatability/ precision?
You may find the concordance correlation coefficient (Lin’s CCC) including CCC plots usefull. Bland-Altmann plots as well.
Dear Mr Shah
I am sure there are methods to test your intent and the concordance coefficient mentioned by Esben Østergaard Eriksen is a good approach.
If I were you, I would think of two things:
1. Internal validity
2. External validity
For internal validity, your measure should have a high internal consistency (Cronbach's alpha, Composite Reliability) and a high explanation power (Average Variance Explained).
For external validity, you should build a nomologic network, i.e. bring up all terms and concepts that are theoretically related. Then, I would test different related concepts and look that your measure correlates with them in a planned manner.
You can also ask a sample of your population of interest to rate/rank the measure or different measures in how much they approximate what you planned. We did something similar when we looked for words for our Implicit Association Test. We gave our pre-testers the target concept and based them to rank different words in relation to the target concept. Then we performed Friedman's test.
As the main goal is to use your new measure as a predictor or as a dependent variable, I would take those predictors that are usually used for the alternative measurement. I would look at the R-squared values to show that the variance explained is higher with your measurement than with a competing one.
I hope it helps you!
Sincerely
Dr Yevgen Bogodistov
To measure the repeatability of a new method, researchers typically perform multiple tests or experiments using the same method and compare the results obtained. Statistical analysis is often used to determine the degree of variation between the results, and the level of repeatability is assessed based on these variations. Several metrics can be used to measure the repeatability of a new method, including the coefficient of variation, the intraclass correlation coefficient, and the Bland-Altman plot. Ultimately, the goal is to achieve a high level of repeatability, which indicates that the new method produces consistent and reliable results that can be used in further research and practical applications.
- Badges
- Science topic
- Similar topics
- Biological Science
- Molecular Biology