I'm doing a reliability study on a diagnostic procedure in detecting a type of cancer. I've had had 15 raters report how certain they are in their assessment, using a Visual Analogue Scale from 0-100.
Now I want to see if there is an association between each raters level of certainty and their accuracy. For each case a rater can only be either accurate or inaccurate, thus my dichotomous dependent variable.
Ideally I would like to statistically 'punish' raters who are dead certain, but inaccurate, or lucky-guessing by being accurate, but uncertain, while 'rewarding' raters who know when they are likely inaccurate or are certain when they have right to be. I'm looking for a test that can reflect and quantify this.
Any suggestions and how to do it in Stata or SPSS?
Dear friend
Is it a reliability study or a accuracy study ? You may want to determine indices of internal validity (sensitivity, specificity) by establishing an ROC curve (Receiving Operating Curve).
Dear prof. Bezzaoucha,
I'm already past that stage and already calculated sensitivity, specificity, accuracy, Cohen's kappa, Fleiss' kappa, Gwet's AC1, McNemars test, ICC(2,k) and more.
Now, this is the missing piece of my puzzle, trying to establish if there is a correlation between raters accuracy and their certainty in their assessment.
After looking around it seems point-biserial correlation is the way to go for my application (https://statistics.laerd.com/spss-tutorials/point-biserial-correlation-using-spss-statistics.php), although some of the assumptions about outliers and normally distributed continuous variable don't hold true, which I haven't figured out how to work around yet. Do you know?
Dr Gemdzhian; not an as helpful answer as you might think. Please elaborate your response.
Why a multifactorial ANOVA when I only have one catagorical/dichotomous variable and one continuous variabel? Wouldn't that be a one-way ANOVA? What are the pros and cons compared to the suggested solution from my link above? And why do you think so many recommends a point-biserial correlation if a simple ANOVA or even Student t-test would do, the latter being much more well-known to most?
Of course you are absolutely right (if you have only one catagorical/dichotomous var.), sorry
Dear friend
The point-biserial correlation coefficient is used when the l dichotomous variable is not artificially expressed in two categories.
The bi-serial correlation coefficient is used when the qualitative variable is not dichotomous by nature but has been artificially expressed in two categories.
Attached is a French file that may be useful (Bezzaoucha A. Tests statistiques en sciences médicales. Office des Publications Universitaires. Alger 2004)
Merci beaucoup!
Je ne l'ai pas étudié le français depuis plusieurs années, mais le langage mathématique est universelle.
- Badges
- Science topic
- Similar topics
- Medicine
- Oncology
- Cancer Research