I have data from two samples (community and clinical/service sample) of adolescents who completed psychological evaluations, and I would like to: (1) determine how well the instrument used for these evaluations is able to discriminate between these two samples.(2) to use the ROC analyses to determine the optimum cut-off for differentiating between the community and clinical/service sample. The dataset has 2 other variables: gender and age. Is this information enough to carry out ROC analysis? I attempted this analysis in stata and the AUCs (ranging from 0.518-0.578), seems to indicate that the performance of the scale was nothing more than chance. I am not sure but I assumed for the purpose of my analysis that the community sample is the negative sample( i.e. most likely to be without mental difficulties) and the clinical/service sample is the positive sample (i.e. most likely to be with mental difficulties). Is that correct? I would be grateful for any guidance on how to implement this model in stata and general advice on model testing.
65
The first thing to do is to is to run a Wilcoxon Mann-Whitney with the option to give the probability that a person from one group will score higher than a person from the other group. So in stata this would be
. ranksum score, by(group) porder
The porder option is the one you need.
This gives the area under the ROC. If this is not promising, that's more or less the end of the analysis. If it looks good, then tell us.
Hi Ronan, Thank you for responding to my request for help.I have now run the Wilcoxon Mann-Whitney and the results suggest that there is a statistically significant difference between the underlying distributions of the psychological scores of the community sample and the psychological scores of the contact/service sample(z = -20.401, p = 0.0000). What is the next move? Can I also ask will it make a difference to the results if one of the samples has over 160,000 observations and the other has only 900?
Best,
Kenisha
23
Why bother with a sample size calculation? You already have lots of data!
Next step is to use Stata to tabulate the test performance at different cutpoints.
. roctab score group, detail
You will have to decide on a cutoff based on some criterion. You will have a trade off between false negatives and false positives. The optimum cutoff depends on the setting. Clearly, if it is vital that you detect cases, then you maximise the sensitivity, at the cost of a lot of false-positives. On the other hand, you may want to minimise the burden on services caused by false positives, so you set the cutpoint to keep these as few as possible (and your prevalence will be important in deciding on this).
Here's a paper on setting cutpoints that you might read to help the decision making.
Liu, X., 2012. Classification accuracy and cut point selection. Statistics in medicine, 31(23), pp.2676–2686.
16
http://onlinelibrary.wiley.com/doi/10.1002/sim.4509/abstract
Thank you both for the advice.
This has really been helpful, I will try the proposed steps and let you know the outcome.
Best,
Kenisha
- Badges
- Science topic
- Similar topics
- Publishing