I am developing a screening tool to identify mental disorders in forced migrant populations. So far we have used CART & ROC & sensitivity/specificity analyses to get a reasonable ROC (.801) with a cut-off score of 2 items but we really need to increase the ROC (i.e. specificity is too low - about 65%).
I am therefore after input from anyone proficient in Item Response Models (e.g. Rasch or other hierarchical models) that will help me to maximise the predictive accuracy of our screening tool.
Many thanks.
Debbie
Hi Debbie,
I think I would I help you. Would you like to talk about?. Please contact me using private messages system.
Luis
Dear Debbie,
IRT models can generally help you identify the items which match your criteria most powerfully. The idea is to find the items with the best discrimination at the difficulty level which matches whatever lines in the sand you are drawing for mental illness. The words discrimination and difficulty of course don't really match this application.
You can then take possible sets of items and evaluate the test characteristic functions to see if you can tweak or adjust the items, their scales, or the criteria applied to make improvements.
Use of IRT and related models in this way is a little bit opposite to their intended function - but works fine. However, if your criteria are already reasonably set why not consider some versions of discriminant analysis and/or robust regression.
hi Debbie, Using Rasch analysis will ensure your scale is at its optimum e.g. removing misfitting items, allowing you to examine if data fit the Rasch model and your scoring categories are working as intended. You could then rerun the ROC to see if a Rasch transformed scale has improved specificity. I am an occupational therapist who has used Rasch extensively and would also be happy to help or I can make some suggestions of people near you if that would be helpful.
BW
Anita
Hi Debbie,
If you've tried CART, you're giving your indicator set about as much chance for success in discrimination as possible, because it is such an opportunistic method (which is why validation is so important). If that doesn't minimize the false positives as much as you'd like, you may have to add to the items in the screener. Alternatively, if you believe that one outcome is arguably more costly than another, then you can adjust the criterion to maximize the target criterion (e.g., if false negative is worse than false positive, maximize sensitivity).
Thanks David. We did use CART to establish the 10 items to be included. However if anything we wish to cull items, not add (ie need it to be brief). Unfortunately the CART licence I was using was a trial and it's now expired.
- Badges
- Science method
- Similar topics
- Methodology