I would use paired non parametric test like Kruskal Wallis test to detect difference. This assumes there is a difference in the outcome. Otherwise if there is no change you are testing test-retest reliability using the same method. Suggest 30 - 50 subjects.

I'm assuming you would like to find some useful illustrations in the literature of things that have been done to demonstrate sensitivity to change of instruments in various areas. I can recommend 3 articles with different scales. For 2 of these the PubMed IDs are 10855461 and 22424915 and for the third, the PDF of the article can be found here: http://her.oxfordjournals.org/content/17/2/211.full.pdf

Hope you find these helpful.

First, you must pay attention to the number of items, also if it's a qualitative measure or a quantitative measure. And you sample size and the items format.

If it's a quantitative measure, pay attention to your sample and the number of items. If both are big (more than 30 items and more than 50 subjets), you must compute Pearson's correlation. If not, you must compute the non-parametric alternative: Spearman's correlation. You can interpretate both in the same way, but remember, Spearman's correlation has less power than Pearson's correlation. I recommend Pearson's correlation in order to quantify the temporal stability of your test. And, also compute Cronbach coeficient to quantify the internal consistency (you only need one measure, the pre or the post test, but I recommend to compute with both measures and compare if there are differences).

If it's a qualitative measure, try to compute a chi-square test, and try to establish the ROC curve in order to detect false positives and so on.

If your measure use two judges, you must compute kappa correlation.

I hope this helps. Good luck with your research. If you need more help, please let me know.

Best regards!

Hiller, W., Schindler, A.C., Lambert, M.J. (2012). Defining response and remission in psychotherapy research: A comparison of the RCI and the method of percent improvement. Psychotherapy Research, 22(1): 1-11.

Here you can find something on it,

Previously you can use the ROC curves to determine the cut off value,

It's the best way to determine that value tuning on sensibility and specified of your instrument...

Best

Pablo

Thanks to all of you. I’m specially interested in the “smallest detectable diference” analysis since it was asked by our counterpart in the study. Can you direct me to some additional specific literature?

Thanks to Merche, Pablo, Stephen and Rupert

Additional information

The instrument is a measure where each item assess the number of days of drug use in the previous 4 weeks (min 0 days max 28 days). It measures different substances so each item should be analized separately. Sample size will be about 100 participants. Sensitivity to change analysis will compare information reported in the test stage with a follow up after 28 days.

We have also performed the the reliability analysis using intreclass correlation (two way random model and absolute agreement method) of a pretest retest (assesing substanse use in the same time period)

I think you are interested in assessing the change in scores or your measure over and above the change that may be observed because of measurement error. Thus, the problem is of detecting the minimal or smallest statistically significant change in score with certain level of confidence that it is well beyond measurement error.

Though, I have not done such analysis in my own work, I hope the following links to some research papers and excel sheets (for calculations) may be of some help.

http://webspace.ship.edu/pgmarr/Geo441/Examples/Minimum%20Detectable%20Difference.pdf

http://mjiri.tums.ac.ir/browse.php?a_id=127&slc_lang=en&sid=1&ftxt=1

http://www.google.co.in/url?sa=t&rct=j&q=smallest%20detectable%20difference%20calculation&source=web&cd=9&cad=rja&ved=0CFcQFjAI&url=http%3A%2F%2Fwww.ucd.ie%2Fstatdept%2Fclasspages%2Fpgrm%2FMinimumdetectabledifference.xls&ei=AH1sUJ_bNYKPrgefyYHYCw&usg=AFQjCNFJbL8eGGDx-KYRR4pl-zW_uwaAqQ

http://ard.bmj.com/content/64/2/179.full.pdf

Instead of attaching the file I attached the link in previous post. Here is the paper

Sensitibity is detected by a ROC curve. But your items are "special", I mean, you don't have to sum items to obtain a puntuation. So, I think to compute a ROC curve won't be a good decision.

All your items are quantitative (days that a person use a drug) and your sample size is big.

I need to know if you're carried out some kind of intervention, or something like that, because it may affect to the results. Like pre and post test in a repeated measures desing.

I recomend you not only to compute Pearson's correlation for each item (pre and post) but also include a T test (for related measures). It would give you some initial information about your temporal item funtioning. Computing the size effect of this T test may give you information about sensibility, but, as I said, is a "pilot" analysis.

What Craig Caldwell says sounds fine, but I don't know if that approach would be the best one (I know nothing about IX Process Advantage).

The link provided by Rakesh Padey is very useful, so take a look at that pdf file.

Good luck with your research!

Tnanks again to all of you. I'll check the information.

Cronbach's alpha. http://www.bmj.com/content/314/7080/572 to measure internal consistency from perception items regarding variation. Good luck, hope it works for you!

Hi Here is some work we presented on this earlier in the year

Thanks Jeremy, it's really comprehensive reference. Do you mind if i request additional details if needed?

Yes that's fine. I am hoping to write a paper on this next year. when looking at the theory I was struck by how much of the research came from quality of life researchers.