Dear Rakesh, I you would like to verify that I have well understood your question. Did you mean that you search a way to associate different indicators with valuable diagnostic value estimated by their ROC Curves.

I use ROC curves sometimes to estimate the cut-off value of the parameters of interest and transform the experimental result in binary data where 0 stand for negative and one for positive.

Then, I submit the data to linear discriminant analysis and principal component analysis.

In this way, i find the best combination of test to use and i have a final equation with the ponderation of each of the associated parameters. With this first experimental model, it is possible to have a graphical extrapolation and then used it as a support for an extensive population of patients and controls.

WHat you are doing depends very much on the nature of the questions generating the data. There are two different types of judgments or indicators that determine what you do. What are the indicators in this case?

I have several comments on these question and perhaps i did not quite understand the essence of the question, so forgive if the answer is not straightforward. First, there are several different names for similar things, some may call ROC regression, some may call GLM ROC but in essence the can be the same thing. So Im assuming you want to combine several tests in a regression and then estimate a posttest probability in which some decision will be based. If ithis is the case then we are talking about clinical prediction (or decision) models and I suggest you take a look at Styerberg's book: http://www.clinicalpredictionmodels.org/ and the rms R package is pretty handy.

If you are talking about to estimate adjusted accuracies of tests when they are combined with different other tests then Zhou's book or Pepe's book are nice.

Zhou made a software called rockit wich was nice 5 years ago when I first gave it a try.

http://books.google.com.br/books/about/The_Statistical_Evaluation_Of_Medical_Te.html?id=kMyXEJEtFmkC&redir_esc=y

http://books.google.com.br/books?id=NWGUtAdOT8kC&printsec=frontcover&dq=Zhou+tests+in+medicine&hl=pt-BR&sa=X&ei=EUkRUbWTGYnQ9ASj4YEI&ved=0CDIQ6AEwAA#v=onepage&q=Zhou%20tests%20in%20medicine&f=false

Cutoffs are not straightforward. By estimating a cutoff one implicit desires to relate a range of the test where the posttest probability is likely to indicate the subject is with the outcome of interest then not. But each cutoff has a different combinatioin of Se and Sp and hence a different Positive likelihood ration. So each threshold has an ability to increasy the possibility to the target condition to be correctly identified. However, posttest probability likely to indicate the subject is with the outcome also depends on other thing then the test result, such as the pretest probabilities, the weights one subjectively or explicitly gives to wrong decision after the test, the setting where the test will be applied (screening vs confirming; combining vs alone) among other things. This means one single cutoff will never be good enough for most of situations.

Recentely my mates and I conducted a review of estimators for cutoffs of diagnostic tests, and we were able to find more then 50 estimators, many of them requiring user data such as costs or utilities. On the other hand, the vast majority of the popular software such as SAS, Stata, MedCalc and SPSS, when they estimate decision thresholds, the method used is not clear. Codes for these methods are really scarce. This paper is under appreciation on JCE. Now, my mates and I are trying to write down this estimators on R code and run a simulation study. But this all is about a single diagnostic test alone.

If you are running a regression and want to know which cutoff should be used for the predictors, my advice is to read Frank Harrel and Douglas Altman papers about this issue. They state that categorizing continuous predictors is just loss of information. One may use modern techniques to stick in the continuous predictor in the regression transformed with splines or cubic splines or other techniques which will capture non linear relationships of the predictor toward the outcome and in the bottom line make much better predictions.

So far, I believe the best strategy to estimate a diagnostic decision threshold from a regression is to conduct the regression with signs, symptoms, (or with the test) and estimate the posttest probabilities (the predicted probabilities). Then, with these probabilities and some user information estimate one or more test-thresholds and the test-treat-thresholds:

Pauker SG, Kassirer JP. The threshold approach to clinical decision making. N. Engl. J. Med. 1980;302(20):1109–17.

Regards,

Dear All

Thanks a lot for your helpful suggestions.

Dear Dr. Lefranc, I also though of running principal component and use the factor score or aggregate score of the variables related to any component as indicator variable for the ROC. However, this will give information about how well the new score (that is composed of several individual indicators) is working in comparison to each of the indicators taken alone.

Let me elaborate a little about the data and the solution that I want. I have assessed certain emotional aspects of two groups of individuals- drug abuse group and control group. I did binary logistic regression and found that some test scores such as emotion regulation difficulties, affect intensity, and negative affect significantly predict individuals with drug abuse. with high sensitivity and specificity. Then, I thought to determine a threshold (cut-off) score that can be used to predict the likelihood of drug abuse. Using ROC I obtained cut-off scores that have sensitivity about .9 and specificty about .85 each with AUC over .8. By comparing the AUC I have also identified the most efficient affective measure that can be used to predict drug abuse.

Now, I am thinking that out of all possible combination of different levels of three variables, there will be several combinations that may work equally effective in predicting drug abuse. A hypothetical example may be that a person with a very high negative affect but moderate level of emotion regulation difficulty may be as likely to be an individual abusing drug as an individual with moderate level of negative affect but extreme difficulty in regulating emotions. Else, there may be only one combination that may turn out to be best in relation to either of the test taken individually or any other combination of test scores (cut-off scores).

I am trying to find out that ,out of the different combinations of scores of the three measures, one or more combination may be the best in terms of diagnostic accuracy. Whether it is in terms of sensitivity, specificity, likelihood ratio, or post-test probability with a given prevalence rate.

I hope that now I may get more specific suggestions as how to go ahead.

Thanking you all once again.

First of all, please clarify:

- do you have multiple input/predictor variates and want to have a cuoff scheme for each of them in order to predict one outcome category? That would be e.g. a decision tree.

- Do you have multiple outcome categories, and some kind of predicted score for each of them? In that case, are the categories independent (a patient may have hepatitis and cancer) or not (a patient may not have stages I and III of the same tumour at the same time)? If they are independent, you can do ROCs independently for each category. If not, you can still draw (1 - ) specificity against sensitivity for each outcome category, but you may need to take extra care about "impossible" regions.

E.g. Landgrebe & Paclik describe a multiple outcome category version of ROC: http://www.sciencedirect.com/science/article/B6V15-4Y3JY7V-1/2/cea3ac8b2b0984043772a214edbd8923

Fawcett wrote some nice and also introductory papers about ROC analysis, I recommend

- An introduction to ROC analysis http://www.sciencedirect.com/science/article/B6V15-4HV747X-1/2/c1653cca4db4e94215437a482fcbecbb and

- ROC graphs: Notes and practical considerations for researchers http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.10.9777&rep=rep1&type=pdf

As Pedro already explained, there is not *the best* cutoff. You'll need to specify the needs of your diagnosis. Sensitivity may be more important than specificity? How much? Neverthess, you may need to have specificity to be at least xxx. and so on.

However you select your working point from the ROC, remember that you need to validate your final diagnostic model with an independent data set - the data set that was used to construct the ROC and find the working point is *not* independent of the model.

Last but not least, lots of questions about ROC have been discussed at http://stats.stackexchange.com/ .

Dear Dr. Claudia Beleites

I have no words to express my sense of thankfulness for your insightful suggestions and specific recommendations/links/suggested readings. It is really very helpful,. I'll try to go through all the references and link.

I have multiple input/predictor variates and want to have a cuoff scheme for each of them in order to predict one outcome category. Of course, like a decision tree. in which certain combination will be associated with a positive outcome.

In fact I wart a combination of cut-off scheme in some form of Boolean expression. For instance, X1> C1 AND X2>C2 AND x3

Hi, you can use principle component analysis to combine multiple test scor in to one variable.. and you can use MedCalc software to obtain optimum cutoff value ..

best

arzu

Dear Arzu Kanik

Thanks for your valuable answer. Dr. Didier Lefranc has also suggested the same.

But the problem is that a single variable created by some linear and algebraic combination (e.g., through regression, principal component etc.) may not work in the same way as the Boolean combination of the same variables.

In the former case, equal variation in all the variables (included in the composite score) is assumed to be associated with variation in diagnostic accuracy. Whereas, in the later case, a combination of different degree of variation in the set of variables is assumed to be associated with variation in diagnostic accuracy..

In the former case the weight of each variable is fixed and the change in the composite variable will be related to the weight of the component variables. While in the later case, one variable may take several scores (within the range of possible minimum and maximum score) and this may combine in several different ways with the variable change in the other measures. Let us assume two variables X and Y and both have only two possible scores 0 and 1. Then we may have the following combinations:

X=0 AND Y = 0

X=0 AND Y = 1

X=1 AND Y = 0

X=1 AND Y = 1

X=0 OR Y = 0

X=0 OR Y = 1

X=1 OR Y = 0

X=1 OR Y =1

If the scores of X and Y will be continuous and may take several possible values then the said combinations will be more in number and complex as well. In this case it may involve such combinations as If X = a fixed score AND Y < a fixed score, X is greater than a fixed score AND Y is also greater than another fixed score and so on and so forth.

Now, one may be interested in knowing which of the aforesaid combination of variations in the scores of X and Y maximally differentiate the patients from control and gives the best diagnostic accuracy.

I am interested in this type of combined cut-off score.

For all, I would say that I am interested in knowing one or a set of combination of scores from among all possible combinations that gives diagnostic accuracy better than rest of the combinations.

My predictors are continuous and the outcome is dichotomous in nature and for each outcome I have 75 cases.

I apologize to all if I am not clear in my expression, as I am not much aware of this type of statistical analysis and it is a new for me.

@Professor Rakesh Pandey : Great. You need not to apologize. Instead, we all should appreciating you for daring step to enable us understand relevance rather more clear. In fact, we should not hesitate to interact at this forum of brain- storming views.

@Prof. Rakesh Pandey:

See e.g. "The Elements of Statistical Learning" (http://www-stat.stanford.edu/~tibs/ElemStatLearn/) for an explanation of decision trees. They also explain the advantages and disadvantages of this classification method.

Two things you need to take special care of:

- decision trees tend to suffer from instability, that is if you exchange a few of your cases against new ones, the tree may change very much. Therefore you absolutely need to check the stability of the models. As very different decision trees can describe very similar decision boundaries, so it is not easy to judge how similar two decision trees are.

- decision trees can severely overfit your data. That is, the existing data set is classified correctly, but this does not hold for new cases.

In practice, bootstrapping or repeated/iterated cross validation allow you to measure the stability of the models' prediction as well as the generalization error (performance for new cases).

In fact, the first aspect of bootstrapped decision trees leads to the so-called random forest models, which are much more stable than single decision trees: a number of decision trees is grown with slightly varying input data (randomly selected cases and input variables). New cases are presented to all these trees for prediction. If you then get the result that 90 out of 100 trees agreed in their prediction, you'll put more trust in that than if only 56 agreed, but 44 predicted the other class.

Whether decision trees or multivariate linear classifiers (e.g. logistic regression) is more appropriate will also depend on your data.

"But the problem is that a single variable created by some linear and algebraic combination (e.g., through regression, principal component etc.) may not work in the same way as the Boolean combination of the same variables.

In the former case, equal variation in all the variables (included in the composite score) is assumed to be associated with variation in diagnostic accuracy. Whereas, in the later case, a combination of different degree of variation in the set of variables is assumed to be associated with variation in diagnostic accuracy."

This is not true in the way you state it: you can take care of different scales/variable weights - one popular way is variance scaling, and more alternatives exist. While it is true that PCA projects for maximal variance (regardless of whether this particular variance helps in the distinction), other methods provide according projections, e.g. linear discriminant analysis (LDA) or partial least squares (LDA).

And while linear classifiers cannot classify the so called XOR problem (class A if both X1 and X2 are low or both are high, class B if they differ), you can generate features that allow also linear classifiers to deal with such a case (e.g. using X1, X2, and X1*X2).

Also, with regard to your OR / AND possibilities above: please note that there are more possibilities:

X = 0 (regardless of Y)

X = 1 (regardless of Y)

Y = 0 (regardless of X)

Y = 1 (regardless of X)

neither X nor Y predict the class.

(as in your list, this specifies one class).

Finally, while hard thresholds on continuous measures seem easy, they can cause lots of trouble from a data-analytical point of view. One fact that you should be aware of is that dichotomization will always throw away information. Dichotomization may be justified if the input variable shows a clear separation between groups of cases and the decision threshold falls into the "gap". However, if you end up cutting a Gaussian distributed variable near the mean, that means that you have high density at your decision boundary. This in turn means that already a low uncertainty on this variate (how precise are the psychological scores of your cases?) leads to many cases ending up on the wrong side of the decision tree. You do not differentiate between someone who barely met the threshold and someone who is way above. But you make all the difference between the one who is barely above and someone who is barely below the treshold.

So, please read these warnings Prof. Harrell has put together:

http://biostat.mc.vanderbilt.edu/wiki/Main/CatContinuous

Dipl.-Chem. C. Beleites

Dept. of Imaging & Spectroscopy

IPHT Jena/Germany

@Claudia Beleites

Thanks a lot for such a detailed discussion and providing helpful tips and links.

Your suggestions, and cautions are really helpful and I agree with your views expressed.

I do agree that dichotomization of a continuous variable not only leads to information loss but also brings several other issues. Though use of cut-off score for purpose of classification is in practice, it is always fraught with danger of classifications for borderline cases.

Even with the dichotomous criterion measure commonly used in clinical setting e.g., patient versus healthy controls, depressed versus non-depressed etc., there is an issue whether they represent true classes distinct from each other or they are created dichotomy for a dimensional variables.

For instance, using the taxometric analysis (initially developed by Paul Meehl) now evidences are accumulating that many of the variables that are used as class (taxonic) varibles are in fact dimensional and depression is one such example.

I once again thank you and all other contributors to this thread for helping me to develop some insight what statistical principles and techniques will be helpful for the problem that I am going to address.

Hello Prof. I had an abstract ECHOCARDIOGRAPHIC LEFT VENTRICULAR HYPERTROPHY (LVH) AND PROGNOSIS – A META-ANALYSIS

Running title: LVH and outcome

Deborah J Hilton MPH a, b Christopher M Reid PhD a, b, Garry LR Jennings MD a, c.

years ago that has been forgotten about now, lost in the system like a lot of things, in that this was never written up, but I am going to email you a page or two where I tried to do an ROC curve. Cheers, Deborah

Hi,

Much of the discussion in response to your question involves ROC curves and finding the optimal discrimination point on those ROC curves. It is widely understood that ROC curves can be relatively insensitive to clinically significant differences in absolute risk. Several approaches based on risk stratification have been developed in part to get around this limitation. These include the reclassification calibration statistic, net reclassification improvement or NRI, and integrated discrimination improvement or IDI. These newer approaches are summarized in several papers by Dr. Nancy Cook. One of her papers on this topic that is intelligible is Cook NR and Ridker PM, Ann Intern Med 2009; 150:795-802.

Best wishes,

Howard

Dear All,

Nice to hear the relevant statement.

Slight expansion of my suggestion above: Dr. Cook presents summaries of techniques that can be used to judge the extent to which a new measure adds to the existing discriminatory ability. These techniques expand the nuances that can be evaluated beyond the area under the curve, or AUC typically examined in evaluations of the extent to which a new measure adds to existing discriminatory ability.

Be well,

Howard

Adding a comment about trees... Claudia commented about instability of trees which have been shown that a least 20000 balanced observations are needed to trees become stable. But the major problem with trees (cart ou conditional trees) is that in general they are less accurate then other regression techniques (MARS, GAM, neural networks, logistic etc) when measured by total error or ROC AUC. Therefore if you still thinking about trees I suggest you replace or compare this analysis with other techniques such as neural network or logistic regression. In the clinical scenario logistic regression has the advantage to allow overall scores and construct a overall probabilities charts and nomograms when their coefficients are available, while neural networks need computers to get the predicted probabilities.

Also I must insist that you should consider not to categorize the predictors. As I mentioned before and Claudia repeated, categorizing is just loss of information. If take a look around you will find many papers about this issue.

Altman. Dangers of using "optimal" cutpoints in the evaluation of prognostic factors. Journal of the National Cancer Institute. 86. 11. 829-835. 1994

At last, Howard mentioned that there other model performance's measures then ROC AUC. I do agree with him, and currently there is some discussion about decision curves using regret and net benefit along with traditional measures.

http://www.sciencedirect.com/science/article/pii/S1885585711003604

http://www.biomedcentral.com/1472-6947/10/51

http://journals.lww.com/epidem/pages/articleviewer.aspx?year=2010&issue=01000&article=00022&type=abstract

Again, this discussion seems to be flowing around clinical prediction models. There is an excelent book on this topic. With several examples with R code. I suggest you take a look to check if this is what (look above at a link I provided).

Regards,

i am happy to see that a very fruitful discussion is going on related to the statistical tools/techniques for clinical prediction and diagnostic classification.

I am thankful to all the learned contributors to bringing to fore the various issues involved in such analyses and the care that should be taken of.

There is no doubt that this discussion has shown several possibilities and approaches to handle the issue under discussion.

However, with due respect to all the contributors, I would like to share that for a novice like me it has now become more difficult to decide what to do. Now I understand and has come to know that the analysis that I was planning is likely to suffer from a variety of problems and several statistical issues are inherent in the method of my choice.

With the permission of all the esteemed contributors, I would once again request for a more specific help in the light of my original question. Please assume that despite all the limitations of the ROC curve analysis, I want to handle my research problem only through this statistical technique then what are the options and how I should proceed. What software and/or data analytic strategy (specifically meant for ROC) are available for this.

I am happy to see that this thread has really stimulated a very organized and meaningful discussion and experts are still joining it and contributing to it.

Hope some more specific and concrete suggestions with link to resources to carry out such analyses would come.

I I would once again say a huge THANK YOU to all the esteemed contributors and especially to Drs. Pedro Brasil and Claudia Beleites for their valuable contributions.

http://www.clinchem.org/content/49/3/433.full.pdf

If you are familiar with R, the packages pROC and ROCR are nice options. However I believe these are not suitable for estimating decision thresholds.

http://metz-roc.uchicago.edu/MetzROC/software/software

Thanks Pedro for sharing helpful links that I have already searched through Google before creating this thread.

I have also installed the ROCR package but as I work only with GUI versions of R as rcmdr, deducer etc., I have not yet been able to try the ROCR but it appears to provide many more options that are not available in other commercial software.

The various software that I have used till date for ROC, I found the SigmaPlot very good in some respect as it provides several information in the output such as comparison of the AUC, LR+, LR- and post test probability and also has option to set the pretest probability and cost.

The MedCalc is also a good software and provides certain other facilities that are not in the SigmaPlot or many other software. For instance, one may calculate sensitivity for a given level of specificity or vice-versa.

It also provides indirectly the facility to conduct MultiRoc type analysis (see the paper linked with my question) as one can do ROC for an indicator while pre-defining the level of another indicator.

For example, if one is working with two indicators or predictors (say X and Y) and conducted ROC for each of them. The results indicated that for X a cut-off 'c' gives the best trade-off between sensitivity and specificity and is associated with highest value of the Yoden index. Similarly, one finds that for Y a cut-off 'd' is equally good. Then using the MedCal one may spicify in the selection option to select cases with score higher than the cut-off say 'c' and perform ROC again for variable Y. Now, if this analysis gives a significantly greater AUC as compared to the AUC obtained for X and Y separately. Further, this new analysis yields a new cut-off for y say 'e'. Then one may conclude that an individual with a cut-off score of 'c' or higher on X AND a score of 'e' or higher on Y is more likely to be case of the diagnostic group with a given sensitivity and specificity.

Thanking you and all others once again with the request to share their views and links to enrich the discussion on ROC analysis. This will not be helpful only to me but a more number of members like me and the prospective members of the RG interested in ROC and Multivariate ROC.

Thanks to all for their contribution and sparing their invaluable time to help me.

Dear Dr. Pedro I wonder why you do not mention the TG-ROC. While searching the Google I came across the following link that contains information about the said programm and author's name displayed on this pate is probably yours.

http://hosho.ees.hokudai.ac.jp/~kubo/Rdoc/library/DiagnosisMed/html/TGROC.html

I would be happy and feel obliged if you can share some more information about it and how to use it.

Though, I am not well versed with R but hope your valuable comments and informative links may turn people like me to be an user of R.

Regards

Dear Rakesh and contributors,

Have you finally find a way to perform a ROC analysis using multiple indicators?

I would like to apply such approach to demonstrate which combination of scores shows the best balance between cost and efficacy in detecting Alzheimer's Disease.

I know some R (the basics) and I have checked ROCR and pROC packages but I can not work out if its possible to include more than one measure to plot the ROC.

a similar approach, but in a longitudinal study, can be seen in this free full text paper:

http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2613777/?report=classic

Any help would be highly appreciated.

Thank you

Combine using logistic regression, use predicted probabilities as test scores to determine cutoff via the Youden index.

Thanks Christos for your helpful suggestion. For the time being I have combined the indicators using a CFA model and have used the imputed score of the latent variable as combined indicator. For cut-off I have used the Youden index as well as the intersection of the curves of sensitivity and specificity.

Regards

Dear Gonzalo

I used optimal combination score as I have just mentioned in my comments in response to Christos Comments.

However, if indicators are few in number and the sample size of the reference category is large then using MedCalc feature one can examine the diagnostic efficacy of an indicator over and above the other

I received a personal message from a researcher regarding the method that I followed for the question under discussion. I am sharing my response to this query here with the hope that if there would be any problem with this approach then it will be resolved here by the esteemed members. Further, this will also be helpful to further this discussion. My response appears in quote.

"Use of predicted probabilities for ROC gives information about how well the linear combination of indicator variables distinguish between a case and a non-case. However, as far as I understand, this method will not help much in determining the cut-off scores in terms of Raw scores.

The method that I followed is detailed below.

First, I selected those indicators that worked significantly better than others by examining the significance of difference in terms of AUC. For this purpose I used Sigma Plot.

After, selecting the significantly better indicators, I conducted an exploratory factor analysis to see whether the indicators can be pooled in fewer yet meaningful factors. The obtained factor structure was then tested using confirmatory factor analytic approach (using AMOS) and if a good fit was noted then two approaches were followed to obtain a single score representing the linear combination of the variables forming a given factor. First approach used the latent factor score (using the option for getting imputed score in the AMOS) and the second approach followed the aggregate score (i.e. the sum of the variables constituting the given factor). The ROC was conducted using both the scores and the AUCs were compared for significance of difference. Further, the AUCs of the composite scores were also compared with the AUCs of individual indicators to establish that the linear combination does better than any of the single indicators. If the combination score performed better than any individual indicator then using the Yoden index as well as the intersection of the sensitivity and specificity the cutoff score for the composite score was computed. For this purpose we used Sigma Plot for computation and the SPSS for plotting the values (we used SPSS because we are well versed with it otherwise it can be done using any software including the Sigma Plot and even the Excel).

The aforesaid approach, however, will provide information that combination of various indicators perform better (or do not perform better) than any of the single indicator in making a diagnosis. It will not provide information about the combination of the individual cut-off scores of various indicators in making a better diagnosis as compared to any of the indicator alone. For this purpose, MedCal ( a software) can provide help. However, we did not use it in our own research as the sample size was not much large for this purpose.

If you are interested in this then I think the use of MedCalc may help you in getting your answer. The general approach (using the MedCalc) is to conduct the ROC for a single (best) indicator and determine the cut-off score. Then to filter the cases using this cutoff score (i.e., select cases with a score higher than or equal to the cutoff score and then add the next best indicator and perform the ROC and determine the cutoff score for this second indicator. The sensitivity and specificity associated with this second indicator (obtained from the cases having a score higher than or equal to the cutoff score on the first indicator) is in fact the sensitivity and specificity for the combination of the cutoff scores of the two indicators.

For instance, the analysis using the indicator X yielded an equal sensitivity and specificity for a cutoff score of say 10 or higher then one will select the cases with score 10 or higher on the variable X. Now on this sub-sample of cases one will perform ROC using another indicator (say Y). If the analysis revealed that a cutoff score of 12 on Y results in a sensitivity and specificity higher than the X alone (say equal sensitivity and specificity of 93%) then one may conclude that a score of 10 or higher on X combined with a score of 12 or higher on Y gives a better diagnostic accuracy ( with sensitivity and specificity of 93%) than X or Y alone. To support this conclusion the comparison of the AUCs of X, Y and the combination of X and Y will be required.

Hope this helps"

Thank you very much for sharing this information!

It would be great to read any paper that used such approaches. Do you have any reference?

Dear Gonzalo

The paper is in preparation and its submission may take some time as it is related to the PH.D. research of one of my student who is in the process of compiling the thesis.

Please keep us updated regarding the publication of your research.

To my experience the choice of cut-off due to max accuracy is not the optimal approach. The cut-off should be chosen according to the clinical application of the test and the "costs" of false positives and false negatives, respectively. E.g. if false positives should be avoided, one has to choose a cut-off at a high specificity level, let's say 95%. The comparison of single tests and combined test should also be performed at this level. Taking this example, one should compare sensitivities at a fixed specificity of 95%. The situation could occur, that the multiple test has a better AUC but in the clinical interesting range the improvement is not found!

Robert

Yesterday I posted my answer to your query BUT somehow it has not appeared

I used 150 cases (75 in each group) and conducted ROC with one indicator at a time -in some analyses the indicator variable was the total score on any scale or subscale while in others it was a composite score in form aggregate (sum)of several indicators. The selection of indicators for creating composite was either based on the observed accuracy of the individual indicators or the findings of CFA

Dear Rakesh,

In a previous post you have cited an answer from a researcher that suggested the following method.

"The general approach (using the MedCalc) is to conduct the ROC for a single (best) indicator and determine the cut-off score. Then to filter the cases using this cutoff score (i.e., select cases with a score higher than or equal to the cutoff score and then add the next best indicator and perform the ROC and determine the cutoff score for this second indicator. The sensitivity and specificity associated with this second indicator (obtained from the cases having a score higher than or equal to the cutoff score on the first indicator) is in fact the sensitivity and specificity for the combination of the cutoff scores of the two indicators."

We are currently applying it in our data from a neuropsychological battery. We want to support such approach with some references but we don't find it. Are you aware about any published paper that used this technique?

If it is possible and you feel that it is appropriate, can you disclose the name of the researcher that have made the comment?

Thank you very much!

Dear Gonzalo, I think the technique is described in this text: http://www.amazon.com/Recursive-Partitioning-Applications-Springer-Statistics/dp/1461426227/ref=sr_1_2?s=books&ie=UTF8&qid=1389370205&sr=1-2&keywords=recursive+partitioning+in+the+health+sciences

Dear Christos,

Thank you so much for your contribution.

I've been reviewing the preview of the book in Google books and a related lecture: c2s2.yale.edu/resources/lectures/LectureNotesOnRP.pdf

which covers chapters 1–4, 6, 8, and 9 of the book.

Unfortunately, I can't find what the approach that I am looking for, which in fact is the simplest.

Could you please be more specific?

Anyway I'm going to try to have full access to the book.

Thanks again

Use Yoden index for your own specificity and sensitivity cut off, then calculate NRI and IDI

Dear Sánchez-Benavides

Sorry for delay in responding to your query

I think that the method that I described in my earlier post (to which you are referring to) is based on the method suggested in the attached paper

I am using Stata, so how can I estimate Youden index, correct classification rate.

Kind regards

Mizanur

can we use IDI or NRI to decide the cut points?