DeCarlo ( Article Signal Detection Theory and Generalized Linear Models

Article Functions for traditional and multilevel approaches to signa...

It was interesting

I didn't know signal detection theory in physics was equivalent to diagnostic accuracy in medicine!

A logistic regression is a model that takes a value of a predictor (or the values of several different predictors) and returns an estimate for the probability of observing the outcome, given that predictor value. In medicine, the outcome coud be something like 1-year survival (will the patient still be alive in one year from now), and the predictor could be the presencence of a disease and comorbidities, the presence of other risk or beneficial factors, no matter whether they are binary (overweight: yes/no) or continuous (e.g. BMI), or some combination thereof. A logistic model would then be a tool that you can use to get an idea how likely a person with a given predictor values will still be alive after one year. Such models are used in research to find out if a particular perdictor (e.g. a treatment) has a noticable (positive) effect on the 1-year survival (given patients with same predictor values except treatment: can we conclude that the probability of a patient recieving the new treatment to survive the next year is higher than that of a patient recieving the standars (or a placebo) treatment?

A ROC is a function of a diagnostic test based on a measured quantity. The test returns either yes or no (it has a binary outcome), and this is outcome depends on whether or not the observed value is above (or below) some arbitrary value (the "cut-off" value). A very simple example is a test to determine the sex of an adult person (male/female, what is equivalent to [fe]male:yes/no) given the observed body height of that person. Say we use the test to identify a person as being male. You can choose any height value as a cut-off. Consider you use 100 cm as the cut off and the test returns "yes" (or "male") when the observed value is larger than that (and "no" or "female" otherwise). For this particular cur-off, the test would perform very badly, since almost all persons would be classified as being male (since almost all persons, no matter what their sex is, are taller than 100 cm). It would classify all men correctly, but it would also classify all women wrongly. With this cut-off, the test would have an excellent sensitivity (all men classified correctly) but a miserable specificity (all women are classified wrongly; the test does not separate males from females). You get the opposite picture when you choose a very large cut-off, say 250 cm. This test would now have a miserable sensitivity (all men are classified wrongly) and a perfect specificity (all women are classified correctly). For cut-off values in-between, you get varing combinations of sensitivity and specificity, depending on what cut-off value you chose. Ideally, you want a cut-off that gives you perfect sensitivity and at the same time a perfect specificity (so that all men and all women are classified correctly). In or example, such an ideal cut-off does not exist. No matter what value you chose, there will always be some men smaller and some women taller (and those would be mis-classified).

A ROC is a plot of the sensitivity and the specificity (typically the complement of the specificity, which is the false positive rate, is shown in these plots) for all possive cut-offs you could chose. You already know that this curve must connect the two points (perfect sens/no spec) and (no sens/perfect spec) for stupidly high and low cut-offs. The interesting part is how these points are connected. A perfect test would connect these points through the point (perferct sens/perfect spec), and a completely worthless test would connect these points by a straight line (every little gain in specificity if paied by an equal loss of sensitivity, and vice versa). Thus, one can use the area under the ROC to see how informative the diagnostic variable (for which you chose a cut-off) is regarding the classification. For a perfect test, this area is maximal (=1), and a completely worthless test will have a minimal area (=0.5). If the area is lower than 0.5, the test is also working, that is, it can distinguish men and women be their size, but the calssification is just opposite to what it should be.

An example for a ROC calssifying the sex of a person by their height is attached (left diagram). The data is taken from https://ourworldindata.org/human-height#height-is-normally-distributed, assuming a normal distribution with the given means and standard deviations, and assuming that a person is equally likely male or female.

One can use the ROC to determine a "good" cut-off. Notably, there is not "the one" or the "best" cut-off, as any choce with a better sensitivity will give you a worse specificity (and vice versa). So you have to balance sensitivity and specificity against each other, and how this should be done is a subject-matter question. Sometimes it is better to sacrifice sensitivity to have a better specificity, and sometimes it is better to do it the other way around. A "best" cut-off can be chosen only w.r.t. a given loss function that quantifies how much correct and wrong classifications "cost". The "best" cut-off would then be the value that leads to the lowest expected "cost". Sadly, such cost function are usually not known or available for any interesting problems.

The connection between logistic regression and ROCs is that you can use a logistic regression to do a classification, and the cut-off you can chose is the predicted probability of the outcome. If the logistic model increases sharply, you will get a large area under the ROC. The logistic model for the height example is shown in the right diagram. Both show the same thing from two different perspectives, as each height cut-off corresponds to a particular probability of the logistic model. In this simple case, we could either choose the height directly or the probability as predicted from the model. However, models may consider several predictors, and a ROC may be constructed choosing a probability cut-off.

Thanks Jochen Wilhelm

Very complete text and very nice figure

I understood that there is no constant indication for use of each (it is just a different perspective in your word).

I have also another idea. If we consider a 2by2 table, ROC has vertical approach (sensitivity and specificity) whereas Logistic regression has horizontal approach (PPV and NPV).

Fortunately, both of them can be used for adjusting covariates.

In evidence-based medicine (EBM) there are different domains of research:

Treatment

Prevention

Diagnosis

Prognosis

Etiology/harm

I understood that these domains are not absolute indications or contraindications for using ROC or logistic regression.

I don't subscribe that a "vertical approach" of a 2x2 table refers to the ROC. The table gives the classification result for one possible cut-off. The ROC is a function of all possible cut-offs.

You are correct that you can get the sens/spec as well as the PPV/NPV from a 2x2 table, but only the the cut-off (as the diagnostic criterium) that was used to generate the paritcular table. Choosing another cut-off (on the same data) would produce a different table. This dynamic is captered in the ROC.

Yes, you are right

My mean was that PPV and NPV cannot be calculated directly from ROC.

ROC curves provide estimates of sensitivity and specificity across the range of the predictor ie how will cutoffs discriminate the two outcome classes. When there is a single predictor this can be done usg the predictor alone without logistic regression. A logistic regression model produces predictions for how likely an individual is to be in each of the two outcome classes. It can do this for single predictors or it can combine them. However ROC curves only assses one aspect of the logistic regression predictions ie discrimination. It is possible the actual predictions may be inaccurate even if there is good discrimination (AUROC, sensitivity an specificity). Accuracy of the individual predictions from a logistic regression can be assessed using calibration techniques. Additionally all model assessment is best carried out using cross validation of in test/training splits. There is more info here https://www.youtube.com/watch?v=v_e85TDYj9Y

Thanks Bernard North for complete answer

Of course ROC can also be used for more than one predictor. Its Stata command is rocreg.

Regards!