When conducting a logistic regression analysis in SPSS, a default threshold of 0.5 is used for the classification table. Consequently, individuals with a predicted probability < 0.5 are assigned to Group "0", while those with a predicted probability > 0.5 are assigned to Group "1". However, this threshold may not be the one that maximizes sensitivity and specificity. In other words, adjusting the threshold could potentially increase the overall accuracy of the model.
To explore this, I generated a ROC curve, which provides both the curve itself and the coordinates. I can choose a specific point on this curve.
My question now is, how do I translate from this ROC curve or its coordinates to the probability that I need to specify as the classification cutoff in SPSS (default: 0.50)? The value must naturally fall between 0 and 1.
Hello Manuel Leitner. The default cut-point you mentioned is a cut-point for the predicted probability (that the outcome = 1). From the documentation (with emphasis added):
CUT(value). Cutoff value for classification. A case is assigned to a group when the predicted event probability is greater than or equal to the cutoff value. The cutoff value affects the value of the dichotomous derived variable in the classification table, the predicted group (PGROUP on CASEWISE), and the classification plot (CLASSPLOT). The default cutoff value is 0.5. You can specify a value between 0 and 1 (0 < value < 1).
Source: https://www.ibm.com/docs/en/spss-statistics/28.0.0?topic=regression-criteria-subcommand-logistic-command
So I don't understand your 2nd question above. There is only one predicted probability value for a given model, no matter how many X (explanatory) variables the model has.
Back to your first question, yes, you can use the ROC table of coordinates that gives the best combination of sensitivity & specificity values for your situation and use the corresponding predicted probability value in the CUT option:
/CRITERIA(value)
I'm still not sure I understood your question completely, but I hope this helps.
Bruce Weaver Thank you for your detailed answer. However, what you mentioned is totally clear to me. My question addresses something different:
1. In the ROC coordinates table provided by SPSS, you get different X-values (values of the predictor) resulting in a specific True Positive (TP) and False Positive (FP) rate when used as cutoff values. However, these values are not which I need to plug into the SPSS GUI, because there the cutoff for the predicted probabilities (default: 0.5) is required. So my question was: can I get this predicted probability cutoff by using an X-value from the table and use it in the equation for the log.-reg. model?
And my 2nd question was about the following: when you have, for example, 2 predictors, you get 2 different curves in the ROC plot. So how can you find the "best cutoff" when having, for instance, 2 predictors (i.e. 2 ROC curves in the plot)?
You seem to want to convert a predicted probability cut-off value into a raw predictor variable cut-off value. I guess you could do that in a model that has only one predictor variable. But I don't think there is any way to do it if your model has two or more predictor variables.
But again, I may well have misunderstood you. In that case, it might help to clarify things if you explained what you want to do in the context of the following specific example. I used Youden's J to choose the "optimal" cut-point, although I am not a fan of Youden's J generally. ;-)
NEW FILE.
DATASET CLOSE ALL.
* Edit path on the GET FILE line to point to the
folder where the SPSS sample files are stored.
GET FILE = "C:\SPSSdata\bankloan.sav".
DATASET NAME raw.
* Estimate model with default as the DV.
* Save the predicted probabilities as PP1.
* Use the default cut(.5) for now.
LOGISTIC REGRESSION VARIABLES default
/METHOD=ENTER age employ income
/SAVE=PRED(PP1)
/CRITERIA=PIN(.05) POUT(.10) ITERATE(20) CUT(.5).
* OMS.
DATASET DECLARE roctable.
OMS
/SELECT TABLES
/IF COMMANDS=['ROC Curve'] SUBTYPES=['Coordinates of the Curve']
/DESTINATION FORMAT=SAV NUMBERED=TableNumber_
OUTFILE='roctable' VIEWER=NO.
ROC PP1 BY default (1)
/PLOT=CURVE(REFERENCE)
/PRINT=SE COORDINATES
/CRITERIA=CUTOFF(INCLUDE) TESTPOS(LARGE) DISTRIBUTION(FREE) CI(95)
/MISSING=EXCLUDE.
OMSEND.
DATASET ACTIVATE roctable.
COMPUTE Specificity = 1 - @1Specificity.
COMPUTE J = Sensitivity+Specificity-1.
DESCRIPTIVES Sensitivity Specificity J.
DATASET ACTIVATE roctable.
AGGREGATE
/OUTFILE=* MODE=ADDVARIABLES
/BREAK=
/J_max=MAX(J).
RENAME VARIABLES (PositiveifGreaterThanorEqualTo = Cut).
TEMPORARY.
SELECT IF J EQ J_max.
LIST Cut TO J_max.
* Re-estimate the original model, first with CUT(.5) then with
* CUT(.3164525). The second time, save predicted probabilities
* as PP2 to compare with PP1.
DATASET ACTIVATE raw.
LOGISTIC REGRESSION VARIABLES default
/METHOD=ENTER age employ income
/CRITERIA=PIN(.05) POUT(.10) ITERATE(20) CUT(.5).
LOGISTIC REGRESSION VARIABLES default
/METHOD=ENTER age employ income
/SAVE=PRED(PP2)
/CRITERIA=PIN(.05) POUT(.10) ITERATE(20) CUT(.3164525).
* Confirm that predicted probabilities are not affected by CUT() option.
COMPUTE PPdiff = PP1 - PP2.
DESCRIPTIVES PP1 PP2 PPdiff.
* Compare the Block 1 Classification Tables from those models.
Defining the optimal cutpoint depends upon what you are trying to maximise. If false-negative errors carry a severe penalty, for example (for example, missing a cancer tumour) then you want to maximise sensitivity, while false positive errors can carry a penalty in terms of the burden they place on subsequent testing (and the worry they cause people!).
There's a very good paper you might read that delves into this, and proposes some solutions. Article Defining an Optimal Cut-Point Value in ROC Analysis: An Alte...
Right on, Ronán Michael Conroy. That's why I said I am not a fan of Youden's J (or Index) generally. It assumes that false positives and false negatives are equally costly. That may be true in some situations, but in many other situations (like those you described), it is clearly not true. ;-)
Dear Bruce Weaver and Ronán Michael Conroy, thank you for your answers. I totally get what you mean and why you're not a fan of Youden's index in general. Your answers are all totally valid, but unfortunately they haven't solved my initial issue so far.
Regarding your answer:
"You seem to want to convert a predicted probability cut-off value into a raw predictor variable cut-off value."
-> It's actually the other way around. SPSS provides the raw predictor variable cut-off value (if the predictor is weight, for instance, it would be kg or pounds), but I need to specify the predicted probability cut-off value (value between 0 and 1). You can see that on the picture I attatched. I get the True Positive Rate (Sensitivity) and False Positive Rate (1-Specificity) for every raw-score cut-off value of the predictor.
However, when running a logistic regression analysis in SPSS via Analyze >> Regression >> Binary logistic and selecting the >> Option button, you need to provide a classification cut-off based on the predicted probabilities of the outcome (y=1) occuring, NOT the raw scores as provided in the ROC curve coordinates table.
So, to my knowledge, I just need to plug in the raw score value (e.g. 30) into the equation for the logistic regression model p (y=1) = 1/(1+e^-(b0 + b1 *X1...)) to get the predicted probability cutoff, which I need to run the analysis in SPSS.
So, what I wanted to know is, if that is right or wrong?
THANK YOU for your help!
If you are talking about the CUT(value) option, yes, you must specify a predicted probability cut-point, not the value of some predictor variable.
It would help immensely, IMO, if you shared the command syntax for what you are doing. I think that the only way you can get an ROC curve using a raw predictor variable as the "test" variable rather than the predicted probability is if you have a one-predictor logistic regression model. But as shown below, you may have to reverse-scale the predictor to get the same ROC curve you get using the predicted probabilities from the logit model.
NEW FILE.
DATASET CLOSE ALL.
* Edit path on the GET FILE line to point to the
folder where the SPSS sample files are stored.
GET FILE = "C:\SPSSdata\bankloan.sav".
DATASET NAME raw.
* Estimate model with age as the only predictor.
* Save predicted probilities to PPage.
LOGISTIC REGRESSION VARIABLES default
/METHOD=ENTER age
/SAVE=PRED(PPage)
/CRITERIA=PIN(.05) POUT(.10) ITERATE(20) CUT(.5).
* Scatterplot with X = Age, Y = predicted probability.
GRAPH
/SCATTERPLOT(BIVAR)=age WITH PPage
/MISSING=LISTWISE.
* ROC curve using PPage as the Test variable.
ROC PPage BY default (1)
/PLOT=CURVE(REFERENCE)
/PRINT=SE COORDINATES
/CRITERIA=CUTOFF(INCLUDE) TESTPOS(LARGE) DISTRIBUTION(FREE) CI(95)
/MISSING=EXCLUDE.
* ROC curve using Age as the Test variable.
ROC age BY default (1)
/PLOT=CURVE(REFERENCE)
/PRINT=SE COORDINATES
/CRITERIA=CUTOFF(INCLUDE) TESTPOS(LARGE) DISTRIBUTION(FREE) CI(95)
/MISSING=EXCLUDE.
* Curve is upsided down, so use TESTPOS(SMALL).
ROC age BY default (1)
/PLOT=CURVE(REFERENCE)
/PRINT=SE COORDINATES
/CRITERIA=CUTOFF(INCLUDE) TESTPOS(SMALL) DISTRIBUTION(FREE) CI(95)
/MISSING=EXCLUDE.
PS - Using your variable Leucocytes, my previous example would become this:
LOGISTIC REGRESSION VARIABLES eventvar
/METHOD=ENTER Leucocytes
/SAVE=PRED(PPleuc)
/CRITERIA=PIN(.05) POUT(.10) ITERATE(20) CUT(.5).
ROC Leucocytes BY eventvar (1)
/PLOT=CURVE(REFERENCE)
/PRINT=SE COORDINATES
/CRITERIA=CUTOFF(INCLUDE) TESTPOS(LARGE) DISTRIBUTION(FREE) CI(95)
/MISSING=EXCLUDE.
ROC PPleuc BY eventvar (1)
/PLOT=CURVE(REFERENCE)
/PRINT=SE COORDINATES
/CRITERIA=CUTOFF(INCLUDE) TESTPOS(LARGE) DISTRIBUTION(FREE) CI(95)
/MISSING=EXCLUDE.
LOGISTIC REGRESSION VARIABLES eventvar
/METHOD=ENTER Leucocytes
/CRITERIA=PIN(.05) POUT(.10) ITERATE(20) CUT(PPvalue).
In the final logistic regression command, insert the predicted probability value that corresponds to the leucocyte value you want to use (in order to generate the desired classification table).
Thank you so much for all your detailed answers.
I've now managed to solve my problem on my own too. It was, in essence, quite simple. I always wanted to obtain the cut-off score to use in SPSS; specifically, the predicted probabilities cut-off score. However, I consistently used the raw values variable as the test variable in the ROC curve command, which naturally resulted in cut-off points for raw values only (e.g., what is the TP and FP rate when using a leucocytes cut-off of 17). Now, however, I've saved the predicted probabilities for my model and used them (the predicted probabilities) as the test variable. Consequently, I now have cut-off points for the predicted probabilities in my ROC curve coordinates (e.g., what is the TP and FP rate when using this predicted probabilities cut-off), which I need to specify for classification in SPSS. Thanks a lot for the numerous responses!
- Badges
- Science topic
- Similar topics
- Mathematics