In ecology linear regression is often used. However, the results of these regressions are used to build classification metrices (don't ask me why) and often the regression model itself is not used.
Consider a linear model y~x, where y indicates ecosystem quality expressed as a continues value between 0 and 1 and x a continues factor affecting ecosystem quality (e.g total phosphorus). If the linear y~x model is created the continues value y is divided in five classes (yc1, ..., yc5) with equally widths (0.8). These classes are used by management to indicate the ecological quality states bad, poor, moderate, good and high. Understandably, we do not have a linear regression anymore and now deal with a classification issue. While the R-squared can appear high, a high number of classes determines the overlap of these. A high number of classes with large overlap decreases the accuracy and probability of misclassification of these states. I understand that the classification accuracy is depended of the model and I known why this happens. But, a simple equation might can highlight this very general issue. Is it possible from the R-squared published in literature to approximate the overlap or accuracy for any possible number of classes?
Dear Wim Kaijser ,
Your question is rather complicated. I kind of see what you are asking. I doubt whether it would be possible to accurately estimate the relationship between these constructs. My advice would be to run multiple simulations and observe the required relationship from them.
Hello Peter Samuels
I simulated the above in multiple loops and interpolated between classes (making the figure more appealing, the number of classes is discrete so not really correct). Although accuracy or error-rate is not the best measure of performance, it is the most direct and easiest to understand. Hence, in management either something is correct/incorrect no matter the unbalance of the data, prediction and number of classes.
Would it be sufficient to show it is very difficult to expect a high accuracy or low error-rate with a larger number of classes transforming the idea of regression to classification? Either we need to find a single predictor variable resulting in an strong "performance" , improve the linear model by fitting multiple predictors, switch to classification models (and lose the interpretability of the linear models, such as R-squared, coefficients, intercept and ability to us multiple different error-distributions) or adjust our expectations and reduce the number of classes (2-3). For this simplistic example in the figure: if we would like an accuracy of ~90% and to use five classes, the regression model needs and R-squared of >~90%. I consider a regression model with an R-squared of 60% already marvelous (based on the topic).
Wim Kaijser
I too find the question a little difficult. However I would give a couple of thoughts:
- Do we get a 'better' straight line from more than 2 points?
- Generally in statistics/math, if we alter the number of classes or class widths then key parameters such as averages will change
The first plot you show appears that the points have been applied with a shotgun. I tend to take anything of this with a large grain of NaCl...
Wim Kaijser before you go any further please be mindful that R2 grows with the number of regressors you include: you can for instance artificially increase it adding multiple permutations of the regressors.
Your question reminds me of how McKelvey & Zavoina construct their pseudoR2, their approach is in the opposite direction but it could be useful.
Dear Alan F Rawle I see that the question is somewhat unclear, I am sorry for this! I will update it an state it somehow simpler.
P.S. Shotguns are apparently favored in ecology. I guess its easier to catch wildlife since you always hit something(?)
Wim -
If you could explain the following, I think it might be helpful: "While the R-squared can appear high, a high number of classes determines the overlap of these. A high number of classes with large overlap increases the decreases the accuracy and probability of misclassification of these states." In particular, i do not know what "overlap" means.
For the model considered with y as ecosystem quality and a lone x as total phosphorus, since there are other variables which impact ecosystem quality in addition to phosphorus, i expect that you might have omitted variable bias, and certainly the model would perform better with a more relevant set of predictors. There is a bias-variance tradeoff such that you do not want either too few or too many independent variables, and further, you want exactly the right ones which work together best. If you are just looking to classify based on one variable, then it seems you miss how that variable works in concert with others. So I wonder how relevant it is to just look at phosphorus, regardless of other variables.
Using too many variables, one may also overfit to a particular sample, but that hardly seems an issue here. Just from a subject matter perspective, you need more independent variables.
If you plotted a graphical residual analysis, it appears that it would look like OLS regression here. That should not happen. For increased predicted y, you should have increased sigma of the estimated residuals. Omitted variable bias should amplify this. (I think sometimes overfitting will do this, but that is hardly the case either.) I can only speculate that a-bx here is an inadequate predicted-y. R may be relatively high, but that is not the whole picture. Phosphorus looks surprisingly highly relevant by itself, but it is an incomplete story.
Cheers - Jim
James R Knaub Some badly written text from my part. Perhaps this is better: "While the R-squared can appear high, a high number of classes increases the overlap between the classes. A high number of classes with large overlap decreases the accuracy and probability of misclassification."
The term overlap is some personal thing which I often check. If we calculate the probability of superiority (referred to as AUC), 0.5 is a 100% overlap. If the AUC is 0.0 or 1.0 then there is 0% overlap. So overlap = √[(AUC-0.5)*2]2. No classification model can accurately predict classes (only a single gradient) if the overlap = 100%.
I agree with you that a single variable is not sufficient. Yet, the question was if there is a method to approximate the accuracy/overlap for an n number of classes with R-squared obtained from literature, making some assumptions about the distribution. The example is purely hypothetical.
Another thing I noticed, besides the switch from regression to classification, is the rotation of the gradients. In the example below x is again phosphorus and y is ecosystem quality (A). Then the x and y are rotated, x becomes y and y becomes x (B). The new x-axis is divided in classes (states) of equal widths (C). However, the bad and high states are based on the "outliers" and only a few samples are in these classes. The moderate state has the majority of all samples but spans the total gradient. If the gradients were not rotated we had equally balanced classes and each state "occupied" a unique part of the gradient (D).
The rotation of the gradients "always" happens. I am not sure if we are now predicting "statistical artefacts" (the outliers) or actual meaningful states. I seem not able to place this "statistical artefact" into the perspective of "applied sciences" (I cannot wrap my head around it).
After some time, sometimes mulling over the issue, there are some "possibilities" to convert regression into a class issues. Although, they do not necessarily address my accuracy related question, it seems "somehow" possible to transform regression to an "grouped" effect-size issues via the residuals or otherwise (for lack of better terminology). For this I used different effect-size estimators. The AUC-one-vs-rest is some Frankenstein experiment. Interesting is that Eta-squared almost follows the R-squared perfectly. I still would like to figure out why. Perhaps someone has a direct understanding as to why? I added the script for someone who is still curious/interested (not sure if there are some mistakes in there though).
Alan F Rawle To come back to your comment about shotguns and ecological patterns. Actually firing a hypothetical shotgun, assuming a linear pattern between distance from the target and standard deviation (based on some arbitrary article). The shotgun seems to perform better (I do not know anything about shotguns). I ignored the type of shotgun, length of the barrel, the weight and shape of the pallets and the effect of number of pellets. And I needed to somehow standardize the difference of the units used in the different studies.
The examples in Fig A display three shots from a hypothetical shotgun fired from 22 m with an incremental increase in width and height. The first shot between 0-33%, second between 33-67% and third 67-100%. The right figure was published in a high ranking journal (which is fine as long as it is not oversold), but the data was not provided so I needed to extract the points by hand and I guess a few are missing. Its kinda funny, perhaps incorrect, but anyhow fun (p-value is added for extra effect).
Alan F Rawle I get more and more stressed out and frustrated by positivity and things I do not get. So ..., either I write everything in an flower covered day-journal drink beer and cry myself to sleep or ..., place my own ignorance, stupidity and things I do wrong into an satiric app: https://snwikaij.shinyapps.io/Debunked/
- Badges
- Science topic