Looking for a way to automatically determine if one R2 is better than another. We are using a second derivative analysis of the membrane potential during action potential to separate the acceleration at the axon initial segment vs that at the soma. So the second derivative normally gives two clearly separate peaks. No issue there. But in some cases there looks like one peak and in others there is a 'hump'. When doing polynomial curve fitting - in some cases I've gotten 0.98 for linear and 0.99 for a 4th degree polynomial fit. So the question is how to tell if a 0.99 R2 is significantly better than an R2 0.98? Or in other words how to tell definitively is there one peak or two?
Hi Kimberle,
When fitting polynomials, an equation (aka Model) with more coefficients will always give you a better fit than one with less. Hence the simple R^2 is not a reasonable way to assess model fitness. However, deciding what model best fits your data is not trivial. There are a large number of different approaches to this problem. The simplest is "Adjusted R^2", which I'll allow you to google, but basically, it takes into account the number of coefficients are in your model, and penalizes you for each additional one (and infinitely penalizes you if the number of coefficients is equal to the number of data points).
There are many other approaches, like the Akaike Information Criterion, the Baysian Information Criterion, and others. Indeed, I'm sure there is enough literature on "assessing model fitness" to make up a whole undergraduate course.
There is actually a better approach than all of these in this case, which is, we can derive an equation that should model the upswing of an action potential, because we know that action potentials follow Cable equations + Hodgkin Huxley kinetics. Now, I can't derive that equation off the top of my head (and I certainly cant do it at 9am on a Monday morning). But it should be possible, up to a certain accuracy at least. You see, often, when we fit data, we have no empirical way to predict what the data should look like, hence you have to fall back on things like Adjust R^2 and guess work. But here we know what the relationship between voltage and time should look like, and so we can pick a model without even looking at the data. In a perfect world, this is what I suggest you do.
That said, there are also classical inferential approaches to compare model fitness, and in this case, I would google "Stepwise Linear Regression". Essentially, you include higher order powers of your data (i.e. one column is the dv/dt, the next column is dv/dt^2, etc) and then the stepwise regression tries and includes each column, and sees which is the best. There are probably lots of other good ways, but the take away message here is, simple R^2 is not a good way to compare different models, and in a perfect world, you can figure out the model a priori.
Hi. - Not my field, so it was interesting to learn something from William. I just have one comment regarding your first sentence, William: It is actually possible to use too many coefficients in your polymomial, such that you do not get a 'better' fit. Please see the example at the beginning of the following:
https://theclevermachine.wordpress.com/2013/04/21/model-selection-underfitting-overfitting-and-the-bias-variance-tradeoff/
However, I do get your point, William, that people probably just aren't usually going to do that in the context here. Your point that using more coefficients than are justified is probably the bigger danger, because we under-appreciate the 'noise,' is well-taken.
Hooroo!
Well you can theoretically get a perfect fit, but assuming you don't get a perfect fit, adding more coefficients will always reduce the R^2. You're also right, that I was using the phrase "better fit" a bit loosely. I should have said "higher R^2". So what I should have said is "Assuming a less complex model does not produce an R^2 of 1, adding more coefficients to a polynomial model will always increase your R^2"
Dear Jacobs
The so high value of the R2 is problematic also. Higher value (more than 0.80) indicate that some of the variables are correlated i.e. multicollinearity problem exists (violation of principle of model building through regression analysis). I would suggest to select those models with value of R2 (range 0.75 -0.85, arbitrary), but theoretically and practically support your concept and results. Model validation is a must before using a model for future reference.
I think that is a dangerous statement Rajiv. You're right, in LOTS of cases very high R^2 values are signs of over-fitting. However, when you are recording time-series data from physiological processes, you can often come close to perfectly fitting the data. For instance, you can often fit the decay of a voltage transient with a single exponential and get an R^2 upwards of 0.95 because the membrane of a cell IS a capacitor, and does follow a first order differential equation. Similar statements can be made about drug elimination, ligand unbinding, and many other examples.
Simply put, high R^2 values are NOT necessarily a sign of bad model choices. It simply depends on the nature of the data.
I should say also that the data are already smoothed and also I am only fitting between the AP threshold point and the latest (in time) peak of the second derivative. So these are the reasons for the good fit. When there is clearly one peak - it is completely linear - and the fit is very good. When there are 2 peaks - it fits very well wtih 3-4 coefficients. It is in between these two that is the question. When there is a peak but no dip after the first peak - just a sort of plateau and then rise to a second peak - this is another example of a more difficult fit. But all of the fits I've done are in the high 90's.
You seem to be asking about specifics, in which case, can I get some specific examples, hopefully in some common file format I can open.
- Badges
- Science topic
- Similar topics
- Biological Science
- Microbiology
- Virology
- Virus