In the first case you use finite differnce formula for the derivative evaluation and the approximation error is about O(delta). This is the reason why your numerical result is differ than the exact result.

Hi Levin, thank you for your feedback, but it is off by a factor 2.

This is something I do often. I use numerical derivatives first because I can program them quickly. When I have time I replace them by analytical derivatives and compare values. I have never seen something off by a factor of 2. Differences are usually on the 6th or 7th decimal.

Hi Hazem,

I don't see any contradiction.

Let us for shot consider only X. Then

(SSQ(at a + delta) - SSQ)/delta =

[(X Calc(at a + delta) - X exp)2 - (X Calc(at a) - X exp)2]/delta =

[X Calc(at a + delta) - X Calc(at a)]*

[X Calc(at a + delta) + X Calc(at a) - 2X exp]/delta ->

(supposing that X Calc is continuous function) ->

2[X Calc(at a) - X exp]*d X Calc/da

Hi Victor, thank you for your feedback. You're right, there is no contradiction when you derive the equations. The contradiction is in the computed values. I have one subroutine which calculates the derivative numerically and another whihc calculates it analytically and the difference is a factor of 2. I am failing to find a bug. So I started wondering if there is a trick in taking the analytical derivative of SSQ.

I would only suggest to test both this subroutines using a simple function.

That should be easy. Thanks

Hi Hazem, I don't want to address your question directly but rather point out that numerical differentiation is a very unstable numerical process. This may be the cause of the problem. A reference is p. 295 of Conte and deBoor, Elementary Numerical Analysis, 3rd ed. McGraw-Hill. Hope things work out. Best, david

Hi Hazem, It is obvious that you have a bug either in your analytical differentiation, or in your numerical calculations. Both methods should give the same result (the relative error should e very small). Notice also that it is conventional to calculate the residuals as XExp - XCalc, and not

XCalc - Xexp.

A comment to David: numerical differentiation of data is very unstable and typically filtering techniques, e.g. Savitzky-Golay, are used. But properly done numerical differentiation applied on well defined smooth functions is stable.

For numerical differentiation, you should prefer symmetric two-sided difference quotient f'(x) ~ (f(x+h)-f(x-h))/(2h) instead of the ordinary single sided f'(x) ~ (f(x+h)-f(x))/h. You can consult any textbook of numerical analysis to see how much better it is.

Another thing to consider is the correct step size h. A good rule of thumb is to use h is near 10^-7*x. However, that depends on the so-called machine epsilon which in turn depends both on hardware and software. And, of course, you cannot use the rule of thumb is x is exactly zero.

The following R code shows how it goes with simple curve fitting problem y = a*exp(-b*t) using synthetic data:

t