If your function locally does not look smooth, then probably your model (using a smooth function) is not appropriate. In principle, there are two ways to approach that problem of a function being badly determined locally.

Actually you can model the function as f(x) + \nu(C), where \eps is a random variable with mean 0 and covariance matrix C and use probabilistic methods to cope with the noise \nu. This can usually be done when dealing with measurements (in reality those measurements do not represent a real stochastic process but rather a smooth function, whose values cannot be determined exactly but only with limited accuracy, and the errors are often normally distributed).

Alternatively, you can represent the function as f(t, B_t), where B_t is a Wiener process, i.e. a Brownian motion. Then you get into the realm of stochastic processes, and you might want to look at Ito calculus and the Ito formula, and you might want to look at the mathematical machinery that was developed for the stochastic Navier stokes equation (there you get stochastic versions of div, curl, etc.)

Thank you so much, Hermann, for your answer. Do you have a paper discussing this issue? If yes, please kindly send to me in my email: [email protected]. I would really appreciate that. Thanks

If I remember correctly, I first stumbled over the Wiener process/curl in the book "Mathematics of Two-Dimensional Turbulence" by Sergei Kuksin and Armen Shirikyan. On Itô calculus you can usually find plenty of information in books on stochastic differential equations, e.g., the book "Stochastic Differential Equations" by Bernt Øksendal.

The other model f(x)+\nu(C) is rather a fitting problem than a stochastic calculus problem. In the second case you assume for your modelling that you observe some continuous (differentiable) process, but your observations are inaccurate. In that case you would try to estimate the function by some Least-Squares or Chebysheff approach (depending on your error model) and afterwards work with the smooth function f, so you do not have to develop a notion of indeterminacy for the differentiation. The indeterminacy is taken care of by the estimation of f. Nonlinear regression and nonlinear least squares is even implemented in various standard software packages like Matlab and R (see, e.g. http://cran.r-project.org/doc/contrib/Fox-Companion/appendix-nonlinear-regression.pdf or http://www.mathworks.de/de/help/optim/nonlinear-least-squares-curve-fitting.html).

@Hermann, thank you so much for your explanation. Ok i will try to search for Ito calculus, however i should admit that if it is possible i would avoid the use of SDE. So perhaps the second method seems interesting.

Nonetheless, is it possible to express the second method not as nonlinear regression but as differential calculus? Thanks

I do not think that the second method can be easily expressed as differential calculus but I honestly have not thought about it.

Why do you want to use calculus in places where it is clearly not appropriate? The assumption that Brownian motion is stochastic is also suspect. Would deterministic chaos and iterative maps be more effective?

@Dear Hermann and Andy: thank you so much for sharing your opinions. If it is possible i would try to stay away from stochastic, because is is very complicated to me. And so does with brownian motion, as i read some papers saying that for real stock price data it is best to use fractional brownian motion (fbm) instead of brownian motion. So i think, as far as it is possible i will try to check the smooth manifold assumption in differential calculus. Perhaps we should invoke a fractal manifold instead? See the link to Ben Adda's paper below. But to be frank, i do not yet find any idea how to introduce indeterminacy in fractal maniofold. Any other idea perhaps? Thanks

http://arxiv.org/abs/0711.3582

Victor, The smooth manifold (continum) hypothesis is mathematically invalid, since Turin proved that almost all the real numbers are non-computable, that is there is no algorythm to reach them. A trajectory on a smooth manifold must visit every point so provide an algorythm for those points  for which there is no possible algorythm. Proof by contradiction. This means we can only proceed from one computable point to another which without giving our test particle a knowledge of points it has not visited means even jumps between points. If there is a binary division of possible points into peak and trough then the motion of a test particle is wavelike (deBroglie), this being the only mathematically valid form of motion. Deciding which set of points on the manifold is simply a matter of subset selection. We have now quabtised space with a field dependent quantum, that is half the deBroglie wavelength. This provides you with the fuzziness you are looking for which is consistent in scale with the Heisenberg uncertainty.

Can the space quantisation, ds, be equated with the differential "ds" of calculus is a matter of checking  that stopping the taking of limits at half the deBroglie wavelength does not produce a different result from ds tends to 0. Planck's method of discovery of h is a fine example of the first scepticism of calculus. Are its assumptions valid in each instance of use?

Working with a fractal manifold would a somewhat difficult task.

@Andy: thank you for your answer. Yes, i agree with you that may be there is limit to differential ds in calculus, and also that fractal manifold is difficult. But can you suggest how to introduce a measure of indeterminacy into differentiation? Perhaps there is a paper or book that i am not aware of. Best wishes