I am trying to estimate how close two time series (experimental and theoretical) are. Have tried correlation but the time series are nonlinear, hence, i would like to know other techniques/index that will show how close two time series are.
You can use rank correlation if the problem is in the non-linear relationship between the values of series.
But if the problem is that both series have trend, you need to find first difference first and then use rank correlation to see if those first differences correlate with each other.
I think you could do this a different way. You'd run a regression on the two time series and then look at the coefficient for either level or trend difference. For example:
regress rate group year group*year
here, I regress the annual rate on the group, year, and interaction group * year. The coefficient of the interaction will be the difference in the trend between the two groups. Of course, if you have non-linear patterns, you could model that with the appropriate polynomial or spline.
Note: I did not introduce the issue of auto-correlation here. You may have to consider that, depending on the purpose of your study.
You need to use index that support non linear models since you are working on two non linear time series.I believe there exist non linear regression, also search for non linear correlation.
If the theoretical time series is the output of a mathematical model you could also try to fit the parameters of the model to the experimental time series and compare a) the fitted parameters with the theoretical parameters and b) the experimental time series with the output of the fitted model by computing the difference (i.e. the residual time series). In the ideal case the residual time series should contain only (more or less) white noise, otherwise either the model should be improved or some preprocessing of the experimental time series might be necessary.