We are tackling exploiting a process simulator that receives some initial input data and outputs time series transition in parameters, and we are comparing these results and real data (using as answer).
However I have trouble in how to evaluate this. Some results show the good correspondence to the real data at the end of simulation but in the interim progress deviate from the read data. While other results show good correspondence at first, but gradually and slightly deviates from the read data, and in the end it results in lager error.
I have no idea about whether the former or the latter is good and want to know how to evaluate it in terms of mathematics.
I would appreciate it if you tell me a hint.
Best regards.,
In purely mathematical sense, what you want to know/evaluate is the distance between two sets, namely a set of observed vs. predicted values. Unfortunately the distance may be defined in many ways, producing different numerical values for the same case. One of the most popular is rms deviation (\sqrt{\sum (pi - oi)2} / n, where n is the number of pairs prediction--observation. It is popular because there are many books dealing with this problem as well as computer programs to adjust unknown parameters in order to get possibly small rms deviation. But for calibration purposes one prefers another distance: maxi |pi - oi|.
In purely mathematical sense, what you want to know/evaluate is the distance between two sets, namely a set of observed vs. predicted values. Unfortunately the distance may be defined in many ways, producing different numerical values for the same case. One of the most popular is rms deviation (\sqrt{\sum (pi - oi)2} / n, where n is the number of pairs prediction--observation. It is popular because there are many books dealing with this problem as well as computer programs to adjust unknown parameters in order to get possibly small rms deviation. But for calibration purposes one prefers another distance: maxi |pi - oi|.
Dear Daich,
Your problem reminds me of the issues we face when validating the outcomes of a simulation model against empirical data in the context of health economics. To me the key question is what you consider a "good" simulation result. Maybe this paper can be useful for you.
Good luck!
Isaac
Article A New Statistical Method to Determine the Degree of Validity...
This is a reliability question, and there are many methods you can consider. The two quantities you need to consider are the Difference and the Correlation. You would like to compare the simulated data to real data and show 1) the differences are small enough, and 2) the correlation is strong enough, that the results can be considered reliable.
The statistics for this are often fairly simple, but the interpretation can be difficult. Reliability is subjective to the application, and is is difficult to quantify how reliable the simulation needs to be before it is "good enough".
It's not surprising that the simulation results get worse over time; predicting the future is hard! You should consider how far ahead the simulation can still provide a suitably reliable answer.
The three authors have spoken well. I think your intention is error minimization in the output of the simulated data vs the real data. Taking the real data result as the bench mark, you could try parameter adjustment in the values assigned to the simulated data until you get the best result which could could give the corresponding best fit of the real data. This is almost synonymous to the suggestion of Dan Eastwood and the mathematical description of Marek Wojciech Gutowski.
- Badges
- Science topic
- Similar topics
- Mathematics