For airflow and turbulence in enclosed environments:

Below 10 % - Good

Below 20 % - Acceptable

see page 14 - 15

Article Evaluation of Various Turbulence Models in Predicting Airflo...

about 10-15%

Dear Sunril,

This is a large question but a good question because in numerous paper people give comparison of model outputs / observation data without any error bars. Moreover the question depends on the parameters you consider and the time and space scale you consider

For basic variables (temperature, sea surface temperature,...) of course the individual error on these parameter are quite low (of the order of 0.1°C for daily to annual average values). So you can consider that model results / data can be compared on the same plot by two curves. It would be good perhaps to indicate the standart deviation to give another usefull figure.

But for quantities like surface fluxes (latent and sensible heat fluxes; wind stress) for instance, the problem is different. Because surface fluxes are affected by lots of errors and you have to give the accuraty of the fluxes as well as those of the model.In that case it is appropriate to compare model/data with error bars.

For instance, at a anchored buoy by supposing that the daily averaged errors on input parameter of a bulk algorithm are: 0.2°C for temperature, 0.1 for SST, , 0.03 for relative humidity (in percent) and 0.3 m/s for the wind, a calculation of errors (by finite differentiation of the formula) on the latent heat flux gives you: 14.6 W/m2. This might bethe order of magnitude of the difference between your estimated fluxes and your model output. So it is essential to give an indication of this quantity. Then you can judge if your comparison is "acceptable".

Best regards,

G.C.

At first I must thank professor Caniaux for his illustrative response.

I presume that the error depends on what we aim from the research. Therefore, it is needed to clear final target among lots of different parameters. In some sensitive cases 10-15% error (mentioned by Soheil) could strongly affect final results. But in my research field (earthquake and structural engineering) up to 10% is OK. Maybe the best way is checking out publications which have numerical verifications.

Kindly Regards,

Reza

It is actually dependent to the parameter type. However, in structural engineering, when an experimental test is compared with an numerical model, an error bellow 10% may be appropriate.

Best regards

Dr. S.M. Seyedpour

This is a very good question. However, it has numerous  and rather subjective answers. It largely depends upon the science question one may have. See below link. In this paper, a comprehensive guidance is available to facilitate model evaluation in terms of the accuracy of simulated data compared to measured flow and constituent values. 

http://swat.tamu.edu/media/90109/moriasimodeleval.pdf

For airflow and turbulence in enclosed environments:

Below 10 % - Good

Below 20 % - Acceptable

see page 14 - 15

Article Evaluation of Various Turbulence Models in Predicting Airflo...

In soil mechanics we are mostly calibrating the model by using laboratory test data (e.g triaxial test results) from initial state to failure - and in this case the model "must describe" the behaviour very well...difference is not many per cents  (after failure ???)...But then in real life when we have analysis before construction the difference between observations and predicted values could be even +-20%, and anyway the prediction is acceptable...these are dangerous words...It depends of course on the case. Thin homogenous soil layer is easier than thick non-homogeneous ground with several spoil layers....Difficulties are very often on the parameters not on the model....

This is an interesting topic in geomechanics.

In my opnion, it's derirable distinguishing between the case of Laboratory tests and boundary value problems.

For laboratory tests, the initial goemetry, and the hydraulic and static boundry conditions are known. Thus, the mismatch between numerical and experimental results should be minimized as much as possible. Even, inverse analysis techniques can be used to this purpose. In these cases, even a change or an improvement of theory or model could be required.

For boundary value problems, e.g. slope stability or foundations, the uncertainties affect all the quatities used as input in the model. Thus, the main issue is to understand which is the sensitivity of the model to your input data. Generally, you can consider your model satisfactory whether it is capable to reproduce (back-analyse) the observed behaviour as it concerns few selected outputs (e.g. displacement in a control point, the amount of a failing volume, etc.)

...just few considerations..

The answer to this question is - it depends!   For example, if you are calibrating an instrument for lab measurements, you are looking for R-square of 99.99, but when working with data from observational studies, you will be happy with an R-square of 50%. So I agree with Cuomo and Bao-Jie. 

A very important question, yet difficult to formulate a generic answer. As long as you adres variation only (so randomly distributed residues between model and experiment) I would say the acceptable limit is defined by the goal the results going to be used. The variation is largely explained by the inherent measuring uncertainty defined by the instrumentation and the measuring setup applied. In this case it is important to have some rough idea of the variability of the relevant processes and their relative contribution to the variation of the parameters you measure. 

When you mean what the acceptable systematic deviation between experiment and numerical result is, then the situation is different. When being faced with systematic deviation it is crucial to understand the cause (numerical errors, missing out some process etc.) in order to be able to judge the applicability of the result to some real world question you're trying to answer. For example, when using a database of some pipe network (e.g. water distribution) it is known that even after extensive validation small deviations between the real world and the model exist, but it is also known that when following  a strict set of validation steps the effect on the end results are within acceptable limits (i.e. you can use the results safely for design purposes.) In this example we also know we are using a, strickly speaking, wrong proces model: in most hydrodynamic models for pipe networks the world is assumed to be 2D (one space dimension and time) while the real world is 4D. Yet it has been shown over and over again that the results from the combination of this simplified process description and a not 100% correct database results in very usable engineering results.

well, hope this helps you a little

best regards

Francois Clemens

This depends upon the experiment accuracy and the range of numerical model simulate the research problem

Hi Khorashad,

For composite materials, a difference of 10% between experimental and numerical results thought to be acceptable.

I draw attention to a few more problems.

In many scientific papers, no one compared the gap between numerical predictions (NP) and experimental results (ER) with absolute absolute error (AE), achieved during field trials. If the latter is greater than the NP-ER, then the statement about the proximity of NP to the ER becomes meaningless, and the model must be redefined.

In addition, all methods of verification and validation of the model relate to optimizing the a posteriori error of the model of the phenomenon under study. The magnitude of a posteriori error is always greater than the threshold discrepancy [S. Rabinovich, "Guide to the expression of uncertainty in measurement", 3d edition, Springer International Publishing, USA, 2017). https://goo.gl/5bofbE], which can be calculated using the information approach [https://goo.gl/m3ukQi].

This question has many subjective answers. It depends on the modelling of the problem. Generally, less than 10% variation b/w experimental and numerical results is considered of a very good rating.

Between 10% - 15% --> Good

Between 15% - 20% --> Satisfactory

Between 20% - 25% --> Fair (depends on the measurments in the experiment)

greater than 25% --> Unsatisfactory

Dear Colleagues,

It is assumed that the experimental results represent the real behavior of the object under test with specific measuring errors. You have to be sure that this error is bound and lies within certain margin.

On the other side the simulation results represent the behavior of the same object based on its theoretical  model Then there are some procedure to get the simulation results:

- development of a physical model for the object

- development of a mathematical model for the object leading to system of equations.

- solving the system of equations

-post processing the the results of the solution to get the intended performance parameters.

The two last steps can be verified and the results of the solution get confidence. While the discrepancy between the real performance and the simulation lies in the difference between the real object and its assumed model either physical or mathematical.

So mostly the discrepancy lies between the the real object parameter and its physical and mathematical description specially if the other error sources are minimized. So, it is expected that as far the numerical solution is close description to the experimental performance the difference will be greatly reduced. It is so that the researchers elaborate the theoretical models to get close enough to the experimental results. T here is great advancement in this issue supported by the effective numerical solutions of the set of equations describing the performance of the object under investigation.

wish you success

In some articles, the authors describe specific test conditions and present a comparison between theory and experiment. A small absolute percentage difference between experiment and simulation, called by the authors “model accuracy within 5-10-15 %”, allows them to declare a good convergence of theory and experiment. On this occasion, I need to make a few comments.

1. In most cases, the authors do not even bother to calculate at the same time the difference between experiment and theory and the total uncertainty of the studied objective function, depending on a large number of variables. For example, I analyzed more than 1,500 articles published in the International Journal of Refrigeration over the past 6 years. It turned out that only in 20% of the articles, the authors found it necessary to compare the achieved experimental uncertainty EU and the difference between the theoretical TP and experimental ED data. Unfortunately, the situation is such that the authors' citation index is growing, and the practical benefits of their research are questionable.

2. If experimental uncertainty EU> | TP - ED |, the value of the proposed model is insignificant, and it is very risky to put it into practice. Thus, the stated correlation between experimental and calculated data does not guarantee that the choice of the model structure will be sufficiently complete.

3. “Statistical significance” between theoretical and experimental data is not sufficient proof of the correctness of the chosen model. A NECESSARY condition for the accuracy of the model is, above all, the SMALLNESS of the calculated total uncertainty of the objective function as compared with the gap between theory and experiment. This fundamental truth did not occupy an important place in the literature of engineering and physics, but should be the subject of serious discussion.

It should be less than 10%

Hello Boris Menin , I completely agree with point 1 of your reply, but I'm struggling trying to understand what you mean by " If experimental uncertainty EU> | TP - ED |, the value of the proposed model is insignificant ".

It means we can't have a numerical error smaller than the EU. Why?

Could you please explain it, maybe providing an example?

Thanks a lot!

Hello, Simona Vermiglio ,

Integrated Experimental Uncertainty EU (its definition is introduced, for example, in [Kim, H.S., et al. (2016) Flow Characteristics of Refrigerant and Oil Mixture in an

Oil Separator. International Journal of Refrigeration, 70, 206-218.

https://doi.org/10.1016/j.ijrefrig.2015.08.003]) of the studied objective function should always be less than the calculated difference between the theoretical curve (TD) and the set of experimental points (ED). In this case, the researcher can be sure of the plausibility of the proposed mathematical model. If EU> | TD - ED | , this means that the experimental errors and the threshold mismatch between the model and the studied object are large. Therefore, the apparent statistical proximity of theory and experiment is highly doubtful. In a wide strip of "blur" | TD - ED | the most diverse models have the right to exist, which in the most direct way reduces the value of the proposed model. Ultimately, the researcher, when proposing a model, consciously assumes that it best represents the object being studied and can be used for PRACTICAL purposes, and not to increase his citation index.

I draw your attention to the fact that the value of EU cannot be infinitesimal, up to the Heisenberg relation. Its smallest value is determined by the class of the model and the number of variables taken into account. Examples with explanations of this statement are introduced for different areas of physics and engineering:

1. Menin, B. (2020). Uncertainty Estimation of Refrigeration Equipment Using the Information Approach, Journal of Applied Mathematics and Physics, 8(1), 23-37. https://www.scirp.org/journal/Paperabs.aspx?PaperID=97483.

2. Menin, B. (2019). Hubble Constant Tension in Terms of Information Approach. Physical Science International Journal, 23(4), 1-15. https://doi.org/10.9734/psij/2019/v23i430165.

3. Menin, B.M. (2019). An Universal Approach for Choosing the Optimal Model when Studying any Physical Phenomenon, Physical Science & Biophysics Journal, 3(4), 1-9. https://medwinpublishers.com/PSBJ/PSBJ16000133.pdf.

4. Menin, B.M. (2019). The Problem of Identifying Possible Signals of Extra-Terrestrial Civilizations in the Framework of the Information-Based Method, Journal of Applied Mathematics and Physics, 07(10), 2157-2168. https://www.scirp.org/Journal/paperinformation.aspx?paperid=95549.

5. Menin, B. (2019). Is There a Relationship between Energy, Amount of Information and Temperature? Physical Science International Journal, 23(2), 1-9. http://www.journalpsij.com/index.php/PSIJ/article/view/30148/56562.

6. Menin, B. (2019). Calculation of Relative Uncertainty When Measuring Physical Constants: CODATA Technique vs Information Method, Physical Science International Journal, 22(4), 1-8. http://journalpsij.com/index.php/PSIJ/article/view/30136/56538.

Best regards Boris

Hello Boris Menin ,

I see, now it's clear to me.

Thanks a lot for your answer and for the documentation you provided.

Best regards.

Sunil That depends strongly on the models, experiments, and their impact or importance. In "hard" Physics, particle researchers require at least a 5 sigma

Thank you David

It should be less than 10%

It should be less than 10%