Hello Edin, they all needed. I don't think we should say one is better than the other. All validation and verification methods are approaching from different dimensions to the model in order to measure its fitness for decision making on behalf of real world system. Regards.

Hello Ufuk,

thank you for your answer to my question.

I use always a combination of different methods (Animation, Desk Checking, Submodel Testing and Historical Data Validation) for validation and verifaikation of discrete event simulation, but it is difficult to say how good (%) the model describes the real situation.

I would like to know if anyone takes a different approach?

Regards,

Edin

Some very interesting issues discuss here from a ABM in contrasts to DES

https://www.researchgate.net/profile/C_Macal/publications

Hypothesis testing is good when you need to validate any data but depends on type of data also. I do discrete event simulation and validation. You also can take a look in my paper.

Dear Edin

Let us all remind a point: we can NEVER prove something (see Popper & Bachelard on epistemological issues) whatever the scientific discipline. More globally, we cannot prove something is true.

therefore, only two ways :

1) You can only prove something is false.

2) You can only gain confidence on you model/hypothesis/theory (a model is a theory and  a theory is a hypothesis). How? By repeatedly test along different gradients (time, space, space scales, time scales and parameters used in the model) to see if your model/hypothesis/theory works (=fits under some criteria, limits of acceptance and circumstance).

So,

Ufuk is right: all may be needed and all are nice. But:

Sensitivity analysis is not a validation test. It is a validation method  through a genericness test. All models are limited in their validity extent and this sensitivity analysis is nice to see this extent. So, I suggest to use it always! If yr model is too robust on parameters that defines your research question, so your model is useless. if your model is too sensitive on selected factors, it shows something but on a too small extent. Both tendencies do not show anything on the valour of your model but shows something on its utility.

(Event validity: is this when you artificially create a shock and see if it expands according to hypotheses = margin behaviour? it is both sensitivity test to not a parameter but a parameter variability (=std along time) and scenario testing.

We do not have time to do everything while researching. We have to select methods. And de facto accepting a model/hypothesis/theory is a social action. Therefore, it depends on the level of acceptance in your research community and only based on that. For instance, modelling rural societies implies that significance is good if it goes beyond 50% at 0.05. It is good enough. In Physics, less parameters and variables, the required degree of significance is higher. So, I suggest to select the gradient which is the most important for you = ... Time?

a) So, apply it on different past periods (= different combination of parameters facing macro data) = Historical Data Validation

b) So, apply it on different future periods (= prospective modelling). 2 possibilities:

* artificial normative scenarios

* historical tendencies + most probable/valuable scenarios (business-as-usual, public policies, economic/social/ecological events...) inducing dynamics = scenarios = Hypothesis testing, as Kanticha said

The selection of the method is also a social action among your scientific community but also is induced by your research question. Your question is precise: along discrete time steps. Time seems very important for your research. So, I suggest to test the valour of your time step = the time scale you choose

Change it and see if, ceteris paribus, results and tendencies are the same.

Conclusion:

a) Your "validation"-confidence-building procedure depends on your research question and your objectives. So, there is not one procedure. However, once you have your subject of research, your object of research, your research question and your postulates. you have your "validation"-confidence-building procedure to assess.

b) separate clearly calibration and validation. a common procedure is to test different values of "micro"  unknown parameters to see which one fits better at the macro level = calibration, then presents it as a sensitivity analysis = validation; it is false

c) based on what you said,

* sensitivity analysis for genericness

* historical confrontation with data

* prospective modelling

* test of the valour of the model by changing the time scale (usually by testing on a less precise scale)

Hope it will be useful: my thoughts are also a model!! a cognitive one

cheers

Have a look at the Winter Simulation Conference Proceedings Repository. There are many good papers on the topic (looking at the topic from a theoretical and a practical perspective).

http://informs-sim.org/

When I teach this topic to my students I also refer them to Stewart Robinson's book "Simulation: The Practice of Model Development and Use" where this topic is discussed in very much detail for DES.

Dear,

I thank you for very useful information.

Since my DES models (10-25 objects) have per object (manufacturing machine) between 5-10 pseudorandom number generators (example with 163 pseudorandom number generators: https://www.youtube.com/watch?v=6FWumJmlCIA ) process for validation and verification of DES model is very complicated and difficult.

I'm looking for new non-standard methods for validation and verification of DES models, that provide reliable results.

A good book for this topic is: Rabe M., Spieckermann S., Wenzel S.: "Verifikation und Validierung für die Simulation in Produktion und Logistik - Vorgehensmodelle und Techniken", Springer-Verlag Berlin Heidelberg, ISBN 978-3-540-35281-5, e-ISBN 978-3-540-35282-2, Berlin, 2008.

Regards,

Edin