Use One-way Analysis of Variance (ANOVA) if the groups you are about to compare are from three and above. Where the comparison will be between two groups, use independent t-test. If u wish to compare two independent variables with each have sub-levels, use two-way ANOVA

If the data you have pre-processed you can try to use WEKA to analyse your data. But if you are good in programming try using R or Phyton languages.

If you want to find out if there are actually significant variations between different configurations of your model, I would recommend ANOVA

Use One-way Analysis of Variance (ANOVA) if the groups you are about to compare are from three and above. Where the comparison will be between two groups, use independent t-test. If u wish to compare two independent variables with each have sub-levels, use two-way ANOVA

Are you comparing different runs of the model, or comparing the model output to case studies?

Pure model: If the model runs quickly (a minute or so each run), then I would run the model a large number of times for each configuration and graph the variability in outcome for each setting. Almost by definition and change you make to model parameters will have a significant effect on model outcome. A statistical test is not necessary. You could set the model to run on several computers and simply wait a week or two while the program runs in the background. Most computers are used quite inefficiently because the humans cannot keep up with computer processing power. As I type this, most of my desktop computer processor is idle.

If the model takes days to run one iteration, then the above will not work (unless you get a more powerful computer and optimize code). Here you will run the model 20 times for each configuration and treat these as replicates. Analysis proceeds as for any experiment.

If the model is compared to case studies (the ones used to build the model but better if new case studies), then you need to decide on specific points for comparison. How do you define "success" for the model. You then can score outcomes as success or failure and model appropriately.

Thank you all for your recommendations, i am researching more about all of your suggestions and i shall reach out if i have any more questions.

Siram:

It’s a big trouble to answer an statistic question without enough information and many may falls in the temptation to do it. You must to start from the beginning and explain clearly your study design, take care of the following items:

1. What are the research questions and hypothesis?

2. How was the study design? Define the treatments and the response variables.

3. How many replicates were done?

4. What would be the effect size that it can be considered significant? At what level and power?

5. Have the responses a normal distribution and homoscedasticity?

I can comment that the statistic solution could be set depending on the answers to the questions above, even if it could be anova, a new question is what kind of anova should applicate, for example in your study it seems that you take measures in diferent times (repeated measures?). The application of an specific method can be possible if you have enough replicates. And last, you must to consider if one of the responses could not to have a normal distribution (bacterial counts in time, for example).

The analysis plan of one research must be a part of the study design and it must to be considered before to start the experiments, the reason is to have an statistic proposal that define what analysis will be better under the experimental conditions and not to try to adjust some kind of analysis after, when the researcher asks himself, "Now, what will I do?" And the crude answer (strictly statsisticaly) could be: Nothing.

It may depend on your data. If you obtained your results form measurements of the same sample at different times, it will need to be analyzed as a time series. This is because the results at different time points may not be statistically independent and this fact would violate an important assumption with respect to ANOVA. Arima models can be used for time series plots and analysis.

https://datascienceplus.com/time-series-analysis-using-arima-model-in-r/

The arima model is included in the stats and TSA packages in R. If the sampling at different times involved the use of independent samples you may get by with ANOVA if all of the other assumptions for this type of analysis are also met.

For choosing the Correct Statistical Test in SAS, Stata, SPSS and R ... see

https://stats.idre.ucla.edu/other/mult-pkg/whatstat/

I think you would set up models with proper parameters over a period of time and compare these models based on estimated values of parameters if I understood your problem correctly.