Dear Florian,

I would like to add a few more thoughts to that question (and unfortunately not the answers to them :)). I think only focusing on the number of parameters can lead to fallacies concerning your questions. I guess the number of parameters is in many cases a good indicator of the complexity of the model and thus might suggest a larger number of model simulations to calibrate a model, but factors such as the number of model parameters that are sensitive for the simulated process, or the correlation of model parameters are essential and can complicate or simplify the problem.

Concerning the number of parameters, I guess the actual number of parameters for the distributed model is even much higher as the set of 18 parameters is provided for each spatial unit individually?

In general I do not fancy the idea of a fixed number of simulations to calibrate a model but rather assess whether a model is calibrated or not, based on multiple objective criteria. I find it more beneficial, to assess the benefits and trade offs of different models individually, e.g. better process representation of the more complex model (if this is really the case) with the trade off of longer computation times and a larger number of required simulations to calibrate the model.

But this comment just adds I further personal opinion to this thread :).

Cheers Christoph

Hi Christoph,

thank you for your thoughts. I am aware that a pure focus on parameters can lead to a bias. Still, I think that the hydrological community does often exactly that and therefore I would be curious how other people handle that problem.

The models I mentioned are not an example from my lab, but just something I made up to make my point. I changed distributed to semi-distributed to avoid further confusion^^

As far as I understand your answer you calibrate your models until they have reached a pre-defined threshold. I'd say this is quite similar to option two and three, as I guess that the more complex your model is the more runs it will need until it is finally calibrated.

This also adresses another point I am interested in. When you calibrated like that, how do you decide on the number of runs you look at? Because when you stop the moment one run makes it above your threshold, you only have one run and cannot look at uncertainties. But if you wait until x runs have made it above your threshold and look at those, you just have introduced another arbitrary choice.

I value your personal opinion about this topic, as I have the impression that it all comes down to that (or at least partly), because this still seems to be an ongoing debate, where no one has found a conclusive answer yet. Therefore, I am happy about every opinion I can hear about that!

Hi Florian,

yes I absolutely agree that this is and I guess forever will be an ongoing debate that has no universal solution. We have similar discussions at our Institute and as you said in the end it comes down to personal opinions. Me and my colleagues for instance follow completely contrary approaches in model calibration. While some of my colleagues optimize the model parametrization and accept a single parametrization to be the final best solution I strongly advocate for model ensembles, that are (more or less) equally able to simulate the variable of interest. Thus, in the end I always accept several model parametrizations to be behavioral.

Concerning the calibration workflow, when I honestly think about what affects my sampling design decisions the most, I have to admit it is the computation time. I typically define the number of parameter sets based on the time budget I have. For example, I always do a global parameter sensitivity analysis, to identify the relevant model parameters for the simulated processes that I will then consider in the model calibration. These are usually quite computationally expensive and I typically design them that they run over the weekend and are finished Monday morning (honestly! :)). The following calibration I typically do sequentially, where I select the number of parameter sets I will run in a way that the runs are finished the next morning or the morning of one of the following days (depending on the run time). I assess these simulations and update the new parameter set based on what I learned from the previous simulations. Ultimately I will result in several parameter sets that I consider as behavioral.

So again I did not give any answer to what number of parameter sets I use in the calibration process :). This is more a very personal description of the workflow I am using in model calibration (and I think that is applied by many other modelers).

Cheers Christoph

Dear Dr Florian, Interesting question, perhaps my comment is not relevant. With due respect, however, it was unclear if the topic is on surface water, groundwater, or both, number of stations, data sets, and so on. It might be useful to see that there are recommendation since 1967 (Ward, RC) for surface hydrology and in 1979 (by Freeze and Cherry) for groundwater, on the fallacies of using lumped parameter calculations. I could add the longer of the calculation interval the less accurate the calculation could become. Perhaps we should put forward a motto by Rushton (2003) on groundwater modelling "Understanding before analysis". Please, do have from Mexico saludos,joel

Dear all, I agree Christoph that to relate the number of parameters with the number of runs needed for calibration perhaps is a partial view of the problem. As stated before the number of parameters could be associated to the number ans complexity of the simulated processes. Other question is the phenomenons to be simulated through the model. Other question is the objective function to be max. or minimise at the calibration process and the acceptable "treshold" o range to accept the calibration. Other issue to be accounted is the time of each run. I think that there is not a unique answer. Here, I am attaching three examples where a 54 parameter hydrologic model is calibrated manually, a two parameter integrated model (from rainfall to dam outflows) is calibrated trough thousands of runs (massive runs) and a stochastic rainfall model (5 or 7 parameters) automatically calibrated with an "optimize algorithm":

Article Risk-based methodology for parameter calibration of a reserv...

Article A Stochastic Model for the Spatial-Temporal Simulation of No...

WATER RESOURCES RESEARCH, VOL. 40, W11102, doi:10.1029/2004WR003218, 2004

Regards

Models with too many parameters are generally unwield and cumulative errors in parameters could result in unrealistic conclusions. The model, paraphrasing OCCAM should be as simple as possible but not too simple. The numbers depend on type model you are looking for. Trial error based on such techniques as MAXIMUM LIKELIHOOD, LEAST SQUARE METHOD etc. can be tried for parameter estimation. A no. of commercially available co,puter programs are available for such an estimation.

No, more complex models require more care to calibrate. Not only the exponentially larger number of possible parameter combinations calls for more evaluations, but the time per evaluation is probably much longer. So I totally agree with Christoph: you will probably be limited by your computation time quota; that is, do not chose a model complexity that would exceed your capacity to calibrate. Moreover, as Iraj said, complex models have more uncertainty and, because of equifinality, are much more prone to over-fitting. Using an ensemble of calibrated models, as Christoph suggested, perhaps by calibrating under a multi-objective setting, could help get a sense of the uncertainty for each parameter estimate. As to how many evaluations to perform for a given model, and barring unlimited computational time, I would look at the convergence of your objective functions vs. time: it will asymptotically approach the optimal score with ever-diminishing returns per evaluation. So Alvaro's idea for a threshold could be implemented for the rate of improvement in the last X number of evaluations: stop when the improvement is not significant.

Dear Florian, dear colleagues,

I have read the answers offered so far with interest. I don’t really have any solution to the original question but I do have some comments.

First, the number of model runs per parameter is likely to be relevant. In a ”one at a time” calibration strategy where only a single parameter is changed, exploration of the parameter space is likely to take longer for more highly parameterized models. It has been my experience that parameter space can be explored adequately when multiple parameters are allowed to vary, using either Latin Hypercube or simulated annealing approached. Thus, under some circumstances, it would be appropriate to calibrate each model using the same number of runs.

Second, in my work with highly parameterized models, I’ve found that only a few parameters seem to influence model behaviour. Some preliminary assessment of model sensitivity using e.g. the highest and lowest plausible value could be performed and the parameter importance assessed. This could help to focus the remaining computer time on sensitive or interesting parameters.

Third, different numbers of runs for different model types could be motivated if, as Felipe has stated, some sort of convergence criterion is used to assess when to stop simulations.

Sincerely,

Martyn

Hi Florian,

The best option, I would say is to use the maximum recommended number of simulation for the different models, optimization packages. Each model has a description about the minimum and maximum number of parameter iteration. In addition, the model uncertainty with the number of parameters depends on the model structural complexity.

The spatially explicit semi-distributed 14 parameter model may be complex unnecessarily or the other 3 and 6 parameter lumped models may be over simplified. Hence understanding the model and how it works will give you a way forward for this specific question.

To handle such problems; you may run each model with their maximum recommended iteration. See the attached article which compares 5 calibration uncertainty programs/packages each of which has a different recommended number of simulations.

Article Streamflow and Sediment Yield Prediction for Watershed Prior...

Hi Florian,

Information criteria like Akaike can be useful for these kind of questions. An available paper is:

Engelhardt, I. , De Aguinaga, J. , Mikat, H. , Schüth, C. and Liedl, R. (2014), Complexity vs. Simplicity: Groundwater Model Ranking Using Information Criteria. Groundwater, 52: 573-583. doi:10.1111/gwat.12080

More details in the PhD thesis from G. De Aguinaga here:

http://nbn-resolving.de/urn:nbn:de:bsz:14-qucosa-71814

Kind regards,

Thomas Reimann

A huge amount of param. to calibrate can be faced by implementing a grid-search of the possible param. values. From the grid, the best combination of parameters can be selected based on a cross-validation process. A sensitivity analysis can be also performed to refine the search. For this, the best way is chossing a methodology considering interactions between parameters. You can check the aforementioned at https://doi.org/10.3390/w10111519

Hi Florian,

Interesting thoughts on calibration procedure for different hydrological models with different levels of model structure complexity!

The idea is somehow recently discussed in the paper of de Boer-Euser et al. (2016) (among many others). They compared simulation results of different models for a catchment with the same calibration strategy. The study shows quite interesting results. I hope that you may find it related to your concern ;)

Looking beyond general metrics for model comparison – lessons from an international model intercomparison study - doi: 10.5194/hess-21-423-2017

Good luck!

I agree with Paul Muñoz

It’s depends so the all conditions