So, the model can fit both situations, although they are quite different, almost opponent each other? Something is missing from the description...

Thank you for your interest, Demetris! The model can predict the steady-state number of cells which is reached after a long growth time. For toxin concentrations which are above the critical level, the prediction will be zero or negative, meaning that there is no realistic steady state achieved because the cells will not grow. The critical toxin level can therefore be found from the model parameters (i.e. the toxin level when the steady state cell number will be zero). Please let me know whether this explanation makes sense.

The way I see it is that you are trying to fit function s = f(TC) where s is your mechanistic model's steady state (assuming that the model has only one?), and TC is the toxic concentration. The problem is that in the data, this function is nondifferentiable at TC = c, where c is the critical concentration. However, your fitting procedure will give a smooth function that is continuous and differentiable at all points, including c. That's why the mechanistic model allows negative number of cells while the data in reality doesn't. I expect that using the second kind of data only is more reasonable, as the first kind cannot be generated by the model in the first place.

 The robustness of a model is in part determined by its goodness of fit to data derived from a range of conditions. The greater the degree of model verification, more likely one should be able to validate the model against independent or partially independent data sets. As such, I would encourage you to test your model against as large a number of data sets as you can reasonably obtain.

Thank you, Hazem and Lloyd, for your interest! Indeed I wanted to use all the available data to test the model, and this was the motivation for using both data types 1 and 2. For example, it may turn out that the model has good fit quality on data type 2, and also good on type 1, but parameters may differ a lot, and a global fit to both data 1 and 2 is bad. This would suggest than a better model is needed, whereas fitting only one type of data would not. 

I totally agree.

Models that must be re-parameterized to fit different sets of data generally have limited utility in explaining the effects of biological and physical conditions on simulated performance.

The level of mechanistic detail incorporated into a model should be a function of the intended purpose for developing a model. As such, a simple model might be appropriate for a simple question, while increasingly complex mechanistic models are appropriate for increasingly complicated questions, hypotheses, and systems.

Thank you, Lloyd. So do I understand you correctly that fitting the same model to both type 1 and type 2 data is justified? If so, then a small follow-up question: suppose the model turns out to fit badly. Then new models would be produced and also fitted. Can all these models be compared by AICc, because all are fitted to the same data (types 1 and 2 combined)?

Igor: An important value of a model is its use in helping to define what a scientist knows and does not know about a particular system. If a model does not fit data well and if the researcher is confident the data accurately represent a system's behavior given a particular set of conditions, then the model requires enhancement and/or refinement. By incrementally adding new features/functions or in some cases refining existing functions one should still have a single model that is sufficiently robust to capture the effects of edaphic, climatic, biotic, and management conditions on model performance.

In contrast, developing a separate model to fit each data set would be analogous to curve fitting and would not help elucidate the underlying causes for departure between model output and specific sets of data.

Thank you again, Lloyd. I certainly agree with your explanation of the usefulness of models and about refining them. I did not mean that the "new models" would be ad hoc and/or completely different from the original model - they can be refined versions. What I was wondering about is: is it justified to fit all model versions to the type 1 + type 2 data simultaneously and do model comparison by AICc?

So dear Igor, you may probably speak about the well known sigmoid model.

If you speak about that, then, yes, it is a good and well tested model in Biology.

Dear Demetris, the motivation for my question was not the specific model, but whether or not it is justified to combine two different types of data into a single weighted least squares fit to evaluate the performance of one (or  more) model(s).

Hi everyone,

I am currently studying a methodology that might interest you. The general framework to assess a model with different outputs of different dimensions, that cannot be compared or aggregated in a single function, is the optimization of multi-objective function, or multi-objective problem.

The idea is exactly to search for the solutions that fulfill best a set of criteria. In the case of Igor, you can choose as criteria : (1) the fit performance over concentration (2) the fit concentration over the number of cells.  You can also add other criteria if you are interested in additional aspect of your model's outputs. The solutions of a multi-objective problem are called Pareto optimal solutions : they are such that if, starting from one these parametrization, you increase the performance for one criterion, then you decrease for all other criteria. The diversity of Pareto optimal solutions (there are sometimes infinity of them !) reflects the diversity of strategies that can be observed amongst living beings when they face a trade-off situation. 

Here are some references on this topic :

- for the analytical derivation of Pareto optimal solutions, see Gobbi et al., 2014

- for the heuristic methods to compute Pareto-Optimal solutions, see the book of Coello Coello et al.

-for the application of the concept of Pareto-Optimality in biology and ecology, see Kennedy 2011.

I hope you will find what you need there.

Book Evolutionary Algorithms for Solving Multi-Objective Problems...

Article Using Multicriteria Analysis of Simulation Models to Underst...

Article On the analytical derivation of the Pareto-optimal set with ...

 Sorry I have forgotten one word

they are such that if, starting from one these parametrization, you increase the performance for one criterion, then you decrease the performances for all other criteria.

 Thank you, Antonin! I will read the references you provided about these interesting methods.

I would not recommend you fit different sets of data to different models but I would recommend fitting each set of data to each model. Fitting the same data sets to different "versions" of a model provides an estimate of the degree of improvement in fit obtained by incorporating each additional feature and an estimate of the degree the model is able to capture system response under different climatic/edaphic/biotic environments.

If you fit each data set to each model, and if model 2 was developed from model 1, then this is analogous to what was done by Wu and Wilson (1998), where the authors progressively added new features to their mechanistic rice crop model to allow the model to better capture the seasonal dynamics of rice growth, development, maturation and yield. If you use this approach, then it is important to appropriately classify goodness of fit as either a verification effort (fit to data partially used to parameterize the model) or validation (fit to independent data not used in the models parameterization or verification).

Wu, G. W. and L. T. Wilson. 1998. Parameterization, verification, and validation of a physiologically complex age-structured rice simulation model. Agricultural Systems 56: 483-511.

Thank you, Lloyd. I follow your arguments about model fit quality assessment and refinement. But I am not sure I understand your advice about "fitting each set of data to each model". Does this mean that all model versions should be fitted to type 1 data only, and then separately to type 2 data only? If so, what should be done if model parameter values differ considerably when fitted to type 1 and type 2 data separately? 

Yes, fit each data set to each model. If model parameters for model A and model B differ considerably and if the parameters whose values greatly comparing the models are directly related to one or more biological attributes and  not to an edaphic, climatic, or biotic environmental state (pH, EC, nutrient composition, parasitoid/predator type or density, etc.) or a species, race or biotype difference then this would suggest one or both models are missing a key facet of the organisms intrinsic biology/ecology/physiology.

In fact, it is not only justified, it shall be a requirement. If the model can only fit one set of data but not the other, then it raises the issue if the model missed something important. Take a look at this well cited paper

Aaron Packer, Yantao Li, Tom Andersen, Qiang Hu, Yang Kuang and Milton Sommerfeld, 2011. Growth and neutral lipid synthesis in green microalgae: a mathematical model., Bioresource Technology, 102, 111-117 (pdf 336k).

Why not?.....as long it is statistically valid.

Thank you, Yang. I will read your paper with interest!

Dear Yang,

I read the paper you cited and it is indeed relevant for my problem. Just to make sure I understand: you were minimizing the sum of squares (SS) written something like this, where d is the data type, i(d) is the data point for a given data type, p(d,i(d)) is the model prediction, m(d,i(d)) is the observed mean, and v(d,i(d)) is the observed variance:

SS = SUM[SUM[ (p(d,i(d)) - m(d,i(d)))^2/v(d,i(d)), i(d)=1..K(d) ], d=1..D]

Is this correct? If so, then the data type where the variances on the data points are the smallest can dominate the SS.

I see huge benefit in using one, rather than two, models. You probably have a range of threshold values of the toxin: different species, different individuals within species respond differently to different concentrations.  With a single model, you can account for this variability, but with a threshold-based model, you can't.  I think if you use log(C/(1-C)) where C is the concentration, for the independent variable, you may even be able to get away with a linear regression

Igor,

The data type where the variance is the smallest should not dominate the errors. Unless there is something wrong with the model. 

Given you have a vector valued model, f(p,x) = (f1(p,x),f2(p,x)), which is a function of a parameter vector p, and independent variables x. Suppose that you had a data set with N measurements of dependent variables y, and independent variables x. Then, for the ith measurement, we can express the measured y values in terms of the model predictions plus an error term: yi = f(p,xi) + ei = (f1(p,xi)+e1,i,f2(p,xi)+e2,i). If the model is correct, and the measurement errors are independent, identically distributed random normal variables, then the model errors are i.i.d. random normal variables, i.e. e1,i ~ N(m1,s1) and e2,i ~ N(m2,s2). Then the sums SUMi[(e1,i/s1)2] and SUMi[(e2,i/s2)2] (where SUMi indicates the sum over the index i) both follow the chi-squared distribution with N degrees of freedom. And, in this case, minimizing SUMi[(e1,i/s1)2] + SUMi[(e2,i/s2)2] provides the maximum likelihood estimate for the parameter vector p.

But, if any of the assumptions are violated, i.e. if the model is wrong, errors in the dependent variable y are not normal, or the errors in the dependent variables x are not negligible, then the reliability of the analysis is questionable. 

Thank you, Shane, for an instructive explanation!