If I understand correctly you have observed X_ijk (mode k on the i to j OD pair) and SOME model prediction of the same quantity (let's call it Y_ijk). Let's assume you estimated the parameters for your model from the observations (maybe using a constrained entropy max model, or a log linear model...) and suppose you have TWO contending model outputs (and you want to gauge which one is better). Right? OK, this is a classic problem and the goodness of fit can come from the model framework (log-likelihood as a measure of the overall quality) or maybe some "root mean squared error" (RMSE). What I would avoid would be some kind of r-squared type measure which would not take into account the constrained model structure. The art to this of course is how much of the observed data are you prepared to use to "tie down" or constrain your model output. If you intend to use the model in some forecasting framework, you should only use exogenous (constrained values) that are themselves easy to forecast.

If the data is not huge, and not proprietary, you could even run a small contest revealing only the parts of the data that are exogenous, to see how well each model uncovers the true underlying modal split.

There is a possibility that you have a bigger problem... you are told that X_ij is the blend of three underlying components (X_ij1, X_ij2, X_ij3) but do not have real data for the modal split! The tactic then is a little bit like what we have done to split the OD pairs for commuters by occupation. (See O'Kelly and Lee in Environment and Planning; on RG).

All best and thanks for the nice question!

Thank you for your answer Morton.

You are indeed right with your basic assumptions. Maybe I can give you some more information.

The data set is not too big : for each mode, I have an origin-destination matrix from European NUTS2 region to another European NUTS2 region. Altogether, taking into account that not all modes can be used from one region to another (it is not possible to use inland waterway transport, because the network for this mode doesn’t cover the whole European continent for instance), I have about 100.000 « observations ».

I use a network model to compute a set of cheapest paths between each OD. One (or more, but I keep it simple for now) for each mode. The « cheapest » path are computed using some kind of generalised cost (« real » cost, value of time, …). The model is calibrated (the generalised cost of 2 of the three modes is multiplied by a factor F1 and F2, so that, for the modal split model a test, the modal split I obtain is equivalent to the observed one.

The model is calibrated on a sample corresponding to 5% of the original matrixes. The input of the modal split model is only the (calibrated) generalised cost of the cheapest route for each mode. The calibrated generalised costs are the applied to the total matrix. The same exercice is done for each modal-choice model I test.

At the end, I can compare, for each OD pair and for each mode, the « observed » demand and the one that I obtain with my different models.

Indeed, the « quality » of the different models must be assessed by a « goodness to fit » method. RMSE is one of these. I came also over variants such as MAPE and others.

For the time being, I did not find any examples or experiences with these methods with comparable data (as I don’t use time series). My question is thus « which is the best estimator to use in my case ? »

Than you again for your answer. I’ll have a look to the reference you give.

Perhaps you might want to consider assessing the market choices for the O-D pairs through getting real customers on the route to provide insights. The methodology is well explained for three O-D pairs, two directions each, for different types of customers in 

Article Understanding Mode Choice Decisions: A Study of Australian F...

Thank your for your answer Mary.

Indeed, several modal-choice models exist, including the very interesting approach you developed at the IAME 2011 conference. The case you develop is based on econometric models, to which classical measures (such as Chi square or the Akaike Information Criterion can be applied.

My problem is somewhat different in the sense I try to compare the performance of models that are not directly comparable (not all are based on econometrics for instance ). So, I have to limit myself to the comparison of observed data and estimated results, whatever is used to estimate.

I have an origin-destination matrix to which I apply several modal-choice models (the modal choice model specifications have no importance here). A the end, for each origin-destination pair, I obtain a market share for each available transportation mode. I have to compare these with the "observed" ones. I have the latest because the origin-destination matrix to which the modal-choice models are applied is simply the merge of the OD matrixes I have for each mode.

The simplest way the measure the performance is to compute the R2 or another correlation factor between my results and the observation. But it is well known that R2 is highly influenced by the out layers (a few very bad predictions has an important impact on the R2). Therefore, other goodness to fit estimators exists, such as MAPE and others. I have tried to apply them, but the results don't converge (the same modal-split method doesn't always come out as the "best" one). 

Thus, which estimator should I use ? May be other estimators exists, that better correspond to my particular needs ?

Thanks again for your answer.

Obviously, as a market researcher, I always think that the best way to understand choice decisions is to ask the chooser, the one who makes the choice and pays for it. For consumer research, conjoint analysis is very useful but there are not enough decision-makers to use this approach in freight where mode choice decisions involving hundreds of millions of dollars worth of freight payments may reside with one individual working for Target. The quality of results from primary research with users is far superior to any method relying on secondary data. The key is always identifying the correct variables for consideration. My suggestions for continued reading always include Thaler and Sunstein's Nudge.

there are well established literature and methods  but best review the real world data and validate.

Thank you all for for constructive answers.

I agree with you that real world data and interviews are needed to construct a solid modal-choice model. However, this is not the point I try to solve. My question was most probably poorly formulated. Therefore,I'll start another question, hoping to better specify the problem.

Thanks again

You might use Performance Measure, like F-measure or G-mean