Been my experience, that the best way to compare stream flow values is with a graph. If you are interested in determining the applicability of two different modeling methods. Try a timeline plot of of the differences between each method and the observed data. Simple is sometimes best.

I used both statistical and graphical (hydrograph) methods to show my results. Sometimes I found problems that graphs looks nice, the statistical results are weird and vice verse. Finally, I decided to go with simple graphs (like you said) and interpret the results using the statistics which matches the graphical results . Thanks Perry. I appreciate your comments.

Dear Chalise,

A number of indicators are used to assess the agreement of results with observations.

Generally, the 1 value indicates the high-performance of the model in the simulation process. For example, the Nash-Sutcliffe index will fit into this category.

But some of indicators will be used to comparing the results between two or more models. The RMSE, RE%, Bias etc. performance indicators can be used for such cases.

Although the visual comparison can give you an overall judgment, but I think that you pay attention more to the values of the statistical indicators.

I recommend reading the attached article.

Dear Dol,

you can see how we work with these goodness of fit tests (GOFs) in this paper:

http://abouthydrology.blogspot.it/2011/11/jgrass-newage-system-first-publication.html

What we usually do is applying these tests as a basis, and subsequently further arguing with qualitative arguments. Plotting and analysing the structure of the residuals, can give some insight on the type of error the models do.

An R package for GOFs analysis is HydroGOF, and you can find it referenced here:

http://abouthydrology.blogspot.it/2012/08/r-resources-for-hydrologists.html

Hope this helps,

riccardo

Since you are going to compare the results of two or more models, and certainly to show that one model outweighs the others, you should conduct the T-Test, a simple, fast and conclusive test, at the same time worldwide acceptable and convenient.

A paired-samples T-Test (two tails) is conducted using SPSS Statistics in order to show that the correlation between the observation data set and the modeled data set is significant so that P-value>>0.05 shows the significance test is passed (for engineering problems P>0.5 is acceptable). On the other hand, a one-sample T-Test or the same previous test with opposite null hypothesis is used to show the significance of difference between two sets of data. In other words, when you want to show the predominance of one model over another, you should use this kind of test so that P

Yes, you can do it. Please refer to the following papers.

1. Halwatura D, Najim MMM 2013. Application of the HEC-HMS model for runoff simulation in a tropical catchment, Environmental Modelling and Software (impact factor: 3.11). 01/2013; 46:155–162. DOI:http://dx.doi.org/10.1016/j.envsoft.2013.03.006

2. Babel, M.S., Najim, M.M.M., Loof, R., 2004. Assessment of agricultural nonpoint

source model for a watershed in tropical environment. Journal of Environment

Engineering 130 (9), 1032-1041.

3. Bennett, N.D., Croke, B.F.W., Guariso, G., Guillaume, J.H.A., Hamilton, S.H.,

Jakeman, A.J., Marsili-Libelli, S., Newham, L.T.H., Norton, J.P., Perrin, C.,

Pierce, S.A., Robson, B., Seppelt, R., Voinov, A.A., Fathi, B.D., Andreassian, V.,

2013. Characterising performance of environmental models. Environmental

Modelling & Software 40, 1-20.

4. Najim, M.M.M., Babel, M.S., Loof, R., 2006. AGNPS model assessment for a mixed forested watershed in Thailand. Science Asia 32, 53-61.

The reference 3 gives many ways of doing the comparison. I have applied few of such methods in the other papers.

Please refer my PhD thesis uploaded in RG which may give you some idea

This 2013 paper, just published by Keith Beven and Peter Young, will almost certainly be of interest to all hydrological modellers, and to those with interests in this discussion:

Keith Beven and Peter Young (2013)

'A guide to good practice in modeling semantics for authors and referees'

In: Water Resources Research

(Early View: Online Version of Record published before inclusion in an issue)

DOI: 10.1002/wrcr.20393

Article first published online: 26 AUG 2013

Abstract

This opinion piece makes some suggestions about guidelines for modeling semantics that can be referred to by authors and referees. We discuss descriptions of model structures, different forms of simulation and prediction, descriptions of different sources of uncertainty in modeling practice, the language of model validation, and concepts of predictability and fitness-for-purpose. While not expecting universal agreement on these suggestions, given the loose usage of words in the literature, we hope that the discussion of the issues involved will at least give pause for thought and encourage good practice in model development and applications.

hi,

hope so, you found solution for the question that has been posted.

I agree with the researchers recommendation on statistical comparison of model results and so on. But i too faced the same problem while explaining with the performance indicators. From many personnel view, i felt scatter plot between actual and simulated flows explain many of the hydrological characteristics. In the developed scatter plot compare the perfect model (45 degree) fit with the trend line of simulated values that can explain more detailed information about your developed model. If you are more curious then even plot the 95% confidence prediction band, from that you can visualize the outliers in your prediction. From the pattern of outliers you can judge whether your model is working for the considered database.

hope so, my idea might furnish you some idea while discussing the model results.

When you seek to model to have ALL involved processes correct you should use Maximum Likelihood Estimates (MLE). You choose a prior distribution function for the residues (differences between model and measurements) that has to be verified posteriori. In most cases this boils down to a normal distribution implying RMSE has to be used as norm for judging the correctness of your model. (There is a truckload of literature on this subject.)

Plot the two flow duration curves against each other. Hopefully you have some means of testing the models against observed data? The Nash-Sutcliffe coefficient and other statistics are useful for putting numbers to the performance of each simulation, but flow duration curves allow you to compare all of the data across the range of outputs and allow you to make a visual assessment of performance. Then you can use that part of the curve that is of interest to you to influence your modelling approach.

Sorry, I will add another paper!

It is synthetic, shows the formulas, explains advantages and disadvantages of the several criteria and finally shows some examples.

Moriasi, D.N. et al., 2007. Model evaluation guidelines for systematic quantification of accuracy in watershed simulations. Transactions of the ASABE, 50: 15.

available at

http://swat.tamu.edu/media/1312/moriasimodeleval.pdf

Finally, I used weight of evidence approach to compare the accuracy of model. Both graphical and statistical methods were used to evaluate the model performance for different time steps. For statistics, I used this reference: Moriasi, D. N., Arnold, J. G., Van Liew, M.W., Bingner, R. L., Harmel, R. D., and Veith.,T. L. (2007). Model Evaluation Guidelines for Systematic Quantification of Accuracy in Watershed Simulations. ASABE, 50(3), 885−900. Thanks everyone for helping me. I hope your contribution will help others too.

Hi,

have a look at the paper of Daniel Moriasi that Alain and Dol already mentioned. Have also a look at Tony Jakemans valuable paper on:

Jakeman, A, Letcher, R & Norton, J 2006, 'Ten iterative steps in development and evaluation of environmental models', Environmental Modelling and Software, vol. 21, no. 5, pp. 602-614.

Martin

I suggest you to take a look at an Aggregated Measure introduced and used in the following papers:

1: WetSpa model application in the Distributed Model Intercomparison Project (DMIP2) ; Journal of Hydrology (2012)

https://www.researchgate.net/publication/229115485_WetSpa_model_application_in_the_Distributed_Model_Intercomparison_Project_(DMIP2)

2: Improving WetSpa model to predict streamflows for gaged and ungaged catchments; Journal of Hydroinformatics (2013)

https://www.researchgate.net/publication/260369060_Improving_WetSpa_model_to_predict_streamflows_for_gaged_and_ungaged_catchments

Article WetSpa Model Application in the Distributed Model Intercompa...

Article Improving WetSpa model to predict streamflows for gaged and ...

I think if you calculate two or three index of evaluations as you already indicated is enough to compare results of your modeling. then use your feelings to decide the model you consider gave 'best' results as compared to observed values.

so, believe yourself of your judgement

This is an everlasting discussion that will not be easily settled. I think it is important to mention here that it is important to mention for what aim you are evaluating your model. Many of the measures mentioned above are for example not suited to validate models for low-flow. Many good papers have already been mentioned above, but I like to draw attention to the work of Hoshin Gupta (University of Arizona) here. I recently read his paper listed below, and I found it very inspirational.

Hoshin V. Gupta, Thorsten Wagener and Yuqiong Liu. 2008. Reconciling theory with observations: elements of a diagnostic approach to model evaluation. Hydrological Processes, 22(18), 3802–3813.

A number of mostly used visual comparison methods are presented in the attached Table.

Source: Vai Vaze, Phillip Jordan, Richard Beecham, Andrew Frost, Gregory Summerell. 2011. Guidelines for Rainfall-Runoff Modelling: Towards Best Practice Model Application

It is very common that hydrologic researchers put extraordinary emphasis on the similarity of measured and computed hydrologic varaiables, mostly runoff. However, this is of secondary interest. It is much more important in practice, what we do with such models. Very often we use them to extrapolate into probability regions not covered by measurements, e.g. extreme flows. If you are not aware of this fact you take the risk of using models as dangerous toys instead of helpful tools. You can find my ideas on the matter here

https://www.researchgate.net/publication/243457593_LINEARITY_OF_HYDROLOGICAL_MODELS_AND_RELATED_UNCERTAINTY

Regards Manfred

Conference Paper LINEARITY OF HYDROLOGICAL MODELS AND RELATED UNCERTAINTY

Dear Charlise,

I completely agree with Raoof Mostafazadeh. In addition, I'd like to stress the importance of the dimension analysis of the model parameters after calibration; using a singular value decomposition technique of the Jacobean (of the optimal parameter set) it can be decided whether or not the parameters are mutually independent; if not you either lack information (observations) or the model applied misses some important process which may impose a riks when using the model with other loads than the ones on which it was calibrated. In that sense it is wise to use a subset of your observations for calibration and another subset for validation. 

best regards

Francois Clemens

My following discussions are about performance evaluation for realtime flood forecasting models, although it may apply to similar applications.

Due to the random nature of flood events and uncertainties in model building (model uncertainty and parameter uncertainty), a flood forecasting model can perform quite differently with respect to different flood events. Therefore, it is also important to address the uncertainty of model performance in model performance evaluation. In the literature, a few research works had evaluated the model performance by using the Nash Sutcliffe model efficiency derived from a multi-event artifactual series. Such practice by no means provides a multi-event overall evaluation and may actually disguise the real capability of the adopted forecasting model.

Model performance evaluation should also be linked to model performance comparison. For example, a very complicated model may be able to achieve good performance (based on the chosen evaluation criteria). However, if the persistent (or the naïve) forecasting model can outperform the complicated model using the same criteria, then the value of the latter is questionable.

For hydrological forecasting (such as realtime flood forecasting) purpose, criteria like the Nash Sutcliffe model efficiency (NSE), RMSE, and mean absolute error are widely used, with NSE being most popular. Although Moriasi et al. (Model Evaluation Guidelines for Systematic Quantification of Accuracy in Watershed Simulations. ASABE, 2007:50(3), 885−900.) recommended using “ Good, if 0.65

Selection of model performance rating criteria. Fore example Nash Efficiency criteria is more responsive for extreme river flows.

These criteria are either distance and weak form-based objective functions. More detail can be find in Table 2 of the attached research article.

Article Streamflow and Sediment Yield Prediction for Watershed Prior...

Regards,

Gebiaw

you can see this article

https://www.mdpi.com/2073-4441/10/7/876?type=check_update&version=1

the authors compare a Hydrologic Engineering Centre-Hydrologic Modeling System (HEC-HMS) conceptual model and an ANN model based on the conjugate gradient method for streamflow prediction

Article How the performance of hydrological models relates to credib...

I recommend you to use the statistical parameters to quantify the matching between measurements and estimated data by model, there are several parameters for this purpose such as root mean square error, the coefficient of efficiency, mean square error,............etc

I have just added the following answer to a similar question at :

https://www.researchgate.net/post/How_do_we_verify_that_a_hydrological_model_is_a_good_model2

Once you have decided on a suitable measure of how good a model is for a given purpose, you would then be interested in comparing those measures for two different models and trying to see whether one model is better than another, potentially taking account of the different numbers of parameters used for model-fitting. A paper by myself provides a start on doing this in a "formal" way: see

Article Statistical Analysis of Empirical Models Fitted by Optimization

Some additional resources on this topic:

Chapter Forecast Evaluation Techniques for I4.0 Systems

Chapter Forecast Error Measures: Critical Review and Practical Recommendations