There are a number of hydrological models to predict the seasonal fluxes of fresh water, sediment and nutrients. But there is no clear guidelines to verify which model is more relevant for a particular purpose.
I can answer to this in my view. Hydrological model simulate various process like infiltration, percolation and flow routing etc., that is driven by different parameter. To verify a good model for particular purposes we should know which parameter is more significantly simulated by the model. Sensitivity analysis of parameter can give idea about important parameters. So we can choose the model based on our purpose.
Capturing the processes operating in the system is not sufficient to ensure that you have a good model as the functional form for the processes (and how these link together to form a catchment) cannot be derived from first principles. The result is that the same process can be represented in a number of ways, and there is no way of determining which is the correct functional form. To verify which model is more relevant for a particular purpose, you need to compare the model output with observations.
Generally, hydrologist rely heavily on the Nash Sutcliffe model efficiency, other wise known as R^2 or the coefficient of determination (or at least one of them). This gives an aggregated measure of how well a model output reproduces the observed time series - usually applied to the discharge from the catchment. However, R^2 is not a good indicator as anything using the sum of squared residuals assumes independent and identically distributed uncertainties (not residuals!), which we know is not the case. See
https://www.researchgate.net/publication/228371805_Representing_uncertainty_in_objective_functions_extension_to_include_the_influence_of_serial_correlation?ev=prf_pub
There are a lot of other performance measures available - for a review, see
https://www.researchgate.net/publication/257550125_Characterising_performance_of_environmental_models?ev=prf_pub
The first task is to determine what observations can be used to evaluate the performance of each model - taking into consideration the purpose for wwhich you want to use the model. You then need to consider how you are going to measure that performance. An objective function llike R^2 will generally give a better value when using a complex model. This is because such objective functions do not take the uncertainty in the model output into account. If this is done (e.g. Peter Young's YIC), then you will get a better indication of the appropriate model. You should also do a sensitivity analysis to understand which parameters are important, and which are not contributing to the model performance (as mentioned by Prachi). This should be extended to consider uncertainty analysis, where the impact of uncertainty in the model inputs (which you will need to estimate) on the model outputs is explored.
Ultimately, you want to chose the model that makes the best prediction - i.e. the one that has the best trade off between accuracy (closest value to the true value) and precision (least uncertainty in the predictions). The uncertainty in the predictions cannot be determined by looking at the model residuals. A more heavily parameterised model will generally give smaller residuals than a model that has fewer parameters, even if the simpler model is a good one. However, the uncertainty in the predictions will almost certainly be higher due to the increased uncertainty in the parameter values (information in the data is being distributed over more parameters, so less information available to determine the value of each parameter). Often an ensemble approach will be beneficial if the models are sufficiently different. The idea here is that the uncertainties in the model outputs are not perfectly correlated (due to the differences in the models), and as a result, the uncertainty in the ensemble prediction is smaller than the uncertainty from any of the individual models used.
Conference Paper Representing uncertainty in objective functions: extension t...
Article Characterising Performance of Environmental Models
generally, the model should be fit for the purpose (the flux u want to model) and should capture the main physical processes at play
However, generally performance indicators are used to check the how good the model is working. Generally the Nash-Sutcliffe Efficiency (NSE) is used by most modellers as an overall performance measure. However, there are other indicators that will allow for a more holistic assessment as follows (with a bias towards streamflow simulations);
Overall performance: NSE - see Nash and Sutcliffe, 1970, the recent Kling-Gupta Efficiency (KGE) - see Gupta et al.(2009) AND the Index of Agreement (IoA) - see Wilmott, 1981
Low flows: log-transformed NSE, the log-transformed Mean Squared Logarithmic Error (MSLE) - see de Vos and Rientjes, 2007
Volume Errors: Relative Volume Errors (RVE) - see Janssen and Heuberger, 1995, Deviation Runoff Volumes (DV) - see Martinec et al, 1989
Volume Timing Errors: volumetric efficiency (VE) - see Criss and Winston (2008)
Shape of Hydrographs: mean squared derivative error (MSDE) - see de Vos and Rientjes (2007)
Peaks - Percent Error in Peaks (PEP)
others are e.g.: RMSE, ME, Coefficient of Persistance, MAE, Relative Mean Absolute Error, PBIAS: Percent Bias etc....
Get the right balance so as to assess different aspects of the simulation - a comprehensive performance assessment!!
I want to add one more thing to it .....
Along with the statistical indicators which have been discussed/told earlier by the peers, graphical indicators like scatter plot along with 1:1 line and normal time-series graphs should also be used for checking the performance of a model. By considering the results of both we can have a complete idea about the performance of the model. (As in case of sediment or nutrient analysis, many cases statistical indicators gives pretty good results but graphical indicators tell the exact picture of over and under estimation).
My current research is concerned with this issue (amongst other things). I am using the Hydrus 1D model to simulate the vertical movement of water and solutes through various soil profiles. Obviously as the model is restricted to the vertical dimension it cannot indicate lateral flow. As such, you can only evaluate the performance of a model within the context of the processes it attempts to simulate.
Furthermore, the performance of the model will rely not only on the numerical equations incorporated in it, but also on the validity and accuracy of the input parameters. I have a paper currently under review which evaluates the effects of pedotransfer functions vs measured hydraulic properties on model outputs.
I am runnning bromide tracer tests in two vulnerable agricultural catchments (grassland and arable) and it is my intention to compare the results from these tests to simulations made using Hydrus 1D. This should enable me to evaluate the performance of this model within the given context.
I agree that a hydrologic model (any model) should be evaluated within the context of the research question or decision making problem. System characterization, data quality and availability and model performance criteria should be established prior to model selection. Performance evaluation is more straightforward and less arbitrary when model assumptions, data limitations, and acceptability criteria are identified. These need to be addressed when publishing the work and serve as the basis for peer review.
Hello! Another consideration is if you want a distributed model. I have provided a GIS based model that is validated for conditions in Sweden but used in many other environments, for example China.
See https://www.researchgate.net/publication/222296604_Non-point_source_critical_area_analysis_in_the_Gissel_watershed_using_GIS
Article Non-point source critical area analysis in the Gisselö water...
That depends on your application and time and efforts you want to put
Good papers related to hydrological modelling efficiency assesment:
https://dspace.lboro.ac.uk/dspace-jspui/bitstream/2134/2733/1/HydroTestFinal.pdf
http://www.adv-geosci.net/5/89/2005/adgeo-5-89-2005.pdf
First link provides also a HydroTest description. It is a free online toolbox, that can calculate many metrics for your data. I'm using it to evaluate my models and I find it really well.
The answers above all relate to the limitations of modelling. Levins famously argued that it is not possible to maximise reality, precision and generality at the same time. the hydrologist Beven identifies the problem of equifinality where the same performance can be attained by varying both model structure and parameter values.
Because of this uncertainty Levins suggests that a robust result can only be attained by the approximate convergence of independent models or lines of evidence. Independence being critical
Therefore my approach to any problem in ecology or involving complex systems is to apply the above principle for a robust result, for example if your confident about the model structure of your system then use a Statistical Baysian model with no forced parameterisation with the best systems model performance until there is convergence. Although, 3 independent model approaches would be better. Then evaluate the paired convergence in another catchment or at another time in the same catchment outside the calibrated data range.
Its all very difficult especially future prediction in a world changing with climate. In that regard its always important to present or frame your predictions within the parameters of a general and less precise spatial empirical model like Vollenvieders eutrophication of lakes across the world catchments.
There are several performance criteria to check the model performance, nevertheless I suggest to take in account orthogonality of selected performance measures!!! Please check this link:
https://www.researchgate.net/publication/230846978_The_search_for_orthogonal_hydrological_modelling_metrics_a_case_study_of_20_monitoring_stations_in_Colombia?ev=prf_pub
Article The search for orthogonal hydrological modelling metrics: A ...
There is an amazing amount of information in this paper. The authors apply 22 statistical metrics to model output for 20 hydrological stations, obtain 5 independent metrics by using principal component analysis, then reduce those to obtain a single measure.
I do not like to use a single index to judge how well a hydrological model performed. WARMF is a GIS based distributed dynamic watershed model. It provided the graphical comparisons of the simulated and observed cumulative hydrographs, frequency distribution curves, and instantaneous hydrographs. It also provided the correlation coefficient for available pairs of simulated and observed flows. Through multiples parameters, the users can judge for themselves how well the model performed.
For those who may not know him, I would like to mention that Carl is one of the pioneer modelers.
From what I know, there's no hydrological model which can be applied universally due different way each natural system operates. Hence, to decide which model to use, model outputs have to be compared with field data and , the one that gives best estimate is apparently the best one to use for that particular area.
Hydrology is only the subject discipline: the model needed depends on the object. If the object is an engineering catchment, the appropriate model will reflect closed system sciences of physical thermodynamics and infrastructure economics; providing alpha-numeric solutions. These models are theoretical, closed system models applying only to physical objects.
If the object is a living watershed (encompassing habitats and ecosystems) the appropriate model will reflect complex open system sciences (ecography/ecology/geography/hydrography~ the geospatial sciences). All stream and river basin watersheds are open living ecosystems, so strictly speakingy, only complex open systems models of rivers and floodplains are scientifically valid. These models are empirical spatial open system models.
The field of watershed ecology engages in real time monitoring of water resources displayed in a Watershed iGiS (image based Geospatial intelligence System). The outputs and solutions are contained in 3D/4D spatial imagery displaying aquifer/surface water distributions and flows for a given point in time using field data from extensive suites of peizometers and meteorological stations.
Spatial data provides ecological indicators of watershed ecosystem performance. Numerical datasets are aggregated historically to reveal seasonal, periodic and episodic patterns of water assignment in the watershed's atmosphere, habitats and regoliths.
Living watersheds are reduced to dying catchments by applying closed system theories (and thinking) to our streams and rivers. Please check your model to see you are not using a closed system model for spatial objects containing habitats and ecosystems.
Very interesting exchange indeed. What is a good model?
First we have to keep in mind G.E.P. Box’s statement "essentially, all models are wrong, but some are useful", and K. Popper’s theories about induction and corroboration. It follows that the term “model validation” should be avoided and that the metrics of model performance are tools to help defining the domain of applicability of a given model and to assess the uncertainty of the results within this domain.
However, this is generally not sufficient to check that the model is able to simulate the observed behaviour of the system. You must also assess to what degree a number of characteristics of the model are compatible with the objective and context of the modelling exercise: principles, structure, data requirements, computing efficiency, ergonomy, etc.
The initial question was about “clear guidelines to verify which model is more relevant for a particular purpose”. Although I think that it is very difficult to define a standard procedure, an interesting attempt have been made within the Dutch GMP project under the form of a "good modelling practice handbook”:
http://harmoniqua.wur.nl/public/Reports/Existing%20Guidelines/GMP111.pdf
Calibration and validation will tell you the suitability of a hydrological model. How well a particular model simulates rainfall-runoff response? This can quantify using some indices.
This is a usual procedure when working with hydrologists in forested areas. They are modeling their watershed, e.g. 100 000 ha area, selecting a series of pixels (say 4 ha) with well defined parameters, where all their results are presented. Than we go there with sap flow system, installed on series of trees well rep[resenting the entire stand at the pixels and are comparing bothsets of data. Since we always also measure crowns and root systems of trees (LAI, sunlitLAI, vertical and radial distribution of LAI, effective crown size, shape and area, absorptive root area, RAIabsorp, approximate active root 3D distribution, the hydrologists can meke their models even better.
In order to say the hydrological model is a good model, you have to first understand the model concept that must fit With Your objective. Then you have to validate Your hydrological model using different validation techniques.
I can recommend the attached publication.
It discusses not only the metrics but also the orthogonality of a group of performance metrics...
Article The search for orthogonal hydrological modelling metrics: A ...
It is million dollars question. It depends on many things, e.g., type and nature and quality of data, human resources, computer power, skill, funds etc etc. For example if you have DEM with fine resolution, good soil classification, crop data, rainfall, surface water, riverflow, groundwater data sets. Then MIKE SHE hydrological mode which is fully integrated and spatially distributed is the best one. It has fully distributed or lumped module for each hydrological process and can be used according to the fund, time and data type available.
Now as this model has produced more scientific outputs, therefore its R^2 and other statistical analysis will be much better as compare to other models.
Hydrological models are very subjective to the field conditions. Even we performed great with fitting the data, we may lose the vision of answering a perticular research question, such as a prediction in a preset scenario.
There are a wide range of evaluators can be used to provide cross comparison among a series of models serving the same purposes. i.e. rating curve of several gauges at a catchment or the accuracy of modules accross disciplines. Nash-sutcliffe efficiency is the most common and widely accepted based on Nash&Sutcliffe(1970). The equation represents the ratio of the square of difference between the observation and the model to the square of the difference between the observation and the observation mean. A NSE below zero tells the model is not even better than just using the mean observation value to be the model.
I totally understand your doubts on the goodness of a hydrological model. Usually it requires you to argue your method to best match your purpose in your thesis or paper. This question can be extended to a philosophic question:"what is good?" A thousand people have a thousand answers to it. Therefore, develop your own criterion based on the methods stated in others papar would be a good start.
River modeling is pyre numerical process so the good model is the model which gives good statistical results, especially in validation.
So you must try many different calibration periods.
Article Simulation of Runoff in West Africa: Is There a Single Data-...
Article The influence of distributed input data on the hydrological ...
Article Taking into account the spatial and temporal variability of ...
Dear Sir
Nash-Sutcliffe coefficient measures the efficiency of the model by relating the goodness-of-fit of the model to the variance of the measured data, Nash-Sutcliffe efficiencies can range from −∞ to 1. An efficiency of 1 corresponds to a perfect match of modelled discharge to the observed data. An efficiency of 0 indicates that the model predictions are as accurate as the mean of the observed data, whereas an efficiency less than zero (−∞ < NS < 0) occurs when the observed mean is a better predictor than the model.
The coefficient of determination denoted r2 , it provides a measure of how well-observed outcomes are replicated by the model. The range of r2lies between 0 and 1 which described how much of the observed desperation is explained by the prediction. A value of zero means no correlation at all; whereas one means that the desperation of the prediction is equal to that of the observation.
literature and the recommendations the model simulation can be judged as “satisfactory” if NES > 0.50 and R2 >0.50, and MAE
My following discussions are about performance evaluation for realtime flood forecasting models, although it may apply to similar applications.
Due to the random nature of flood events and uncertainties in model building (model uncertainty and parameter uncertainty), a flood forecasting model can perform quite differently with respect to different flood events. Therefore, it is also important to address the uncertainty of model performance in model performance evaluation. In the literature, a few research works had evaluated the model performance by using the Nash Sutcliffe model efficiency derived from a multi-event artifactual series. Such practice by no means provides a multi-event overall evaluation and may actually disguise the real capability of the adopted forecasting model.
Model performance evaluation should also be linked to model performance comparison. For example, a very complicated model may be able to achieve good performance (based on the chosen evaluation criteria). However, if the persistent (or the naïve) forecasting model can outperform the complicated model using the same criteria, then the value of the latter is questionable.
For hydrological forecasting (such as realtime flood forecasting) purpose, criteria like the Nash Sutcliffe model efficiency (NSE), RMSE, and mean absolute error are widely used, with NSE being most popular. Although Moriasi et al. (Model Evaluation Guidelines for Systematic Quantification of Accuracy in Watershed Simulations. ASABE, 2007:50(3), 885−900.) recommended using “ Good, if 0.65
https://www.researchgate.net/post/In_modeling_applications_is_only_calibration_of_the_model_sufficient_In_what_instances_is_only_calibration_of_a_model_applicable
I would give a simple solution. Apply the model on historical data and check the results with actuals to know the accuracy & predictability of the model.
I have 30 years of daily flow data. I tried to verify the model(s) with the Nash-Sutcliffe Efficiency. However, the NSE resulted in a negative value when using the entire 30 daily years of record, however, I do get a good fit with using every year data instead of daily data. Furthermore, I noticed that the coefficient of determination works wells for the larger data set (daily) as well as per annum data-sets.
Verification, in the context of mathematical computer-based hydrologic models, stands for the stage in the model development process whereby the developer/tester ensures that the model code properly reflects/programs the governing equations and the subsequent solution techniques. This may be done using an already verified model with similar capabilities/processes in simulation of hypothetical cases. Or, alternatively, using measurements of state or output variables in controlled lab experiments. However, if you mean how to validate a model, then this concept applies to a specific watershed and it is done via calibration and validation stages using observed state (rarely available) or output variables (mostly outlet discharge). Nash- (i.e. CE) is one the best goodness-of-fit criteria while R2 is not.
In my opinion, Check the predicted values with the observed ones to check the applicability of the model using various performance indicators available in the literature.
Dear Shimelis
it depend on your purpose for choosing parameter and then finding the best software for it.
but i think you can find your answer in its Manuel.
-BEST
Just an add up, NSE focuses mainly on extreme values. So a good strategy would be to use it along with other statistical tools , R^2, volume error, Percent bias, and visual inspection of resultant hydrographs (observed and simulated)
a model is merely a tool, and a good tool is a one that help you doing the work. So, the quality of a model is completely dependent on the purpose of the modelling. If you want to reproduce with accuracy and precision observed data, e.g. to fill gaps or make short term prediction, the main criteria will be the goodness of fit (NSE, RMSE, etc.). If you want to identify major factors or processes, test hypotheses or test complex scenarios, you may prefer a physically-based model that will not be so performing on reproducing observations, but gives a realistic view of the system functionning. If you have to simulate many watersheds in a short time just to check simple assumptions, you may go for a simple conceptual model, etc. Just remember : "all models are wrong, but some are useful".
Apply the model on historical data and check the results with actuals to know the accuracy and predictability of the model.
As far as I'm concerned, it depends on you research objective. For example you can chose hydrodynamic model when you focus on flooding, sedimentation, etc. For sure, you can not do your research using a water quantity model. The objective function (NSE, NSElog. etc.) only represent the performance of simulation. There are so many models nowadays. We can not say which model is the best. However, we can find out the optimal model for our research.
The choice of the model also depends on the availibility of data.
Many models request so many data to run, that are difficult to find for our research sites, that too often many parameters are filled in with values coming from general tables, not related to our study site. This is a main drawback of such use of over-parameterized models.
Dear Hydro geologists.
Prof Shemelis Setegn
We have noted good work and discussion on varied topics on hydro geology.
We have been discussing a connection on a study of hydro-geology wth seismology.
When you have aquifers becoming drier day by day, with use of ground wtr, the sponge effect that is offered by the GW Regime is lost or reduced.
This phenomenon when viewed on e quakes and trmor effects, leads us to opine that more severe impacts will be felt on urban high rise buildings in the region.
Has anyone studied this as simulated on a model or on research probing Etc
If so, can share the details.
If not, can a lab based demo-test be simulated to study the impacts and effects.
Wish our RG Members to respond to the comment and offer inference Etc.
for the good of communities.
Well wishes.
Such an interesting topic with lots of useful insights.
I would like to make a minor contribution to the discussion. As FutureWater states (https://www.futurewater.eu/methods/modeling/), Hydrological models, are important because challenges regarding the number and diversity of water-related issue are countless, and are expected to increase in the future. As such, hydrological models help us analyze, understand and explore solutions for sustainable water management, in order to support decision makers and operational water managers. Mostly, hydrological models are involved in two main situations: process understanding and scenario analysis.
In this view, propositions can be made as a checklist to evaluate the goodness of model, provided that no model is perfect, but rather "better at describing reality (observations) and helping us gaining more understanding." I would like to quote here 7 propositions originally stated by Prof. Andreassian Vazken (find the original document here (direct pdf link: https://www.dfh-ufa.org/fileadmin/Dateien/Forschung/veranstaltung_water_climate/Andreassian_Vazken.pdf for reference).
Proposition 1: A ‘good’ model is one that gets a good (numerical) grade (i.e KGE, NSE, etc...). Since a grade brings a relative information (as being an element of comparison), it might be critical to combine different grades which do no converge depending on the objective of the model.
Proposition 2: A ‘good’ model is one that is deemed good by experts.
Proposition 3: A ‘good’ model is one that can be used in extrapolation (issue of robustness). Such issue might be critical for changing catchments, where differential split-sampling test for instance might be used to assess the ability of the model to extrapolate.
Proposition 4: A ‘good’ model is one that can work in degraded mode (issue of robustness again). Performance of the model should be quite good with a relatively bad quality of the inputs, and performance is expected to increase with the quality of the inputs.
Proposition 5: A ‘good’ model is one whose limits are known.
Proposition 6: A ‘good’ model is one which is numerically sound. Most of the hydrological models require calibration (at least partially): sound numerical behavior is a prerequisite for smooth multidimensional response surface and efficient calibration.
Proposition 7: A ‘good’ model is one which contains the right equations. However, there is a trade-off, as the « physical purity » of a model is an utopia: we do not know the ‘true’ catchment-scale equations, and upscaling physical properties from the lab-scale to the catchment (or computation unit) scale is a matter of faith. Nevertheless, for equivalent efficiency, one can (or should) favour formulations which have appropriate physical justifications.
Hope it adds up useful thoughts to the topic.
A model is considered good, if the objectives are achieved by applying the model for which it has been prepared eg to control flooding, for use in hydroelectricity etc
hi, when he associates reliable data with the same environmental conditions as his study.
Dear Sir Roland Yonaba,
You should also quote several of the statements you provided as your "minor contribution to the discussion" which are originally from the FutureWater website:
https://www.futurewater.eu/methods/modeling/
Dear Prof S Setegen,
Just a requst, it is apt that in 5years since raising this valued discussion, there have been many responses and guidelines for working at present and for future.
Pl consider a detailed summation and website with RG's nod for benefit of communities on varied functions.
With well wishes for solution providers in present and future.
by comparing the results of this model with the observed values using for example the different parameters, NE (Nash-Sutcliffe model efficiency coefficient), R²; RMSE,...etc
Any model which gives a output close to ground reality is a good model. The output is always as good as the input. If the basic parameters assumed are close to ground reality the output is bound to be realistic.
I just want to add that, fitting models to data and assessing RMSE, NSE, PBIAS, AIC, BIC etc gives only a crude assessment of model suitability. It is always useful to examine the model residuals after fitting. For example the model residuals may be biased against flow (e.g. residuals are larger and more biased at high flow).
I am just new in this research field, and really appreciate to come through these amazing discussion. I used to apply only the best-of-fit indicator (R2, NSE, RMSE, Pbias, relative peak and volume error) to evaluate based on the ground observe, and it's new insightful from your input to investigate the model residuals after fitting.
Thank,
Modeling a hydrological model is a good model when it can predict future changes. Such a model is created through a well-accumulated database in periods of decades or centuries. If the model also describes the surrounding regions, then it will be even more credible. Wishing you success in your research.
برای تأیید یک مدل بایستی اهداف خود را مشخص کنیم که کدام پارامتر توسط کدام مدل شبیه سازی می شود و چقدربا واقعیت مطابقت دارد.. بیشتر بخوانید
To verify a good hydrorological model, it must simply be validated in the field.
In order to measure the accuracy of the models, You should use from statistical metrics such as determination coefficient (R2), root mean square errors (RMSE), present relative error (RE%), Nash-Sutcliffe efficiency (NSE), Kling-Gupta efficiency (KGE) and etc. to see which one from hydrological models for predict paramerers have better performance respect other.
A good hydrological model is a model that is validated in the field. In order to verify this validation, several tests can be applied.
The field measurements, by calibration with it, as the result of the model give a reasonable approach to these measurements as this model is good in the representation of the reality
By correlating its value with various physico chemical water quality parameters.
A model is a mathematical reduction of the field processes. It can most often not be directly verified on the filed, as the conceptual shortcuts they imply just can not be observed.
So the only way to check the validity of a model is to improve its statistical metrics efficiency.
Maybe this paper chould give some help: https://www.researchgate.net/publication/43261199. Meanwhile, I think a model is good if it is useful for one purpose; while maybe not so good for others.
Xu Xiuquan I agree. A model can not capture every components of hydrological processes perfectly. But for a certain purpose, we generally can find a appropriate model.
Hello Mr Shimelis Gebriye Setegn
The performance criteria of a hydrological model can be simple (ratio of simulated and observed water volumes), or be the subject of calculations generally inspired by statistical methods aimed at standardizing the comparison between the result of the simulation or of forecast and observations. Detailed descriptions of these criteria can be found in the work of Nash and Sutchiffe (1970), Beven and Binley (1992), Franchini et al., (1996) and Siebert, (1999). To quantify the performance of the models, there is no universal evaluation criterion. The general principle is to compare the calculated flows with the observed flows. The criteria can be interpreted in terms of quality (adjustment of the model to reality), robustness (conservation of the performance of a model from the calibration phase to the control phase), and reliability (conservation of the performance of a model from one basin to another), Miossec (2004). Many criteria are used in hydrology to assess the sensitivity of the models, including:
1. the Nash Criterion;
2. the correlation coefficient R;
3. the coefficient of determination R2;
To verify the performance of a model you should compare the finding results (model outputs) with the observation (mesured values) , the RMSE, r , R^2 should be calculated also.
A minor contribution to the discussion regarding model performance criteria. There is a very good review from Bennett et al. (2013), discussing the use of such criteria and their relevance for typical modelling objectives.
Article Characterising Performance of Environmental Models
Also, one might find Moriasi et al. (2015a, 2015b) papers interesting for that matter.
Article Hydrologic and Water Quality Models: Key Calibration and Val...
Article Hydrologic and Water Quality Models: Performance Measures an...
Hope it adds up useful elements to the discussion.
Regards,
Roland.
Un modelo matemático tiene que tener la potencialidad de representar el funcioniento de un sistema natural. Debe representarlo con la información disponible. Más precisos más datos requieren. Por ello el mejor modelo matemático es aquel que puede ser alimentado con información precisa y confiable. Implementar un modelo muy elaborado.vonduce al fracaso "basura entra basura sale" "'garbage input - garbage output"
I think one way to assess the suitability of a model for a particular catchment behaviour is the use of model efficiency commonly know in hydrology. Graphical evaluation (plot of observed versus simulated variables) can as well be used.
I recommend you to use the statistical parameters to quantify the matching between measurements and estimated data by model, there are several parameters for this purpose such as root mean square error, the coefficient of efficiency, mean square error,............etc
We judge the reliability of the model by calculating statistical parameters LIKE NSE R2 RMSE RE... also by visual comparison between flow observed and simulated. I talk when we use ranfall runoff model.
Dear Shimelis Setegn, Good models are those that can predict the future thanks to an accumulated database of the past of real research. In the history of meteorological research, the human body has accumulated a lot of data. If this data is collected and analyzed, a very good simulation model can be created. The work is not small, but with a good team a lot can be achieved. Wishing you success in your research.
Existen muchas formas de verificar la bondad de ajuste de un modelo matemático hidrológico. Coincido que se pueden tener en cuenta los análisis estadísticos de la etapa de calibración de los valores observados y estimados. La representación en un eje de coordenadas debería representar una recta de pendiente 1 y ordenada al origen 0. El Rncuadrado debería ser superior a 0.80
The best way by comparing the results of the model with the measurements (in situ values), when the result close to the measurements the model has a good representation of the reality
For verification of good hydrological models, the peak of hydrographs and time of peak should at best match for observed and the simulated model. The performance of the models can also be predicted by evaluation of statistical parameters like Nash-Shutcliff coefficients(NSC), coefficient of determination(R-square), volumetric errors, Pbias etc.
Say that a model is good or that its calibration is satisfactory when:
the difference between the piezometry calculated by the model and the piezometry measured in the field is not very high, that is to say more or less 4 meters,
-The assessment calculated by the model is not wrong, sometimes the assessment given by the model has no link with the reality on the ground. This prameter is to be taken into account, because the calibration is done by trial and error.
It should be noted that sometimes the correspondence between the piezometers is good but the balance sheet does not reflect reality.
I recommend the AvgRelMAE measure proposed in this paper:
Article Measuring Forecasting Accuracy: The Case Of Judgmental Adjus...
The advantages of the AvgRelMAE are summarized and illustrated in this chapter:
Chapter Forecast Error Measures: Critical Review and Practical Recommendations
The major idea of the AvgRelMAE approach is to average relative out-of-sample performances using the geometric mean. This approach was proposed in this thesis (p. 62):
Thesis Integration of judgmental and statistical approaches for dem...
Generally, instead of the MAE you can use another loss function you used to optimize your predictions, but, as explained in the above thesis, the geometric mean is the only correct approach to average relative performances across series.
On p. 63 you can also find the AvgRelMSE measure to evaluate forecasting performance under quadratic loss.
To measure and compare bias you can use the AvgRelAME (p. 64 of Davydenko, 2012).
References:
Thesis Integration of judgmental and statistical approaches for dem...
Chapter Forecast Error Measures: Critical Review and Practical Recommendations
Article Measuring Forecasting Accuracy: The Case Of Judgmental Adjus...
First, we must know our system particularly its inputs, outputs and limits. Once the first simulation has been launched and the results obtained are at our disposal, we must proceed to checking the results and this is done by:
-a comparison between the calculated piezometries and the measured piezometries, the difference between the two must not exceed 04 m,
-the second check consists in studying in detail the balance sheet of the water table. The latter must reflect the reality on the ground. If this is not the case, we must review the assumptions made.
This is general procedure to minimize the difference between simulated and observed values of various significant parameters affecting a particular phenomena. The first step is to determine significant parameters and then evaluate the performance of the model based on performance indicators.
Nah. A good model is one that can be taken to other situations and has a good generality in prediction but maximise the precision and reality as is feasible (Richard Levins). Otherwise, all models will suffer from equifinality. Its possible to get the same measured answer from different combinations of parameter values and also model structure within one system these cannot be teases out.
Good luck it a universal problem and why models have a bad name on top of they=ir good name from the engineering complexity that can produce high resolution in time and space that data cannot
Once you have decided on a suitable measure of how good a model is for a given purpose, you would then be interested in comparing those measures for two different models and trying to see whether one model is better than another, potentially taking account of the different numbers of parameters used for model-fitting. A paper by myself provides a start on doing this in a "formal" way: see Article Statistical Analysis of Empirical Models Fitted by Optimization
In my opinion, a model always has its drawbacks, but the the most important is to have reliable data (Limits, piezometry, flows, permeability, storage, precipitation, evaporation, exchanges between water tables and concentration) well distributed in the ground, and in order not to induce errors (divergent model), it is necessary to understand the functioning of the aquifer (inputs-outputs) which allows first of all to realize a good conceptual model.
A general and straightforward approach to testing any model's capability, provided you have enough data, is cross-validation. For this, you reserve one part of a dataset for model-testing and fit the model using only the (usually larger) subset of data, then run that fitted model to create predictions for the test subset. You can repeat this for different reserved subsets. If you don't like the results produced for the test subsets, then you would judge the model to be unsuitable.
I completely agree with you Hichem, what you say is true but above all fire a first treatment of the data using maps such as those of piezometry, permeability, transmissivity and storage coefficient. These maps are the starting point. In addition, we must make a good discvritization of the groundwater to ensure our choices.
Combine between direct field measurements and hydrological modeling is the best way to judge a model accuracy
- To ensure reliable results of hydrological models, it is essential that the models reproduce the hydrological process dynamics adequately. Information about simulated process dynamics is provided by looking at the temporal sensitivities of the corresponding model parameters.
- For this, the temporal dynamics of parameter sensitivity are analyzed to identify the simulated hydrological processes. Based on these analyses it can be verified if the simulated hydrological processes match the observed processes of the real world. We present a framework that makes use of processes observed in a study catchment to verify simulated hydrological processes.
- Temporal dynamics of parameter sensitivity of a hydrological model are interpreted to simulated hydrological processes and compared with observed hydrological processes of the study catchment.
- The results of the analysis show the appropriate simulation of all relevant hydrological processes in relation to processes observed in the catchment.
- Thus, we conclude that temporal dynamics of parameter sensitivity are helpful for verifying simulated processes of hydrological models.
- I attached the link for more information: doi:10.5194/hess-19-4365-2015
Tout dépend de degré de précision demandé donc de votre objectif de départ
mais d'une façon général, pour un économiste le bon modèle doit reproduire la réalité pour accepter les prévisions
We can consider the model as the best when it will confirm the excellent water movement minimizing the disturbance. It will also ensure the greatest protection of community of bank line.
A hydrological model is considered good, if by its application we get desired information eg water quality, productivity, suitability for various purposes, toxicity, saprobity etc.
There is no single answer to am emvironmental provblem since many problems are quite complex and one problem is connected with another problem. Many hydrological models target one or two major parameters. I have seen that EPA's HSPF model has been used in many systems targeting multiple environmental parameters. It works good in general. Again, this is not universally true. All models have pros and cons.
The verification is mainly based on objective functions, including Nash-Sutcliffe Efficiency (NSE), PBIA, and correlation coefficient (r2). Among those, NSE is a specialized one for hydrological modeling, which reflects the difference between observed and simulated flow. If the NSE value for your simulation is over 0.70, its performance can be counted as acceptable.
Modeling remains within the reach of many researchers and it can be applied in many fields, such as hydrology and hydrogeology, that is to say in an underground environment, this is the case of water tables or superficial. the case of rivers, wadis and lakes. In all cases the models are based on observations made in the field. Each case is a special case, but the menu is the same. This is why the limits of validity of the results are similar but they are not transposable.
by comaprison model results with real case , that's what we called mdel callage.
Un modèle est dit bon quand il tourne c'est à dire quand on modélise un scénario (sécheresse ou infiltration =,...), on obtient des résultats tangibles et vérifiable par le bilan obtenu après simulation.
I suggest you use Pseudo R^2 statistic in case of logit linear regression.
- Badges
- Science topic
- Similar topics
- Mathematics