Which area are you working with? I'm working with a Groundwater model for North-Africa (Modflow-NWT with Model Muse). I think, as you said you have large residuals, your model is still not ready for automatic optimization/ calibration and I think you should revisit other conceptualization of your model first. There might be something wrong with the data inputs, assumptions (aquifer thickness, type, spatial variation of HK, initial heads etc.). I haven't tried running PEST or any other automatic optimizer for my model, rather I emphasized more on the conceptualization of my model and tried changing my assumptions (initial heads and a varying aquifer thickness) manually, and there has been significant improvements in the model, even though I am not done with calibration.
There are many ways to create the pilot points, but if you know python (or are willing to learn) the pyemu software has some nice tools that make it pretty easy.
https://github.com/jtwhite79/pyemu
However, there is a good chance Fahad is correct. If your parameters are hitting unreasonable upper or lower bounds during the calibration you most likely need to reevaluate the model. More than once my models have produced unexpectedly high residuals... more often than not there was an error in either my observation values (e.g. flow rates (L3/T) rather than annual averages or cumulative volumes) or model structural errors (e.g. wrong boundary conditions, especially specified head or thin layers that required huge K in order to simulate the same transmissivity).
I'm working on a model of Mzab Valley (North Africa). It's an arid region, so there is no river conditions. However, I made a drain for a downstream zone where water emerges from the aquifer.
I want to calibrate the model on steady state. I calibrate the model manually by changing hydraulic conductivities, and residual errors deacresed significantly. But continuing in the same approach, and after changing only one value of hydraulic conductivities, the model diverges.
Regarding observations, I have heads on the entire zone (>120 values) and one flow observation on the drain downstream.
I try to use automatic calibration, using zonal PEST and the model also diverges.
Until now , I haven't try pilot points for calibration. So, it's interesting to try pilot points calibration to see what it gives.
By "diverges" I assume you mean the residuals increase dramatically?
Are your weights (in PEST) reasonable (e.g. 1/std)?
Are the head or flow residuals increasing? My guess is flow residuals...
If your flow residuals are large, make sure the model output makes sense (i.e. is the flow in your downstream drain unreasonably high?). Make sure your MODFLOW-NWT solution is converging and has good mass balance errors (percent discrepancy < 1%). If not, you may need to adjust your NWT parameters (e.g. DBDTHETA, HEADTOL, FLUXTOL).
In fact, for direct modelling it was diverging because inner iteration in PGC solver was too high, which provoque oscillations on wet/dry conversion. This enhance divergence probability. So, I decrease the number of inner iterations in the solver, and it works.
Then I calibrate hydraulic conductivities of the model manually to a certain extent and launch Automatic calibration with PEST module.
However for PEST calibration (Zonal) type), the error doesn't decrease and gives the same hydraulic conductivities in each iteration.
This is a tough one to solve without the model files, but here are my thoughts. What is the objective function (phi) when you run PEST? What percentage of the objective function are the head observations versus the one flow (drain) observation assuming you are using two observation groups? Depending on the magnitude of the flow out the drain, you may need to decrease the weight by several orders of magnitude in order for its contribution to phi to be roughly equivalent (less or more depending on your calibration objectives) to the head observations. You want PEST to be able to match both the heads and the drain flows not just one or the other. Also, look at your SEN and SEO file to see which parameters and observations, respectively, are the most sensitive to the calibration. However, if the model is unstable, you need to iron that issue out first because PEST will not help you calibrate, if the model doesn't converge every time it runs the model. Good luck.
Decreasing the number of iterations will cause the solver to stop before converging. That will likely give you large mass balance errors (percent discrepancy) and your simulated values will not be meaningful. If anything you would want to increase the number of iterations so you can hopefully find a solution.
What are your convergence criteria (head change and residual change)? Make sure you have MUTPCG = 0 so you can see what cells are not converging... this may give you clues about your model construction and structural errors (i.e. if you have very thin cells that are drying out, causing wet/dry oscillations, etc).
Before you do anything in PEST you need a stable, convergent model with low mass balance errors and reasonable initial parameter values. Otherwise your sensitivities (Jacobian) will be meaningless and cannot guide parameter estimation.
After searching and trying different possibilities, I found that the problem was in PEST parameters, exactly I made convergence criteria (PHIREDSTP) and (RELPARSTP) lower in order to push PEST not to converge rapidely, and it decreses the error considerably. But it remains not sufficient for my model. I envisage to enhance calibration by giving superior number of parameters (hydraulic conductivities from 40 to 80) in order to give PEST more flexibility) , or inserting pilot points with initial conductivities from the best calibrated model. Which strategy do you think that is more helping?
However, I wish you could help on how I could use parameter sensitivities to enhance calibration process?
Do you mean you made RELPARMAX, FACPARMAX and/or ABSPARMAX lower? These variables control how much parameters change while seeking the minimum objective function. PHIREDSTP and RELPARSTP only tell PEST when the parameters are "good enough" based on how much PHI has changed.
Adding pilot points will also add more parameters. If you add 100 pilot points you will have 100 more K parameters... or 200 if you are doing horizontal and vertical K. If you have multiple processors to run a highly parameterized model, I think pilot points are definitely a good approach. However, if you are limited to a few processors you need to carefully consider the trade-offs between parameters and processors since each parameter means another model run.
I am still concerned about your statement:
"for direct modelling it was diverging because inner iteration in PGC solver was too high, which provoque oscillations on wet/dry conversion. This enhance divergence probability. So, I decrease the number of inner iterations in the solver, and it works."
Because decreasing the number of iterations in the solver will almost certainly cause the solver to stop before reaching convergence. This is a SERIOUS problem because it will corrupt your Jacobian and render the PEST process meaningless... PEST will be chasing a PHI dominated by numerical noise rather than your observations of interest.
I highly recommend you take a close look at your process model (MODFLOW) before running PEST:
Ensure the model is converging! If the number of iterations reported in the list file is equal to the number you specify in the solver input file, your model is not converging.
If your model is not converging, look at the non-convergent cells (row, col) reported in the listing file. Make sure all input values make sense... check units on parameters, boundary conditions, etc. High K contrasts (orders of magnitude) can also cause the solver to not converge, so occasionally you may need a transition between high and low K zones.
Make sure your mass balance errors are small. Look at the "percent discrepancy" reported in the listing file. If your percent discrepancy is greater than 1% you may want to reevaluate your MODFLOW model.
Use the budget table in the listing file to make sure all the IN and OUT values make sense.... for example, if a boundary condition is provided in cubic meters per day but the model is run in cubic meters per year the corresponding item will show up as a very low value in the budget table.
Makes sure your head tolerance and flux tolerance is set low enough to produce a meaningful Jacobian.
Actually, I realize that the problem was in the conceptual model, so I rebuild a new conceptual model appliying principle of parsimony, with a few hydraulic condctivities (7 parameters) based on geologic pattern. So, I succeed to build a converging model.
PEST succeeds to estimate parameters and reduces model to measurement fit while insuring good derivatives and problem well-posedness by cheking output files (*.cnd, *.mtt, *.sen).
Surely, with parsimonious model (7 K parameters against 40 K before), well-posedness is insured but it becomes more difficult to reduce model fit to an acceptable limit. So, after estimating K parameters with PEST, I launch the model with pilot points.
However a new problem arises when I found that misfit is more important for pilot points model. Any suggestions?
note: I have used SVD regularization for pilot points.
I don't think 7K vs 40K parameters "insures" well-posedness for an GW model, even if you have a huge number of head and flow observations, but using prior information on your parameters (Tikhonov regularization) can help. Are you using prior information?
By "misfit is more important" do you mean your phi is higher or that the simulated vs observed values are greater?
I mean that I have verified well-posedness through PEST output files and it's ok. (*.mtt, *.cnd, *.sen) like indicated in the document intitled: "Approaches to Highly Parameterized Inversion: A Guide to Using PEST for Groundwater-Model Calibration" (https://pubs.usgs.gov/sir/2010/5169/pdf/GWPEST_sir2010-5169.pdf).
Regarding pilot points calibration, I used SVD regularization. However, I do not possess much prior information values, and when I insert those I have using with preferred values Thikonov regularization, I receive an important bulleyes on resulted hydraulic conductivities map.
I mean by misfit that simulated vs. observed values are greater. And, I know that this is not due to high phi, because PEST stops after an indicated numebr of iteration of non-reducing phi (In my case the number is 3).
If simulated vs. observed values are greater, your phi should be higher unless you have changed your weighting (either on obs or prior info).
One way to avoid "bulls eyes" is to use prior information that promotes homogeneity where the value of each point is equal to the value of the adjacent point:
You can define as many of these equations as you like; the more points you "link" with this prior information the more your prior information will push the parameters toward homogeneity.
Or, you can use a mean value (e.g. meanK parameter) and multiply that by an array representing heterogeneity with prior information as:
pr_pp1 1.0 * log(prior_pp1) = 0.0
pr_pp2 1.0 * log(prior_pp2) = 0.0
Where K in each cell is the array value (from interpolation/kriging) multiplied by meanK (the average of all K values in that zone/geologic material. The prior information equations above push the parameter values towards the mean.
To calibrate the model, the first step is to know the area. What are the sources of richarge and drainage of the aquifer, and how much they both drain or richarge the aquifer. So you need to read a lot of reports on this. Then make sure in the model that the data you provided is not logical and does not exist. Then, according to the observation wells position, change the amount of richarge and drainage in different regions.
Note that when the error is high, it means that the wrong information has been entered and must be checked. Because the PEST does not respond when the error is high.
Trial and error approach is the best method for calibration process of a model. if your model can not be calibrated, you most correct the boundary conditions and then run the model again.