Dear Madusanka,

the results you get are strange, likely wrong.

If you match the CDF of two datasets, the mean value will be identical.

Anyhow, you can find here a function for CDF matching:

http://hydrology.irpi.cnr.it/repository/public/code/insituvssat

It is applied for matching the CDF of satellite and in situ soil moisture data, but the method is the same it is applied on climate studies. See for reference:

1) https://www.researchgate.net/publication/259299035_Impact_of_climate_change_on_flood_frequency_using_different_climate_models_and_downscaling_approaches?ev=prf_pub

2) https://www.researchgate.net/publication/229237881_Soil_moisture_estimation_through_ASCAT_and_AMSR-E_sensors_An_intercomparison_and_validation_study_across_Europe?ev=prf_pub

Article Impact of Climate Change on Flood Frequency Using Different ...

Article Soil moisture estimation through ASCAT and AMSR-E sensors: A...

In my experience mean will improve but RMSE or correlation coefficient improve not much. The reason is that you are matching the probability distributions to observations but the temporal structure is lost in this method.

Dear Luca and Abha, I did not mean the overall mean. I meant the monthly means are not improving. Thanks for your inputs.

So, you can apply CDF matching separately for each month. This is usually done in many papers..

basically the type of cdf also matters.As in case of extreme events the quantiles of weibull or generalised paerto is matched.So depending upon your datatype...matching particular type of cdf is also important.

- Luca is right.

- If you correct precipitation, please consider also the fractions of 'dry' and 'wet' days (see Piani et al.). Otherwise results will also be poor.

- wich distributions are you using and at which time resolution do you apply them?

Dear Holger, Currently I am using qmap (R package). I changed my code to get the daily precipitation in one month for whole dataset each time and then do the mapping with gamma distribution with 0.1 each step. But I am getting some error so I am trying understand that. Luca provided a matlab code with his link, I am going try that after my battle with "R". I am doing for daily precipitation. Thank you for the input

Monthly transfer function with daily data should do.

Have a look at: Statistical bias correction for daily precipitation in regional

climate models over Europe. Piani et al. 2010. Theor Appl Climatol (2010) 99:187–192. They are not the first to describe it, but this is a standard paper. I also had implemented it in Matlab, worked fine for me. However, as said, the difference between precipitation and other variables is that you must separate / consider wet and dry days. There is no sense in getting the mean of [Rain Norain] right.

Piani et al. acknowledged that by the term "cdf(0)". Actually its simple. I am sure you can implement that easily in R (if it's not there already).

Dear Madusanka,

if you are interested, we could provide you a better Matlab function that addresses exactly your purpose. The one I provided is suited for validation of satellite soil moisture data, not for downscaling climate data (even though in both cases the procedures are very similar!).

Dear Luca,

Please share that with me. Can you please send me a link or email me ?

[email protected]

Thanks for your kind support

I am struggling with the same problem of bias correction of regional climate model and would like any script in R or matlab doing it.

asarr

Hi,

you can find attached the quantile mapping method we have written in Matlab.

Kind regards,

Luca

Thanks Luca, I'll give it a try and let you know.

Cheers,

Abdoulaye

Hi Luca,

I noticed your routine is for monthly data, do you think it is usable for daily data? The work i am doing is for daily rainfall and temperature this is why I am asking.

Thanks,

abdoulaye

Actually, the routine considers daily data. The matching is done separately per each month.

Hi Luca,

Could I also look at your bias correction matlab script?

Many thanks,

Catriona

Here is the routine!

thanks for that!

hi luca i'm not understand this line in your script:

SATCDF= polyval(COEFF,SATmont)+SATmont;

can you help me

thank you

It comes from the polynomial fitting curve as in Brocca et al. 2011 RSE (see attached link, in figure 2b). Hope it helps.

Article Soil moisture estimation through ASCAT and AMSR-E sensors: A...

Hi Luca,

I have two questions:

1)- can you add some comments on your script to be more clear ? especially in these lines:

SATint= interp1(PSAT,sort(SATmont),sort(POBS),'nearest','extrap');

DIFF=sort(OBSmont)-SATint;

COEFF= polyfit(SATint,DIFF, 5);

SATCDF= polyval(COEFF,SATmont)+SATmont;

2)-I' m not understand which data did you used to draw the Figure 2b because the OBSdata and SATdata don't have the same size ? same for the other figures 2a,2c,2d

thank you so much

You can find at the attached link the code with comments.

Good luck!

Data Cumulative Density Function (CDF) matching approach

Hi luca, i' m not understand which data did you used to draw the Figure 2b because the OBSdata and SATdata don't have the same size ? same for the other figures 2a,2c,2d

thank you 

the plots are made by sorting the data and plotting the cdfs. the vector length can be different.

Month is a built-in function in Matlab, that you can easily find in the web.

Dear Madusanka,

Did you (or anyone else) manage to implement your bias correction routine in R using the qmap package?

I am trying to perform bias correction on a gridded data set which is long (about 100 years), so I am looking for the best strategy to get it started.

Thanks in advance,

Thiago

I got the same error likes mr. Kian Abbasnezhadi !

it seem that matlab did not understand my Date (D_OBS). and (D_SAT) as well.

Dear Luca ! Could you please send us your data in excel file instead of dataSM.dat? Does your code is worked with daily rainfall or just only monthly rainfall? Thanks

Tae date are simply expressed in the numeric format of Matlab. To get the correct data in Excel, just subtract 693960.

The code work with daily data, not monthly. The quantile mapping is performed for each month, separately.

Luca, the same logic can be applied to monthly data right? 

- choose a training period for both observed and modelled data, 

- fit a quantile function on this period, 

- apply this function to future modelled data 

@Thiago: I guess yes!

The success of quantile mapping also depends on the implicit assumption that the bias in your data is a monotonic transform from the "true" observations.  to the extent you have other - i.e. non-monotonic- biases, quantile mapping will not be effective - and can even degrade the data you are trying to correct.

Have you done some basic exploratory data analysis to determine the structure of the bias you are trying to correct?

Thiago, did you managed to have progress porting this to R? or anybody else?

Abdoulaye, yes I did. I didn't really port Luca's code to R but I created my own implementation following package "qmap" examples.

Please see the code below:

# Load package and sample data

library(qmap)

data(obsprecip)

data(modprecip)

## call to fitQmapRQUANT and doQmapRQUANT

qm4.fit

Great Thiago, I'll try it and hope it will work? Will let you know.

Cheers,

Abdoulaye

Thiago,

are Moss, GEIANGER and BARKESTAD  stations (points)? For the case I plan for bias correction it will be more difficult because it is spatial unles I do area average.

Abdoulaye

Yes, they are. You could do the same I did - convert your image in data frame (including the timestamp) to run the analysis and then convert it back to an image. The raster package is capable of both reading spatial files and converting them to data frames. If your image is too big (in terms of spatial coverage and resolution), you might need to break it down or run clusters to get it done.

Dear Luca Brocca !

May i know that the 2 maltab code (DBS) and QMAPP you've shared can be applied for the temperatures? Thank you ! 

@ Thiago

It worked just fine.

Thanks

I am trying to do bias correction for CORDEX rcm data using Regression Analysis, but the  results are increasing  compared to the original before corrected, Am asking if there is any means of improving RCM data?

Hi Madusanka and Edmund,

as Stuart worte above - without knowing more about the data and the bias, it is not possible to answer your questions. One easy thing you should do is to calculate for every month the time mean quantiles of the model and your reference data to see how the bias is distributed across the data range. Also check if the time mean of the median differes from the time mean etc.

Hi

I am trying to do bias correction for RCM data using qmap in R.

Could anyone send a script in R for the bias correction. 

Thanks 

Afzal

Dear Muhammad,

checck this two links;

https://cran.r-project.org/web/packages/qmap/qmap.pdf and https://github.com/SantanderMetGroup/downscaleR/wiki/Bias-Correction-and-Model-Output-Statistics-(MOS)

You may try to get bias corrected data from CCAFS portal (http://www.ccafs-climate.org/data_bias_corrected/) for use in impact studies. You are required to provide location (lat & lon) and select scenario(s) (Historical, RCP8.5, 6.0, 4.5, 2.6), CMIP5 models, reanalysis dataset (AgCFSR, AgMERRA, GRASP, Princeton, Station, WFD, WFDEI), period, variables, correction method and output format.

I think there is no need to complicate things. Bias=observed(quantities)-predicted(quantities). But note that the location of the station should be the same for the predicted and observed quantities. Get the departure from the historical record and apply it for the future projections to correct the RCMs. It is impossible or meaningless if you get the departure at station B and use it at station A. The correction should be applied at the same station or location. 

Hello everyone;

I have used the "QMAP" (Quantile Mapping) package in R for downscaling of future rainfall and temperature projections. I have an observed data period of 1980-2010 and I want to use it for downscaling the future variables. But when I run the package, the output data of the package has the same characteristics as the observed data. Do you know how can I apply "QMAP" to downscale future data with my observed data?

I have attached a file for more transparency.

Regards

@Thiago Veloso dos Santos ! I am using qmap for soil moisture data. When I use "fitQmapDIST()", there is an error like this ("Error in fitQmapDIST.default(obsprecip[, 1], modprecip[, 1], distr = "bernweibull", :

'mledist' failed to estimate parameters for 'mod' with the error code 1") can you help me? thank you very much

Stuart S. Schwartz sir, can you please suggest some method or how to check the monotonicity? Are you suggesting about systematic bias?