Hi Gabriel,

First you need to identify how you intend to use the downscaled data, as there are multiple methods (and types) of statistical downscaling. Are you interested in daily means, extreme values, duration of spells? Also do you plan to use the transfer functions to obtain future projections of your climate variables of interest, and if so what is your time horizon?  

Anyway, to answer your question, after performing PCA analysis on your ERA-40 output, one option is to regress the leading principal components (predictors) versus your climate variable of interest (observed at a local scale). Then you repeat the process 90 times.  Other alternatives, involve creating synoptic map classifications of the leading PCs and then matching individual daily events with a certain pattern. 

With both approaches you will need to divide the data in training and evaluation datasets, as usual. So you can asses the performance of your models.

Best,

Thanks for replying guys. Paul, your paper is sadly way beyond my current level of understanding. I am grateful for the effort though and maybe I will be in a position to try out some of the things you suggested somewhere down the line.

Carlos, I will answer your questions one by one. Hopefully, you'll get an idea of what I want. Feel free to ask anything further.

I am interested in daily means mainly. That is my current area of focus right now. When I have further increased my knowledge of this field, maybe I can try extreme events.

Next, yes, I want the transfer functions to obtain future projections maybe to compare with possible GCM simulations. I found CMIP5 projections that contained predictions up to 2070.

As for the method, I have the NCEP2 and the ERA40 data for the region for the years 1979 to 2002.

The predictors I used are:

1) Air Temperature at 2m

2) Mean Sea Level Pressure

3) Geopotential Height at 500MPa (not sure about the unit)

4) Specific Humidity

I have a grid of 9 latitude x 10 longitude x 12000 days for each predictor. How do I arrange this data to feed it to PCA?

Up till now, I'm taking each location (so total of 90 locations) and each variable per day. So I form another table of 360 (ie, 90 locations x 4 variables) x 12000 days.

I performed PCA on this table and I took the principal components for which the eigen vectors contribute up to 98% information. Let's say that is 50 PCs.

So I get a new table of 50 x 12000. I'm confused about what to do after this. Please help if you can.

Hi Gabriel,

     If I understand your problem correctly, after you obtained the 50x12000 matrix, you will need to relate this matrix to your 'predictand' variables by mean of transfer functions. This is observed local variables of interest. Let's say you are interested in streamflow, then you would have K x 12000 matrix, holding the values of your predictand at K locations.  So, 50x12000 is related/associated to Kx12000 using some kind of transfer functions.

   Hope this clear your doubt.

Cheers

Hi Gabriel,

So after you have your matrix with 50 PCs x 12000 time levels, you relate those 50 coarse scale predictors to your local scale predictand (e.g. weather station data) of size 1 x 12000. You can also try regression methods that relate the 50 PCs with multiple predictands,... so the most simple linear regression will be Y=AX+B, where Y is the weather station data, X your matrix of principal components and A, B coefficients of the regression.

Cheers,

I'm not getting the correct output here. I don't understand why.

This is what I'm doing step-by-step.

1) Extracted the data from ERA40 and CGCM3.1 (T64) for the four variables as listed above into separate 3D matrices (longitude x latitude x time).

2) Interpolated the GCM data onto the grid points for ERA40.

3) Rearranged the data so that I get a 2D (location x time) matrix for all variables. [Here, location is a combination of longitude and latitude]

4) Since I'm only interested in the monsoon months, I extracted the data only for the days of May - August time period of every year.

5) Combined the data into one matrix as follows:

      a) Each row contains the data for a single day.

      b) For each row, I have the columns as [location 1, variable 1], [location 1, variable 2], [location 1, variable 3], [location 1, variable 4], [location 2, variable 1], [location 2, variable 2], [location 2, variable 3], [location 2, variable 4], [location 3, variable 1], etc.

6) Performed PCA on this matrix. Taking only the top 80 PCs. (I'm assuming the PC's are the x values in R's prcomp output and the Eigen vectors are the rotation values.)

7) Since I have to use the same "principal directions" (which I assume to be the Eigen Vectors), I found vector projections of the GCM data on the ERA 40 vectors.

8) I performed clustering on these PCs using c-means clustering with 3 clusters, taking the membership of any 2 clusters as the output.

9) I combined the 80 PCs and the 2 memberships to get a 2D matrix of size time x 82 for the ERA 40 and for the GCM data. (Hence 2 matrices).

10) I used the ERA 40 data set created in (9) above to train the svm implementation of e1071 package of R with its automatic scaling.

11) I used the model trained in (10) above to predict the output using the GCM data set created in (9) as the input.

Guys, please help. I have no more ideas on this.

I've done everything exactly as I should. I'm using verified packages that are proved to have no errors. I cannot solve this anymore. I don't even know what is going wrong.

Hi Gabriel,

   I am not sure what kind of errors that you encountered, since you didn't describe your errors in detail. However, are you expecting your step 11) to produce a daily downscaled output that matches your observation in time (one-to-one)? Having said that do you expect the observed time series and the downscaled time-series to have good correlation? If this is the case, you would not get it.  

   I am not quite sure what's the nature of your GCM simulations. However, since you're doing this on daily basis, the GCM using in climate simulations are not expected to generate exact day-to-day variations as those of the observed. What you want to obtain is actually the correct 'statistics' in the climate sense. If your GCM is the climate projection type, you'd not even get the year-to-year variations. Having said that, you would not expect to find 97/98 El Nino in those GCMs runs as you would in the observation. What matter is the over all statistics.

    Hope this helps.

Thanks Liew. I understand that using multivariate regression does not produce good results. My aim is to show that fact and to try to improve it by designing a neural network or deep learning approach to the problem.

Actually I found the problem with my case. It seems there is an extra step in the svm function of e1071 package of R. It performs scaling of the data automatically. So I didn't need to scale the training data separately.

But I also scaled the testing data once by the normal method and then again to normalize it to the training data. I have no idea whether it is accurate or not but it gave some results with an error of order 10^-4.

Paul, I respect your expertise in this matter and I completely agree to your explanation. Even I doubt how weather or climate can be so simply related. But I have little say in the matter. My project guide wants me to do it this way, so I am doing it in her way. I'm not really in the climate science field so I'm apprehensive about being able to fully comprehend your paper. However, this project also gives me a chance to learn something about data analytics.

Dear Gabriel Broadwin,

have you tried to use new methods for downscaling? I worked with artificial neural networks for downscaling (paper attached), for it developed a way to insert data from IPCC AR4-models and observed data (eg rainfall), I am with a Phd student using neural networks to identify uncertainties of the models the AR5, and I can tell you that are encouraging the use of neural networks.

Best, David