Thanks, Guillermo! I had the same thought about the difficulty with interpretation and meaning, particularly RPSS. Essentially, I'm interested in is how far off the forecast is compared to observations in each grid cell in a given month of the year. To me, RMSE and MAE seem appropriate to answer that.

Another restriction might be, I might only have about 25 years of observations (depending on the variables, which are determined by user surveys), i.e., a sample size as small as 25 for each month (definitely not more than 30, which is the limit of the model hindcast years). I assume that prohibits a lot of sophisticated analysis, such as sorting years by El Niño, La Niña, and neutral, and maybe even looking at skill for extreme cases (e.g., unseasonally dry/wet/cold/hot months).

A simple and powerful approach is ranking the forecast outputs within a probabilistic framework and evaluate that using the Information theory and the cross entropy in particular to compare with other potential source (e.g. climatology). It si very simple to be computed and it is quite powerful.  It could be simply applied even in conceptual experiments regarding specific selections: ENSO years or other.

Here a paper on that: Riccardo Benedetti, 2010: Scoring Rules for Forecast Verification. Mon. Wea. Rev., 138, 203–211.

doi: http://dx.doi.org/10.1175/2009MWR2945.1

http://journals.ametsoc.org/doi/abs/10.1175/2009MWR2945.1

It depends on what attribute of the forecast you want to measure. I recommend you take a close look at Jollifre and Stephenson, 2011: Forecast Verification: A Practitioner's Guide in Atmospheric Science, 2nd Edition.

The International Research Institute for Climate and Society (IRI) has part of their web site devoted to verification of realtime seasonal forecasts using a variety of metrics, because different applications of the forecast will require a different measure of quality.

http://iri.columbia.edu/our-expertise/climate/forecasts/verification/

On that page there is also a link to a document that goes into more detail on the scores and how to calculate them: http://iri.columbia.edu/wp-content/uploads/2013/07/scoredescriptions.pdf

Toni, to the simple skill metrics also belongs anomaly correlation coeff. Usually both are used to get an impression about forecasts performance, while the rmse says how accurate the forecasts are, the corr coeff says about the level of correspondence between forecasts and observations. Here is also a nice link to a summary of different verification metrics http://www.cawcr.gov.au/projects/verification/#Introduction

Thank for sharing, and thanks for the insights, Iuliia! I discovered this website last Friday, too. It's great! Another good one for ensemble forecasts is http://www.cawcr.gov.au/projects/EPSverif/scores/scores.html.

Also, CPC on their Climate Test Bed website has a document called "NCEP Metrics for Evaluating Climate Models and Forecasts" to verify deterministic, categorical, and probabilistic forecasts with: http://www.cpc.ncep.noaa.gov/products/ctb/ and http://www.cpc.ncep.noaa.gov/products/ctb/Metrics_climate_models&fcst_15May2015_v2.docx.pdf

Hi Toni

there are a whole range of scores which can be split into deterministic and probabilistic types but I would stay away from scores that are affected by mean bias, as the mean bias is normally removed from seasonal forecasts. 

If you are measuring something with a strong signal like the El Nino Southern Oscillation then I would recommend a centred correlation score, which is often used.

For fields like rainfall you could take a regional mean and then again use correlation, but it would also be good to look at ROC scores and reliability for pooled points over some region.  Reliability is often emphasized as an important quality measure as it means the forecast probability matches the observed frequency which is needed for good seasonal forecasts.  Note though that many probabilistic scores are affected by spread which could in principle be added to the forecast ensemble after it is made, whereas although they are often criticized for being deterministic, correlation scores can not be improved in this way!

If you want to see some scores in use then check out the paper by MacLachlan et al. from 2015 where various scores are used on a very new seasonal forecast system: http://onlinelibrary.wiley.com/doi/10.1002/qj.2396/pdf

Best regards

Adam

As Angel Munoz suggested, I would definitely recommend the book by Joliffe and Stephenson on forecast verification.

Regarding the choice of scores, my feeling is that using more "simple" scores helps convey key messages, but probabilistic scores provide crucial information in terms of forecast reliability.

I usually use both deterministic (correlation, RMSE) and probabilistic scores (Brier Score, RPSS) to get a clear picture. The key is to clearly grasp what it is you are verifying when you choose one score rather than another, and for this the links provided above by other RG contributors can really come in handy!

In addition to all the useful information that the other RG contributors have provided I would say that since there is no single score that can summarise all the model skill in a single output, it is important to firstly know which aspects of the forecast you value most and, after that, go for the score/s that can best assess these features.