Dear Asong,

You are asking an interesting--and complex--question. Before you spend too much time in evaluating particular metrics, or at least in parallel with that effort, you might want to consider exploring, discussing and assessing the nature of the difficulties inherent to your question:

- Ground-based observations of precipitations are point measurements, while satellite remote sensing estimates are areal values: there is a spatial scale issue. Also, precipitations stations on the ground are usually few and far between, while satellites acquire data everywhere, or, more accurately, at a large number of locations, typically organized as a dense grid (pixels).

- Ground-based observations of precipitations are normally time-integrated measurements (i.e., accumulations over predefined periods), while remote sensing-based estimates are typically instantaneous. Also, the timing of satellites overpass may be fairly arbitrary with respect to the timing of ground acquisitions.

- Ground-based observations of precipitations are often close to direct measurements (little interpretation is required), while satellite remote sensing estimates are derived using more or less sophisticated, always indirect algorithms, and the results also depend on the performance of mathematical inversion methods.

- The accuracy and precision of direct measurements (on the ground) and of derived estimates (from space measurements) are likely to be quite different.

- Precipitations include such subtle processes as dew deposition, drizzle, rain, hail, snow, etc. Most ground-based instruments are optimized for liquid rain, with limited or no capacity to measure other types of precipitation with an acceptable accuracy. Similarly, satellite-based techniques may be appropriate only for certain types of precipitation, and only within certain ranges of values.

- For the purpose of validation exercises, it may actually be useful to acquire information from multiple sources, so you may want to consider benchmarking your ground-based observations and satellite-derived estimates in an area that is also covered by a weather radar: that instrument generates data that can be interpreted in terms of precipitation rate. Indeed, the backscattering cross-section of a single droplet is proportional to the 6th power of the droplet radius and the inverse 4th power of the radar wavelength: see http://marine.rutgers.edu/dmcs/ms552/2009/radarbasics.pdf. So radars can be quite sensitive to the size distribution of precipitating droplets when operated at the appropriate frequency.

The point is that your validation exercise can be based on specific metrics, but these should be designed to address these issues. They should also be operated within specific spatio-temporal domains, keeping in mind the limited range of sensitivity exhibited by each instrument to the different types of precipitations. It is unlikely that any given metric is better than all others under all circumstances.

In the end, the problem is that Method A gives you result X while Method B yields result Y, and the metrics tells you how far apart these are from each other. But how do you decide which one to trust? A possibly more convincing approach is to individually ingest the various sources of information into a three- or four-dimensional variational data assimilation (3D- or 4D-Var) system, and quantify which data source generates the more reliable results. That may represent quite an investment in learning, technology, etc... but the evaluation method yields an independent measure of pertinence and quality.

Good luck with your investigation. Michel.

I fully agree with Michel, validation of satellite precip is not an easy task.

Anyhow, for which application are you planning to use satellite precip data?

A practical validation strategy could be to use satellite precip data in your application and check if you are able to improve predictions for your specific purpose. I you will improve results wrt the use of ground observations, it will be highly relevant and interesting.

Regards, L

The are also signal losses and this can be done by applying interferomatry to gain those that are lost during data acquisitions. 

Dear Asong,

This article could be useful for you:

Toté, C., Patricio, D., Boogaard, H., van der Wijngaart, R., Tarnavsky, E., & Funk, C. (2015). Evaluation of Satellite Rainfall Estimates for Drought and Flood Monitoring in Mozambique. Remote Sensing, 7(2), 1758-1776.

I hope this helps a bit.

Very insightful contributions from you all. I am beginning to get a feel of the complexities involved and the level of "accuracy" one can ascribe to satellite products based solely on validation with ground-based observations. As @Michel mentioned, one has to be careful in making conclusions here given the various sources of uncertainties involved. 

Again, the purpose for doing this is important (Luca and Elena above). Having said that, I'd rephrase your question by asking: How and to what extent can satellite based sensor returns improve the estimation of [streamflow, soil moisture, etc]? I put it in these terms because often we do studies driven by funding which comes from platform focused stovepipes and forget that the user is less interested in the stovepipe and more interested in using all the information available (multi-sensor estimation) in estimating something else.

I say rainfall rather than precip because categorization into various forms of precip adds a lot more complexity.

I say satellite sensors rather than satellite based rainfall or precip because satellites have a wide variety of sensors and time-space characteristics which can provide information. 

A good example of this is:

David Kitzmiller, et al, 2011: Evolving Multisensor Precipitation Estimation Methods: Their Impacts on Flow Prediction Using a Distributed Hydrologic Model. J. Hydrometeor, 12, 1414–1431.

doi: http://dx.doi.org/10.1175/JHM-D-10-05038.1

You might also look at articles about NMQ such as:

Jian Zhang, et al, 2011: National Mosaic and Multi-Sensor QPE (NMQ) System: Description, Results, and Future Plans. Bull. Amer. Meteor. Soc., 92, 1321–1338.

doi: http://dx.doi.org/10.1175/2011BAMS-D-11-00047.1

Steven V. Vasiloff, , 2007: Improving QPE and Very Short Term QPF: An Initiative for a Community-Wide Integrated Approach. Bull. Amer. Meteor. Soc., 88, 1899–1911.

doi: http://dx.doi.org/10.1175/BAMS-88-12-1899

and then follow the work of authors such as David Kitzmiller, Yu Zhang, DongJun Seo, Jian Zhang, Ken Howard, Steven Vasiloff, Robert Kuligowski

and articles about CMORPH and PERSIANN

I would like to add my two cents to this wonderful discussion. 

When we talk about the utility of a satellite product (in terms of hydrologic utility), there is statistical and hydrologic evaluation.

My general approach for statistical evaluation goes as below:

1. Grid wise correlation and bias computation (although I prefer to do my analysis at a catchment scale) can give interesting spatial patterns. The decomposition of the overall time series into low, medium and high rainfall intensities gives an idea about the systematic dependence of errors on rainfall regime.

2. You can then try to investigate the systematic dependence of the performance (in terms of correlation/bias) on topography (in terms of mean grid elevation) and climatology (in terms of mean grid rainfall). This will give an idea into how the satellite rainfall estimates vary in different climatic and topographic zones, and whether there is a general trend to it.

3. Next comes threshold statistics (probability of detection, false alarm ratio, equitable threat score, etc.). The paper below summarizes all the threshold statistics. I prefer doing such an exercise separately for different rainfall regimes.

Schaefer, J. T., 1990: The Critical Success Index as an Indicator of Warning Skill. Weather Forecast., 5, 570–575, doi:10.1175/1520-0434(1990)0052.0.CO;2. 

4. You can also use the Taylors diagram and Bland-Altman plot.

For hydrologic evaluation, you can look at different model performance indices while forcing the model with satellite precipitation data. Depending on how long the satellite data is, your calibration methodology may vary. The below paper uses an interesting calibration mechanism while evaluating 1-year of IMERG data.

Tang, G., Z. Zeng, D. Long, X. Guo, B. Yong, W. Zhang, and Y. Hong, 2016: Statistical and Hydrological Comparisons between TRMM and GPM Level-3 Products over a Midlatitude Basin: Is Day-1 IMERG a Good Successor for TMPA 3B42V7? J. Hydrometeorol., 17, 121–137, doi:10.1175/JHM-D-15-0059.1. 

Hope I could be of help. We currently have a manuscript under review which shows all the statistical and hydrologic evaluation in much detail. Once it is out, I can attach a link. 

Cheers!

the following paper may help you find the answer to your question:

Article Evaluation of the Global Precipitation Measurement (GPM) Sat...

Hi

We publish this short article in a early career journal evaluating two P global products recently produced, making the assessment with the precipitation indexes of climdex.

This allows to assess long term variations of climate, not just the overall performance or error analysis as you were asking. These indexes represent the more extreme aspects of climate and a validation using them is essential in order to gain more insights about the estimates of satellite or reanalysis data.

Hope to be helpful.

Article Analysis of precipitation features estimated by reanalysis d...

Regards

Hi

If you use Precipitation from remote sensing products, you can improve their performance using local raingauges. The merging algorithm described in this work have proved to be successful in reducing the remote sensed field error. In the paper the parameters of the merging are analyzed against the raingauges density so that one defines their value with the features of the local raingauge network and, therefore, one will not need to perform a calibration procedure.

Conference Paper Analysis of the kernel bandwidth influence in the double smo...

The Water Cycle Integrator of the eartH2Observe project have several global precipitation dataset at

http://www.earth2observe.eu/

Regards

Hi, After validation you can extent the time frame of data that you have and improve its accuracy by merging information. Want to improve the performance of your hydrological model on a scarce data basin?!! We introduce you to our new article where a satellite-reanalysis-gauge merging algorithm is detailed. Article Improving Rainfall Fields in Data-Scarce Basins: Influence o...