I have a forecast variable f=[0.2 0.5 0.7 1 0.9] and corresponding observed values o=[0.1 0.4 0.5 0.9 1 ]. How to estimate forecast probabilities and observed relative frequencies for reliability diagram.
The diagram puts 0 to 1, representing probabilities, on the y-axis for observed, and the x-axis is for forecasted probabilities. That seemed backward to me, until I thought that you start with a "forecast" on the x-axis, and then the observed result would an approximate function of that, and thus on the y-axis. I think they might actually be described as "point estimates" of probabilities. Anyway, apparently you look at the point estimate for the forecast on the x axis, and then the observed point estimate is a function of that forecasted value, including a residual - to borrow a term from regression - so it would then be a random variable. You are looking at the performance of the forecast.
So 'perfect' performance would look like a straight diagonal line. But as the authors point out, this won't happen, even if you have a forecast system that is "perfectly reliable." There are inaccuracies, such as not having perfect data, and enough of it. They do go into an attempt to account for inaccuracy because of sample size restrictions, it appears. It may not cover every problem, but the paper presents a good exercise to show that inaccuracies can lead you to the wrong conclusions. I did not peruse the article completely, and it isn't my area, so i do not know how valuable it is, but it does have a number of examples of the "reliability diagrams" you requested, and a discussion of accuracy, so I thought you might be interested.
The society that provides support for that journal, and others online, seems to supply a number of articles free.
I think this might be of value, unless I misunderstood your question.