In my opinion, the following aspects need to be considered when comparing estimation models:

1) Method must be Objective

a) Explainable, Understandable: Easy to learn

b) Verifiable: Leaving no room for discussion

c) Reproducible, Repeatable: Repeating the estimate by self or others leads to the same result

d) Simple to do: Taking a reasonable amount of time & effort relative to the work estimated

2) Outcomes must be Correct

a) Complete: What is excluded and what is included must be clear

b) Precise: A realistic 90% probability range, no pseudo-precision or single number in relation to the purpose of the estimate (ballpark, contract, ...)

c) Accurate: The actual result should fall within the given range 90% of the time

Increasing aspects under 2) typically mans compromising aspects under 1).

greetings,

Paul Siemons

Let me point you to the attached paper, which is related to this topic.In said paper, the performance several objective models are compared against each other, once some basic objectivity and congruence guidelines have been established, as well as particular indicators are selected.

In particular, the Mean Magnitude of Error Relative to the estimate (MMER) was used.

I think that the discussion in there can give you some pointers to address some of the ideas that Paul Siemons and Robert Schaefer mentioned before.

Article Application of Gamma Classifier to Development Effort Predic...

I would like to suggest another measure in the place of the MRE.

A proper estimate indicates a reliability interval, i.e. E+-e with P=90%.

The estimate is accurate when the actual value A falls in the interval [E-e, E+e].

Let the Quality Indicator Qi for the estimate be defined as Qi = | A - E | /e

Then Qi < 1 means the estimate is accurate and Qi > 1 means it is inaccurate. In fact, the closer Qi is to 0 the more accurate the estimate was.

The beauty of this approach is that is can be applied to compare initial ballpark estimates [with wide ranges] against later detailed estimates [with smaller ranges].

For MRE this is not the case because the width of the range [reflecting the uncertainty in the estimate] is not taken into account.

Take a look at these two estimated ranges E1 and E2 and the actual A:

[-------------E1------------]

[--E2--]

------------------------A-------------------

In this situation E1 is an accurate estimate because A is within the range while E2 is inaccurate because A lies outside its range. Using the MRE formula would yield the inverse result because A is closer to E2 than to E1 in absolute terms. Hence, in my opinion, the MRE is a totally useless construct. for the calculation and comparison of estimation accuracy..

for what it is worth :-)

Paul Siemons

After adding the text, my scheme got mixed up; please refer to the scheme below:

[-------------E1------------]

[--E2--]

-----------A-------------------

Actually, in a previous work (which I think is cited in the paper I mentioned) a similar analysis was discussed regarding MRE, which yielded MER as a better indicator of accuracy. Now, the Qi could be interesting :)

Some of you hear anything about Agile Methods? I use it to estimative, planning and project execution.

Yes I have heard about it, however agile methods are used very rarely and less frequently. There are more ways and methods that can help in project estimation and management like:

1. Lines of Code (LOC) based estimation

2. Function Points (FP) based estimation

3. Object Points based estimation

4. Use-case based estimation

5. COCOMO - II model

All these estimation techniques provide you with the effort and cost depending on the type of project it is used to and its complexity.

Thanks

Syed Usman Ahmed

Yes!, in fact the problem of estimation in Agile Methods is that it give more priority to things that have more value to the user, so, the other things have estimation done after the start of the project, or are never estimated and never done. In this way you have an idea of the end of the project, but you know that it will have changes, because you are always re-estimating and re-priorizing the work that you have to do.

Here is a big issue in evaluating the effectiveness of an estimation model. The effectiveness, in simple terms, can be defined as the variance between the estimate and actual. The actual effort varies due to many factors (requirements volatility for one) as pointed by various project management studies. You can collect the variance for many projects but while analyzing you have to account for the variance due to many of the project management methods that could have caused an increase in effort. Without such an account, the variance between actual and the estimate cannot be attributed just to the estimation method.