As shown by our experience in evaluating small projects (up to 1-2 million) - the implementer (both the CSO and the International Organizations) have neither the resources nor the capacity to make a high quality project evaluation. The result is a game of imitation evaluation:
The customer puts in ToR 20-30 evaluation questions (which he found in online) with no budget, with limit of 15-20 working day (including field mission), ask detailed methodology without access to project documentation etc.
The evaluator taking his standard offer, change keywords and use other tips to imitate tailor-made methodology. Later, as such calls usually for individuals, the evaluator is limit his/her work by interview and desk review without in-depth stakeholders analysis, triangulation etc.
Usually such project has no clear logical model, no any M&E system. So actually there is no any chance to talk seriously about efficiency and effectiveness at all.
But 80% of calls for proposals in developing countries are such.
Anyone familiar with any mechanism for organizing a high-quality evaluation of a small-scale project without profaning the process itself? Are there any publication or tools/standardized methodology solving such case? If not - Is there a demand for such a tool / methodology? Do you know professors who may be interested in such research and development?
Haji Saediman , thank you for your replay. You are right - and this is exactly what I am concern about.
Let imaging rural development project funded by EU and implemented by CSO/NGO. Total budget let say 300 000 euro. Budget for evaluation and auditing can not be more that 3-5% of the budget - so, for evaluation they can allocate maximum 10 000 euro. Travel and allowances for 10 day mission for 2 people is 3-4k euro minimum. Local transport, logistics...
So... for 3000-5000 euro you are in choice:
1. do good work development methodology, do stakeholders analysis, good and correct sampling, triangulation. And you have no budget for data collection in this case.
2. use standard methodology and instruments, little-bit tailored for the project during desk review, and be concentrated on quality of collected data and it independence.
Good evaluator (and we as well) use 2nd approach.
And my question, do market interested in publicly available such methodology and tools as standardised and verified? If yes, who should pay for it?
These problems are not limited to developing countries, because nearly even change-oriented organization that gets funding wants to spend as much of the money as possible on their core goals. From this point of view, any money that is spent on evaluation is money that wasn't used for the purposes they proposed.
One way to work with this situation is through "community based participatory research" where the evaluators do as much as possible to engage with the whole range of stakeholders, and not just the program sponsors. It can cost as much as "professionally" based evaluations, but much of the money goes to hiring local community members, which is likely to be valued by the sponsors. However, it can be time consuming.
I do not understand why an innovation would proceed without a logic model, but you report there is no logic model, so start the evaluation by building one. Get key actors to agree the logic model is acceptable. Then discover whether events unfolded per the logic model.
This only tells you about process. Unless you have strong reasons for believing the logic model, even its successful implementation tells you nothing bout impact. Can you, possibly, obtain outcome measures from non-equivalent comparison groups?
William Rhodes , thank you for answer. You are 100% right. This is what I am doing actually. Be proud that I do goo job, I am at the same time feel looser because nobody pay for it. As result - my team now one of the best and respected on building logic models in the region ;).
But anyhow this is not my question, actually.
Do you think, that open universal methodology for evaluations with small budget (up tp 3000 USD) can help? Do you know university/professor who can interested in such work?
David L Morgan , thank you for comment!
"any money that is spent on evaluation is money that wasn't used for the purposes they proposed" - for output-oriented organization/project it is right. Money for management never goes to beneficiary. But for Impact-oriented/policy-level project evaluation and other methods of independent prove of concept, verification of success, conceptualization of best practice - is actually one of obligatory outputs.
"community based participatory research" is very good idea, but for environments where communities exists and strong enough. In countries, where disruption of communities was a governmental strategy for long period of time (including CIS region), community-based evaluation and public oversight working very bad.
Anyway, it good to think in this direction during project design.
But when your junior evaluation applying to conduct evaluation for 2-3k$, this approach not helps to get answer to my question. In your opinion, do you think, that open universal methodology for evaluations with small budget (up tp 3000 USD) can help? Do you know university/professor who can interested in such work?
Haji Saediman , yes, UNDP and other international organizations has such policy. Quality and impact of it - topic for long discussion. But my question is not about the system - I know it very well and in some cases has been involved in design of such system.
My question is different actually.
- Badges
- Science topic