Dear Dr. Seema Mahato ,

This link is very useful for you about CA:

https://www.betterevaluation.org/en/plan/approach/contribution_analysis

Hi,

I have not done any work in the education assessment field, so can't really help on that front.

A few CA articles i have notice recently:

Downes, A., E. Novicki and J. Howard1 (2018). "Using the Contribution Analysis Approach to Evaluate Science Impact: A Case Study of the National Institute for Occupational Safety and Health." American Journal of Evaluation.

Ton, G. (2017). "Contribution analysis of a Bolivian innovation grant fund: mixing methods to verify relevance, efficiency and effectiveness." Journal of Development Effectiveness 9(1): 120-143. Available at http://dx.doi.org/10.1080/19439342.2016.1231702

Buregeya, J. M., A. Brousselle, K. Nour and C. Loignon (2017). "Comment évaluer les e ets des évaluations d’impact sur la santé : le potentiel de l’analyse de contribution." Canadian Journal of Program Evaluation 32(1): 25-45.

Budhwani, S. and J. C. McDavid (2017). "Contribution Analysis: Theoretical and Practical Challenges and Prospects for Evaluators." Canadian Journal of Program Evaluation 32(1): 1-24.

Buckley, A. P. (2016). "Using Contribution Analysis to evaluate small & medium enterprise support policy." Evaluation 22(2): 129-148.

Stocks-Rankin, C.-R., H. Wilkinson, C. Lightowler and S. Morton (2014). Summary of PROP Contribution!Analysis. Available at http://blogs.iriss.org.uk/prop/files/2013/09/Summary-of-Impact-PROP-Contribution-Stories-September-2013-updated-January-2014.pdf.

Contribution Analysis

Also Available In: Portugues

Contribution Analysis is an approach for assessing causal questions and inferring causality in real-life program evaluations. It offers a step-by-step approach designed to help managers, researchers, and policymakers arrive at conclusions about the contribution their program has made (or is currently making) to particular outcomes. The essential value of contribution analysis is that it offers an approach designed to reduce uncertainty about the contribution the intervention is making to the observed results through an increased understanding of why the observed results have occurred (or not!) and the roles played by the intervention and other internal and external factors.

Contribution analysis is particularly useful in situations where the programme is not experimental, i.e. not in trial projects but in situations where the programme has been funded on the basis of a relatively clearly articulated theory of changeand where there is little or no scope for varying how the program is implemented. Contribution analysis helps to confirm or revise a theory of change; it is not intended to be used to surface or uncover and display a hitherto implicit or inexplicit theory of change. The report from a contribution analysis is not definitive proof, but rather provides evidence and a line of reasoning from which we can draw a plausible conclusion that, within some level of confidence, the program has made an important contribution to the documented results.

Steps​

Six steps are taken to produce a credible contribution story:

1: Set out the attribution problem to be addressed

Determine the specific questions being addressed.  Not all cause-effect questions are useful to pursue. Contribution analysis is less suitable for traditional causality questions such as: Has the program caused the outcome? To what extent, quantitatively, has the program caused the outcome?  These often are not that useful because they treat the program as a black box and don’t get at the fact that there are usually many causes involved. Contribution analysis is more appropriate for contribution questions: Has the Program influenced the observed result? Has the program made an important contribution to the observed result? Why has the result occurred? What role did the intervention play?  and for management questions:  Is it reasonable to conclude that the program has made a difference? What does the preponderance of evidence say about how well the program is making a difference? What conditions are needed to make this type of program succeed?

2: Develop a theory of change and risks to it

Develop the program logic/results chain describing how the program is supposed to work. Identify as well the main external factors at play that might account for the outcomes observed. Based on the results chain, develop the theory of change upon which the program is based. This theory of change should lead to a plausible association between the activities of the program and the outcomes sought. The theory of change must include the assumptions made in the results chain and the inherent risks as well as external influences such as donor pressure, influences of peers and resourcing levels. Some links in the theory of change will be fairly well understood or accepted. Others will be less well understood, contested or subject to significant influence other than from the program. In this way you acknowledge that attribution is indeed a problem

3: Gather the existing evidence on the theory of change

It is useful to first use existing evidence such as from past related evaluations or research, and from prior monitoring, to test the theory of change. It sets out the intended results (outputs, intermediate and end outcomes). What evidence (information from performance measures and evaluations) is currently available about the occurrence of these various results? The links in the theory of change also need to be assessed. What evidence currently exists on the assumptions and risks behind these links? Which are strong (good evidence available, strong logic, or wide acceptance) and which are weak (little evidence available, weak logic, or little agreement among stakeholders)? What evidence exists about the identified other influencing factors and the contribution they may be making?

4: Assemble and assess the contribution story, or performance story, and challenges to it

With this information, you will be able to assemble your contribution story that expresses why it is reasonable to assume that the actions of the program have contributed (in some fashion, which you may want to try and characterize) to the observed outcomes. Now you have to assess it. How credible is the story? Do reasonable people agree with the story? Does the pattern of results observed validate the results chain? Where are the main weaknesses in the story? There always will be weaknesses. Weaknesses in the story point to where additional data or information is needed.

5: Seek out additional evidence

Having identified where the contribution story is less credible, additional evidence is now gathered to augment the evidence in terms of what results have occurred, how reasonable the key assumptions are, and what has been the role of external influences and other contributing factors.  Augmenting evidence can include the collection of additional, new data such as from surveys, field visits, administrative data, focus groups, national statistical data, etc. as well as the synthesis of evidence from other research and evaluations.

6: Revise and, where the additional evidence permits, strengthen the contribution story

With the new evidence, you should be able to build a more substantive and so more credible story, one that a reasonable person will be more likely to agree with. It will probably not be foolproof, but the additional evidence will have made it stronger and more plausible.

Using a generative perspective on causality to infer that a program made an important contribution to an expected result that has been observed, contribution analysis argues that a reasonable contribution causal claim can be made if:

• There is a reasoned theory of change for the intervention: the key assumptions behind why the intervention is expected to work make sense, are plausible, may be supported by evidence and/or existing research, and are agreed upon by at least some of the key players.
• The activities of the intervention were implemented as set out in the theory of change.
• The theory of change—or key elements thereof— is supported by and confirmed by evidence on observed results and underlying assumptions—the chain of expected results occurred. The theory of change has not been disproved.
• Other influencing factors have been assessed and either shown not to have made a significant contribution or their relative role in contributing to the desired result has been recognized.

Issues

Some issues might arise when taking this approach with regards to:

• Reducing uncertainty about the contribution the intervention is making to the observed results.
• Inferring causality in real-life program evaluations.
• Confirming or revising a programme’s theory of change – including its logic model.

Risks and assumptions are labelled as [O] over which the intervention has no or very little influence, or [I], where the intervention can (should) have an influence, direct or indirect, or [C] where the intervention should be able to directly control.

Resources

Guides

• Addressing attribution through contribution analysis: using performance measures sensibly. This article from the Canadian Journal of Program Evaluation outlines the key steps involved with using contribution analysis as a tool to discover the contribution a program has made to particular outcomes.
• Contribution Analysis: An approach to exploring cause and effect: This brief from the Institutional Learning and Change Initiative(ILAC) explores contribution analysis and how it can be used to provide credible assessments of cause and effect.
• Mayne, J. (2011). Addressing Cause and Effect in Simple and Complex Settings through Contribution Analysis. In Evaluating the Complex, R. Schwartz, K. Forss, and M. Marra (Eds.), Transaction Publishers.
• Mayne, J. (2011) Contribution Analysis: Addressing Cause and Effect in Evaluating the Complex, K. Forss, M. Marra and R. Schwartz (Eds.), Transaction Publishers; Piscataway, New Jersey.

Source

• Mayne, J. (2008) Contribution Analysis: An approach to exploring cause and effect, ILAC methodological brief, available at http://www.cgiar-ilac.org/files/ILAC_Brief16_Contribution_Analysis_0.pdf

Thank you Dr. Mayne and Dr. Turky!

I appreciate your promptness at responding to my question and sharing invaluable information.

Regards,

Seema

Hi Seema

how are you?

I plan to use Contribution Analysis to evaluate a programmatic approach to assessment within the work-integrated learning setting. If you ar doing work in this area it would be great to keep in contact.

Kind regards,

Janica Jamieson

Dear Janica,

Thank you for reaching out! I am doing well. How are you?

Yes, one of my evaluation projects is in this area, and it would be my pleasure to exchange insights and ideas with you.

Best,

Seema

And I of course would be interested in hearing how your use of CA went!

Absolutely Dr. Mayne!

It would be an honor to share my experiences using CA, in a higher education setting, with you. Your publications on CA, especially the 2011 article on "Addressing Cause and Effect" is extremely insightful and will be a great guide for my work.

Best regards,

Seema