Module 5: Reviewing and integrating results

04

Attributing results to programme interventions

The questions about what the programme and its interventions are achieving assess the extent to which observed changes are the result of programme activities.

This assessment has traditionally been the preserve of evaluation, rather than monitoring. Nonetheless there will be circumstances during the course of the programme in which it is important, for monitoring purposes, to make a rigorous assessment of what an intervention is achieving. These might include:

  • Where a decision to scale-up, modify, or cancel an intervention needs to be made
  • Where robust evidence that an innovation piloted by the programme works is needed to persuade other market actors to adopt it

The question of how to evaluate market systems programmes is discussed in detail in the forthcoming BEAM Exchange Evaluation guidance. This section instead provides a summary review of the issue of attribution for the purposes of monitoring.

Approaches to establishing causality

Assessing what an intervention has achieved hinges on the question of causality. The two main classes of approaches to establishing causality for market systems programmes are as follows:

  • Approaches that allow impacts to be attributed to the intervention. 
    These compare the effects of an intervention to a counterfactual situation, or what would have otherwise happened. Where appropriate methods are used carefully, these approaches make it possible to make causal claims about the intervention as the cause of the impact, and to measure how much of the impact can be linked to the intervention.
  • Approaches that show whether and how an intervention has contributed to observed impacts, along with other factors. These approaches set out to make a plausible argument for causality, identifying outcomes and then tracing the mechanisms through which interventions may have influenced them, while paying careful attention to the context. 

Using a counterfactual

Experimental and quasi-experimental methods compare the results for a treatment group of beneficiaries, who participate in an intervention, and a control group who do not. The approaches work on the assumption that in all other respects, the treatment and control groups have the same characteristics, and that differences in outcomes between them can therefore be attributed to the intervention. 

Randomised Controlled Trials (RCTs) are an example of an experimental research design, in which beneficiaries are randomly assigned to either the treatment or control group. However, while RCTs have been used successfully in some market systems programmes, the characteristics of these programmes means that it is often difficult or impossible to select a sample in a truly random manner. For instance, in an intervention which pilots a new product or service with the intention that other market actors are then encouraged to copy, those who benefit from the intervention (the treatment group) will not have been selected randomly, but will have selected themselves. 

Randomized controlled trial (RCT)

BetterEvaluation's basic steps on implementing an RCT.

In quasi-experimental methods, the two groups are not selected randomly, but by making use of some other characteristic in order to control for observed differences. While this approach may be more appropriate than a RCT, the risk of self-selection and other biases are still present.

Quasi-experimental methods for impact evaluations

Use quasi experimental methods to get around the challenge of creating a valid counterfactual.

Quasi-experimental design and methods

UNICEF brief on when a quasi-experimental approach is appropriate, and some of its limitations.

So-called non-experimental approaches are based on comparing the results of an intervention with a counterfactual, but without the use of a control group. These approaches use hypothetical predictions about what would have happened in the absence of the intervention to establish the counterfactual.

Systems dynamics modelling in industrial development evaluation

The use and benefits of systems dynamics modelling for evaluation.

Challenges with counterfactual-based designs

Market systems development programmes face a number of challenges when designing studies to establish causality using counterfactuals:

  • Difficulties in establishing a counterfactual. The aim of diffusing innovations throughout the system can make it difficult to establish a counterfactual with people that have not been affected in one way or another by the programme
  • Self-selection bias. As discussed above, the aim of programmes is often to encourage forward-looking market actors to adopt a piloted innovation. Their willingness to do so makes them qualitatively different from other actors, so comparisons between treatment and control groups may not be meaningful  
  • Simultaneous interventions carried out in parallel. Market systems programmes often implement parallel interventions at different levels, such that it is hard to disentangle the effects of each. (For instance trying to influence policy at a national macro level, while also intervening at the micro level in a particular sector)
  • The need for adaptive implementation hinders the establishment of baselines or control groups, for instance where the geographical focus of an intervention changes.

Evaluating systems and systemic change

Guidance on developing an evaluation framework and empirical approaches for identifying and monitoring systemic changes.

Establishing causality without a counterfactual

Rigorous approaches to establishing causality are still possible where constructing a robust counterfactual is not possible. 

Several methods exist which embody a similar underlying philosophy and are typically used in 'theory-based evaluation'. These examine the causal chain of events (as set out in the theory of change) that connect an intervention with observed outcomes. Evidence is then gathered and reviewed to assess whether the causal mechanisms specified in the theory of change are plausibly responsible for the result, or whether competing hypotheses provide a better explanation. The concept of contribution, rather than attribution, is at the heart of what the claims these approaches make about causality. Examples include outcome harvesting and contribution analysis.

Such approaches run the risk of bias, that is, a systematic tendency to over or under estimate the importance of the causal relation. There are several sources of bias, including respondents who give a particular account for reasons of self-interest, or to please the investigator, as well as biases among those conducting the evaluation. The potential for bias can never be ruled out, but it can be minimised by taking a rigorous approach that specifies in advance what issues will be considered, the nature and sources of data, research protocols etc.

Addressing attribution of cause and effect in small n impact evaluations

The various approaches that are available for small n impact evaluations.

Outcome harvesting

Outcome harvesting is an approach which collects evidence of what an intervention has achieved in a given period. It starts by identifying a range of outcomes associated with an intervention. Both qualitative and quantitative evidence is then collected in order to consider how far the intervention contributed to the observed change. Outcome harvesting is particularly appropriate for an 'outwards-in' approach that starts from observed changes in the market system and attempts to trace these back to programme interventions.

Outcome harvesting

Understand Outcome Harvesting and how it has been applied in real situations

Contribution analysis

Contribution analysis also looks at the theory of change, and aims to build up evidence that demonstrates the contribution made by an intervention 'beyond reasonable doubt', while also establishing the relative importance of other influences. The approach draws on the idea that an intervention's theory of change can be used to infer causation by assessing whether the processes that it aims to initiate have in fact occurred. In contrast to outcome harvesting, contribution analysis takes the intervention as the point of departure, and then works its way 'outwards'. 

Contribution analysis: An approach to exploring cause and effect

Key steps in the process of contribution analysis.

Developing a narrative of transformation

A similar approach to establishing causal links between an intervention and observed effects is to undertake a detailed analysis and explanation of how the intervention has worked with market actors and within the wider context. A rich narrative can then be constructed to describe transformations in market systems. Such narratives communicate the programme team’s understanding of the role of a programme in such transformations. 

Timelines, critical incidents and systems: a nice way to understand programs

Reflections on interpreting critical incidents to understand programme performance.

Go to Module 6 next or return to the Monitoring Guidance.

Read practical stories on the success and failures of attributing results to programme interventions from GEMS4, Katalyst and STARS.