Evaluations not only support learning, but are essential to establish credible evidence to show that market systems development can make a difference.
We evaluate programmes using market systems approaches to understand how and under what circumstances the approach works or doesn't. Evaluation not only supports learning, it is essential to establish credible evidence to show that market systems development can make a difference. This is critical to validate current and justify future donor investments in the approach.
However, not all programmes applying a market systems development approach are evaluated effectively. One reason is that the aims and characteristics of market systems approaches create particular challenges for evaluation. These include, for example, the complex nature of market systems which means system-level change is unpredictable, and the need for adaptability in intervention. As we become more aware of the characteristics of market systems and gain more experience, the challenges become better understood and have been documented in various publications.
There is, however, growing recognition among evaluation professionals and donors who mandate evaluations that there is no single one-size-fits-all method and that evaluation designs need to be adapted to the programme that is being evaluated. For example, to be ready to react to adaptations in the programme plan, the Ghana MADE evaluation team planned close-out assessments for activities the programme pulls out of. This might not be feasible for all evaluations.
Besides adapting to the specific programme, good evaluation design should be clear about its purpose and scope. Is the purpose of the evaluation accountability towards the donor or to contribute to learning and further development of the particular programme? Or to contribute to the body of evidence about the effectiveness of the approach? What should be evaluated and where? Only when the purpose and scope are clear can the appropriate methodology be developed.
Variety of theory-based methodologies
Theory-based approaches for evaluating economic development initiatives have become the default mechanism for evaluating market systems programmes. Theory-based evaluations start with a programme theory or theory of change, and then search for evidence to support the theory.
Programme theory provides a framework for the evaluation design, but this approach does not prescribe particular methodologies or tools – on the contrary, evaluating the programme theory is at the centre of the evaluation. This is in contrast to methods-driven evaluations of the past that put a particular method - like experimental designs - at the centre. Therefore, using a theory-based evaluation framework does not automatically solve all the challenges in evaluating complex and adaptive programmes. There is still considerable room to choose from different methodologies and use a variety of tools when using a theory-based approach. Examples for theory-based methodologies that are currently used are contribution analysis or realist evaluation methods.
BEAM Exchange is working with a number of ongoing programme evaluations to capture their methodologies and lessons learned so that we can share them with you and the wider community. We hope that the evaluation synopses can inspire future evaluation design and the lessons within can inform evaluators who are venturing into evaluating market systems development programmes.
The first evaluation synopses include the DFID-funded programmes Ghana MADE and PrOpCom Mai-karfi and the multi-donor funded Kenya Market Assistance Programme (MAP). The synopses provide an overview of the programme, the evaluation methodology, the set-up of the evaluation team, and how the evaluation collaborates with the programme management and monitoring team. They also capture the lessons emerging from these generally young programmes. One important learning is that collaboration and coordination between the programme monitoring team and the evaluators can meaningfully support and simplify the evaluation effort.
The collation and sharing of the synopses is the beginning of BEAM's work on evaluation. The synopses will inform an extended set of guidelines on the key steps and considerations for planning and undertaking an evaluation of programmes that use market systems development approaches. We are also working on an introduction to relevant evaluation methods and tools, and a guide on how to decide which to apply. Please contact us to request a draft version. We would happily share earlier versions so you can test them in the real world and provide us with your feedback – this will help us make the guide more relevant.
Marcus Jenal is BEAM's Monitoring and Evaluation Lead.
BEAM Programme Index: