How well does the MSD approach work? BEAM’s 2019 Evidence Review

Last month we published the 2019 BEAM Evidence Review – our third such report.

It is part of a sustained effort to help decision-makers assess the MSD approach by gathering and collating documents that report the outcomes and impact of programmes on businesses, incomes and the livelihoods of people living in poverty.

The review draws on our growing database - the BEAM Evidence Map – which now contains 152 evidence documents that meet our independently assessed inclusion criteria. Although over two-thirds of documents relate to initiatives in the agricultural sector, there are increasingly results reported from other fields such as energy services, health services, financial services and tourism.

The 2019 Review shows that many programmes using the MSD approach achieve significant results. We classify documents according to the principle level of results they describe. And evidence of systemic change – meaning markets work more efficiently and inclusively - is the main story in fifty documents. Evidence of how such change is leading to poverty reduction predominates in a further forty documents.

The review explored some selected examples of MSD intervention initiatives from a few different programmes in detail in order to illustrate how the approach can deliver results with both scale and sustainability.

• In Malawi the MOST programme worked with government agencies and private input-supply businesses to improve access to oil-seed inoculants. The initiative benefited over 38,000 farmers in 2017 (with annual benefits valued at US $3.7m) and was projected to reach over 65,000 farmers in 2018.
• In Indonesia the PRISMA programme worked with animal feed firms to promote better pig-rearing practices. The initiative improved the incomes of almost 50,000 farmers in 2017 (with annual benefits valued at US $26m) and impact was projected to double in 2018.
• In Pakistan the MDF programme worked with silage producers and livestock farmers on year-round access to fodder. The initiative reached 11,000 smallholders by 2017 (with annual benefits valued at US $15m) and the benefits are projected to reach 100,000 farmers by 2020.

These examples illustrate the potential for MSD programmes to achieve systemic change that leads to significant growth and poverty reduction. Most importantly, the relevant studies predicted continued increases in outreach and income from these successful initiatives. These ‘projections’ are credible when we consider the major long-run impact of past initiatives such as Katalyst’s work in the Bangladeshi vegetable sector, ALCP’s work in the Caucasian livestock sector or FSD Kenya’s work to create the M-Shwari digital savings service in Kenya.  Credibility is reinforced by the fact that both PRISMA and MDF, like ALCP and Katalyst, were audited as achieving the DCED Standard for Results Measurement. 

Of course, the scale of success in these cases is not representative of all initiatives undertaken by MSD programmes. Indeed, the entrepreneurial nature of the MSD approach presumes that many, perhaps even most, initiatives fail. In well-managed programmes (i.e. that are responsive, adaptive and effective at learning) the costs of failures should be modest. So the measure of a successful MSD programme is whether the aggregate benefits of a few large-scale successes outweigh the costs of the more numerous small failures.

This leads me to two important additional caveats with this year’s review:

• First, we still have too few ex-post impact evaluations which look at the overall performance of MSD programmes including successes and failures. Only one extra study of this nature was added to the Evidence Map during 2018.
• Second, we clearly still have a problem of publication bias: positive stories are being told, but failures are generally buried.

The good news is that several new impact evaluations of major MSD programmes are in the pipeline during 2019/20 thanks to some big investments in real-time evaluation and evaluation units dedicated to individual programmes. Some of these should be available for next year’s review.

Inevitably even these evaluations will not satisfy all observers. MSD programming simply does not lend itself to the most rigorous forms of ‘controlled’ evaluation. This is because MSD interventions are not uniform treatments that can be randomised. Our beneficiaries (i.e. the consumers, producers, businesses engaged in markets) are self-selected; and it is usually impractical to create experimentally-valid ‘control groups’ or prevent their cross-contamination. Finally, interventions necessarily evolve over time in response to changing knowledge and market dynamics.

In these regards the MSD approach is similar to other areas of development cooperation and aid that deal with dynamic problems in complex environments with multiple actors and functions. It has clear parallels with systemic and politically informed approaches, such as Problem Driven Iterative Adaptation (PDIA), that are emerging in fields including state building and social development in response to disappointing results of conventional programming. And we know that adaptive approaches (MSD included) throw up well-recognised evidence challenges. This is one reason the joint DFID and USAID-funded GLAM initiative was created.

Clearly, we who advocate for the MSD approach need to pay attention to the calls for ‘evidence’ and continue our efforts to respond to the legitimate needs of decision-makers and politicians who have to justify public expenditure. In my view, this is a communication challenge (celebrating the big success stories) just as much as an evidence challenge.

We need to remember that it may take longer to achieve measurable impact when the aim is for enduring system change. So we also need to follow through with ex-post evaluations, three or five years after programmes finish. And we have to be champions of a development cooperation culture that seeks to shift the conditions that hold problems in place, not just ameliorate their symptoms.

View the webinar (Oct. 2019): Reviewing the evidence - how effective is the MSD approach?