Politics of evidence: can I get a green card?

Do we have enough evidence to know if market systems approaches work? What kind of evidence is 'good enough' and who decides? Read what a lively debate on the 'politics of evidence' reveals about evaluation approaches.

We had a lively debate on the politics of evidence at the recent BEAM Conference in Lusaka, where almost 200 senior-level representatives from donor and implementing organisations gathered to discuss systems approaches for building inclusive markets and reducing poverty. While implementation issues received plenty of attention during the conference, what was really striking (and surprising) to me was the deep and almost universal interest in evaluation.  

Armed with voting cards in green, orange and red, conference participants expressed strong opinions around some pretty important questions:

• Do we have enough evidence to know if market systems approaches work?
• What kind of evidence is 'good enough' and who decides?
• Does anyone use (or even look at) the evidence that already exists?
• If calls for 'more evidence' are legitimate, could we approach the task in a better way?

The evaluation buzz reached a peak on the second day at the plenary session on market systems and the politics of evidence. It was organised as a panel discussion, with Simon Stevens (DFID) providing a donor perspective and Matt Ripley (ILO) representing implementers. I was there to speak to the issues as an evaluator, social scientist and lead investigator for evaluation issues with USAID’s Leveraging Economic Opportunity (LEO) activity.  

The session opened with a brief presentation on the BEAM Evidence Map. So far, the map includes 57 empirical resources (case studies and reports) organised by type of intervention and type of result. Just as you might expect, there is a lot of variation in terms of evaluation methods used and the ways results are measured. (By the way, BEAM is still seeking additional contributions.)

From where I sit, the 'evidence' discussion tends to hide more than it reveals. If we’re serious about learning, we need to work together to address some fundamental questions:

1. What do we evaluate? 
(Evaluators call this the 'evaluand'.) It’s ludicrous to lump together the entire range of market systems interventions (as I did in the first bullet above). After all, why would we expect every type of systems approach to succeed (or fail) equally? At the same time, it isn’t very helpful to treat each intervention as unique. Instead, we need to delineate and categorise the most common types of interventions so we can begin to build a coherent body of evidence on each type.

2. What do we mean when we say that an intervention 'works' or 'doesn’t work?' 
Having compiled results on value chain and market systems programmes (see here and here and here), I can confidently say there are lots of apples and oranges (and bananas and strawberries) out there! Some evaluations report on the number of people reached but use different definitions of outreach. Others measure (dissimilar) outcomes, while only a handful look at inclusion, systemic change or sustainability. We may not agree on what success looks like, but could we at least agree to measure a common range of possibilities?

3. How do we select evaluation methods? 
This has generated a rancorous debate for at least 15 years (and I have something to say about 'rigour' below). For the moment, let me point out that we currently lack methods for evaluating sustainability and systemic change, even though we consider them to be integral results for market systems approaches. We should know more soon about evaluating systemic change, since a number of organisations are working on this (including LEO and BEAM), but I am not sure if anyone is developing practical approaches for evaluating sustainability.

4. How do we interpret, communicate and use evaluation findings? 
As all seasoned evaluators know, almost any finding can be communicated with a positive or negative spin. Politics play a major role in the interpretation, communication and use of evaluation findings. Following up on comments he posted a week or two before the conference, Matt Ripley expressed practitioners’ frustrations with the distorted incentive structure surrounding the generation and use of evidence.

Judging from the assortment of green, orange and red cards, it may not be possible for the community to agree on these issues. Perhaps, as Ben Taylor later proposed, we should examine this as a system of evidence and work to remove constraints to the supply, demand and exchange of evidence.

Let me suggest that we start by approaching evaluation issues with greater humility and appreciation for alternative points of view. Let’s move beyond endless arguments over methods and come to understand that rigour is about transparency and a sober understanding of the limitations inherent in every method. Let’s agree that a rigorous study is one in which the strengths and weaknesses of the conclusions are clearly communicated. Let’s move beyond sound bites and become more deliberate about learning.

This blog was originally posted on Microlinks, and is republished here with kind permission.