What next on assessing systemic change?

How we can better measure systemic change.

The aim of achieving systemic change is an inherent feature of market systems development programmes. However, it can be a challenge to show that market systems programmes and projects can reach a deeper level of change than other programmes that directly deliver goods and services to the poor. Consequently, measuring systemic change is a popular topic. Over 140 people participated in a recent webinar co-hosted by the BEAM Exchange on assessing systemic change. The popularity of the event shows that there are many unanswered questions on how to actually measure systemic change. 

What is systemic change?

Systemic change can be defined as alterations in the structures or dynamics of a market system that lead to new patterns of behaviour of the actors in the system. For example, instead of looking at how many units of a particular product are sold, we need to look at relationships between sellers and buyers, or changes in the understanding of the value of quality within the market actors. What systemic change looks like needs to be defined individually for each project. The ultimate aim of working towards systemic change is to achieve sustainable poverty reduction at scale. (Read more about systemic change here).

Three elements to measure systemic change

During the webinar, the speakers each presented an element that needs to be in place within a project to be able to measure systemic change.

First, there needs to be a clear explanation of how systemic change is defined in a project. According to Jim Tanburn (DCED), the definition should match the intention of the project: to tackle the root causes of poverty. The definition must also contain the three aspects of scale, sustainability, and resilience.

The second element is a pathway describing how we imagine the project interventions lead to systemic change. Matt Ripley (the Lab) used an example of an ILO project in the horticulture sector in Timor-Leste to describe one possible pathway. The pathway began when the project partnered with a company to pilot ways to overcome problems faced by smallholders. Change pathways can be described using results chains or other causal models. Matt made it clear that the trick is not to limit the results chain to the pilot intervention. The chain should include points from which wider systemic change is triggered; for example, if the piloted innovation proves to be profitable for market players involved in the pilot. From there, the results chain needs to continue to cover wider ripples of change that can be further supported by the project. 

The third element is a conceptual understanding of systemic change. Without a framework that helps us conceptualise systemic change we do not know where to look for it. The results chains show how we intend to achieve change. But as Matt pointed out, reality is messier than what can be reflected in results chains. We need to keep our eyes open for changes outside of the anticipated pathway. 

Tim Tuckett (SAMARTH) introduced the conceptual model his programme uses to look for systemic change, the ‘Adapt, Adopt, Expand, Respond’ (AAER) framework. The SAMARTH team first observes the market actors that are directly associated with the programme. They look for signs of these actors adopting or adapting the innovation introduced through the programme. They then look at changes in the behaviour of market actors not directly associated with the programme to find out if the use of the innovation is expanding into new businesses and areas. Eventually, the programme is interested in finding out whether the overall market responds to the pro-poor innovation, particularly through looking at non-competing market players.

What next for systemic change?

The three elements represent an example of how projects currently tackle the challenge of assessing systemic change caused by their interventions. Some of the tools, such as the AAER framework, are relatively new and not used by many projects. Experiences on how to operationalise it still need to be collected and shared. Results chains, in turn, have important limitations when targeting systemic change, such as their inability to reflect unanticipated consequences of interventions (read more about their limitations here).

Based on the questions we received before and during the webinar, it is clear that all three elements outlined above need more clarification and discussion. Practitioners want to learn: 

• how to deal with the long-term nature of systemic change when implementing short-term projects
• how to attribute observed (systemic) change to a project’s interventions
• how to use the AAER framework in practice
• definitions of sustainability, resilience and how they are dependent
• the development of pathways to systemic change and how to incorporate the dynamism of market systems and emergence of unanticipated changes – things that happen 'by accident' or 'along the way'
• the relationship between systemic change and scale.

Can you help?

Rather than let these questions go unanswered we have invited the speakers of the webinar to share their thoughts and ideas on the BEAM Exchange LinkedIn group. We are also planning to host more webinars on the topic showcasing innovative projects and introducing more tools and guidance. We will tap into the collected wisdom of our community for innovative cases and valuable experiences, and gather more questions. If you have any thoughts on the above questions please comment below or join the discussion on BEAM’s LinkedIn group. 

Thank you to all who participated in the webinar. I look forward to more fruitful exchanges on measuring systemic change in the future. 

• Read more about systemic change and watch the webinar recording
   

Marcus Jenal  is the BEAM Exchange lead on results measurement and impact evaluation.