A pragmatic approach to measuring system change

We've developed A pragmatic approach to assessing system change that programmes can apply themselves

In recent years, growing recognition of the need for ‘systemic’ approaches in development programming has put pressure on implementers to measure ‘system changes.’

This is important - we need to capture system changes in order to make smart intervention decisions, as well as to be accountable for the impact of our work. But it has become clear to me - over drinks, in personal emails, on WhatsApp and during training - that this growing pressure is also stressful for practitioners on the ground. Concepts of system change in development are maturing, but measurement practices in the field have struggled to keep pace.

Thankfully, this is changing. There is a growing consensus on good practice for capturing system changes. Our recent work - developed and funded by a cross-field collaboration and built on hard-won programme experience - presents this emerging consensus in A Pragmatic Approach to assessing system change that programmes can apply themselves. This comprises an Overview supported by a How-to guide with tips, templates and plentiful examples.

Even better, the principles and good practices that underpin the suggested approach are relatively straightforward, especially in comparison to some of the debates about system change! Here are some of the practices we recommend:

1. Identify a system: if you want to talk about whether a system has changed, you must first be clear what ‘system’ you’re talking about. Identifying the system includes identifying the most important interconnected systems too
2. Work out your starting state: you can’t assess ‘change’ without knowing how things are working in the first place.
3. What’s your vision, what’s your plan? By explicitly laying out your desired state for the system and a plan for system change, you build a foundation to later assess against. That assessment, in turn, provides critical information for revising the vision and plan.
4. Track the intervention: to work out whether or not a change is becoming embedded in the system, you’ll need to track its sustainability and scale. The Pragmatic Approach suggests these guiding questions:

In a development context, tracking the intervention also involves evaluating development outcomes. After all, programmes don’t just want to know whether a system has changed - we also want to know whether that change had a positive impact.

5. But don’t just track the intervention: perhaps the most common mistake people make when assessing system change is focusing solely on the changes introduced by an intervention. The Pragmatic Approach reminds us that it’s important to zoom out and assess changes in the system as a whole too. It calls this the ‘helicopter lens’ and suggests the following questions as a starting point:

The helicopter lens involves revisiting the questions asked when assessing the starting state and seeing if you can work out how, why, and how much key indicators have changed.

It’s important to do this because a single intervention is never the only thing changing a system. It’s likely that other interventions, as well as things completely unrelated to the programme, are also changing the system. Sometimes the combined effect of these changes even leads to detrimental effects. This kind of information is important to feed into the system strategy and future intervention plans.

6. Check in regularly: systems are always changing (whether in ways attributable to a development intervention or not), and it is difficult to predict exactly how and when change will happen. System change assessment, therefore, can’t be reduced to a one-off activity. Instead, check in regularly - annually is about right for many programmes - to capture system changes for reporting, and to gather the information needed for adaptive management.

7. Use the information to adapt and improve: many programmes start assessing system change because they need to report on the results they are achieving. But capturing system change isn’t just about reporting; it also gives programmes the information they need to adapt and improve their work.

These practices aren’t complicated, but they are effective. Are they enough? If you follow these practices, will you really, truly have captured system changes?

The question will continue to be asked, but for those who don’t want to engage in the philosophical debates, this approach provides a way to put what has been learned in the field so far into practice. If it’s good enough for making implementation decisions and good enough for reporting - and experience from multiple programmes and donors suggests that it is - then, well, it’s good enough for me.

The above ideas - and many more - are described in A Pragmatic Approach to Assessing System Change. Our Overview paper provides a short, readable introduction to the approach, showing how assessing system changes can be practical and valuable. Our How-to guide includes many additional tips, templates, and worked examples to help implement the approach in practice, ensuring that you will be able to measure, report, and use information about system changes.