Seven tools for measuring systemic change

Read about the tools USAID's LEO project is using to capture systemic change. 

How can we effectively understand and measure systemic changes in the market systems we engage in? That is a question that I have been focused on for some time. Today in my role with the USAID-funded Leveraging Economic Opportunities (LEO) project I am pleased to share a concrete effort to investigate this topic further.

Background to our interest in measuring systemic change

Since late 2013, LEO has focused on the issue of measuring systemic change. Out of this interest came a field-based investigation of methods being applied in Uganda and a literature review and synthesis of efforts to evaluate systemic change. It became clear that more needs to be learned on the best ways to observe and capture this change. In particular, what are the most practical methods and tools? This is a strong priority of USAID missions who are seeking ways to not only capture systemic change once it has occurred, but to capture early signs of change that would suggest the system is moving in a positive direction. Capturing systemic change is also a focus for the BEAM Exchange community, as the first BEAM webinar and other blogs on the topic show.

Our agenda for investigation and our partners

With that goal, LEO and its partners have set an agenda focused on three key areas:

- Selecting and testing methods and tools for capturing systemic change
- Identifying and classifying indicators of systemic change
- Synthesizing approaches for adaptive management.

To accomplish this ambitious agenda we have joined with several partners, including the BEAM Exchange, the DCED, MaFI and the SEEP Network. We are equally collaborating with the complexity-aware monitoring trials that are being conducted by USAID's Learning Lab.

Understanding methods and tools for measuring systemic change

Based on the key areas listed above, we started our focus on selecting and testing methods and tools to measure systemic change. By starting here we are not suggesting tools alone are all you need. Tools complement other important needs, including the ability of projects to effectively conceptualise what they assume systemic change looks like in their context. They also need the capacity to effectively use information that is collected to inform decision-making. Yet tools and methods is an area where we identified there is a lot to learn.

To launch this investigation, we began by identifying a long list of methods, tools, theories and frameworks with potential application. In reviewing them, we found that these collected resources were intended for a number of different purposes. Some frameworks (e.g. the Adapt-Adopt-Expand-Respond Framework) were most helpful for identifying potential systemic changes rather than measuring them. Others were quite interesting but would either require extensive adaptations to be relevant to market systems facilitation or were focused on a narrow subset of systemic change (e.g. governance reform). This process of elimination led us to a shortlist of seven tools. These include, in no particular order:

• Most significant change
• Outcome mapping
• Outcome harvesting
• SenseMaker
• Social network analysis
• Systemic action research / participatory systemic inquiry
• Standard measurement tools to capture key indicators

The first six each have relatively structured guidance for their application. However, we created a seventh ‘tool’ that encompasses the use of traditional tools (e.g. in-depth interviews) to capture systemic change with market actors and project teams. Such tools are often the most commonly applied to understand systemic change, despite the lack of guidance that exists around their use for this purpose.

Next steps

What is clear from an initial look at the list is that they collectively allow us to capture different systemic changes, and at different points during a project’s life cycle. They also differ in the extent to which they are indicator-based and capture expected strategy versus emergent (non-planned) changes. However we want your feedback. What have we missed? What tools strike you as the most promising? Which tools do you want to learn more about? We would greatly welcome your participation at the webinar to discuss these topics. Beyond the webinar, we are keen to identify practitioners interested in piloting approaches to measuring systemic change or who are already doing so and would like to share their learning. If either of these descriptions matches you, please reach out as we would love to hear from you. This summary flyer explains more about our focus and the best ways to contact us. 

Ben Fowler is a Principal Consultant at MarketShare Associates (MSA), a consulting firm focused on implementing and measuring innovative economic development approaches. Ben has worked on measuring and evaluating systemic change in numerous systems facilitation projects. He has written a number of papers on results measurement, including a recent literature review and synthesis of approaches to capturing systemic change for USAID and a paper that informed the Donor Committee for Enterprise Development's revised guidance on systemic change. Ben has over 10 years' experience in economic development across 32 countries, including long-term roles in Kenya and Guatemala. He previously worked for MEDA and the Aga Khan Foundation.