Field trials of tools to monitor systemic change and early change
BEAM partnered with USAID's LEO project to support activities that directly involve practitioners and generate new knowledge products.
How does one define and recognise significant, enduring, pro-poor change in market-systems resulting from the activities of development agencies? What are the defining features of these kinds of intermediate outcomes? What practical methods and tools can we use to monitor systemic change and early change? We are exploring these questions through a series of tests with projects that are experimenting with tools for measuring systemic change, and profiles with projects that have already experimented with those tools.
Approach
The USAID-funded Leveraging Economic Opportunities (LEO) project has been working to improve monitoring practice by identifying promising methods and tools for monitoring systemic change and early change, which generally occur prior to achieving project goals of sustainably reducing poverty and malnutrition at scale. BEAM, following extensive consultation with implementers and other actors involved in market systems programmes, identified ‘improved tools for evaluating systemic change’ and ‘identifying systemic change’ as the two themes where the difference between the interest and the availability of information was largest.
BEAM and LEO have come together to bridge this gap. We have identified a series of core questions that closely match LEO’s focal questions. In combination, they are the following:
- How does one define and recognise significant, enduring, pro-poor change in market-systems resulting from the activities of development agencies? What are the defining features of these kinds of intermediate outcomes?
- How can we identify early changes that reflect progress in a market systems facilitation project, before systemic changes and final project goals have had enough time to occur?
- What practical methods and tools can we use to monitor systemic change and early change?
- How can these results provide feedback to improve the management of facilitation activities?
In March 2015, a webinar on tools to capture systemic change was co-hosted by LEO, the BEAM Exchange, DCED and MaFI. The research led shortlist of seven relevant tools presented at the webinar, which include: most significant change, outcome mapping, outcome harvesting, SenseMaker, social network analysis, systemic action research / participatory systemic inquiry, standard measurement tools to capture key indicators.
The LEO/BEAM-sponsored tool trials are intended to address the above questions through a series of tests with projects that are experimenting with tools for measuring systemic change, and profiles with projects that have already experimented with those tools. Also, they aim to identify and classify indicators of systemic change and synthesize approaches for adaptive management.
It will enable missions and implementing partners to understand whether the programming they are funding is leading towards expected results when still early in an activity’s lifecycle, and to capture evidence that systemic changes are occurring.
Outputs
-
Synthesis Report: Testing tools for assessing systemic change
The seven tools were narrowed down the seven tools to three, to be tested by existing programmes: SenseMaker in Mozambique; Network Analysis in Sierra Leone and Outcome Harvesting in Georgia. The research also profiles existing programmes using these or the other tools.
We worked with three programmes, testing tools like SenseMaker and Outcome Harvesting and documenting the learnings to share. The research provides learnings on how programmes can best use the tools to measure and identify systemic change (e.g. identify early changes that reflect progress) and use the results to improve adaptive management; compare the tools (e.g. for what type of systemic change does each tool work best); and provide guidance on the practical aspects of the tools (e.g. ease of use).