Recognise that a big contribution to a small system change may be more important than a small contribution to a big system change.
In this three-part blog series, we share lessons on the process of evaluating the Cambodia Feed the Future Harvest II programme focused on what systemic changes the programme made progress on in the horticulture sector (check out the full report: Assessing systems change in Cambodian agriculture).
The lessons are:
- Evaluate how the ‘puzzle pieces’ of the system are, or are not, fitting together. (Blog 1)
- Recognise that a big contribution to a small system change may be more important than a small contribution to a big system change.
- Start from the programme strategy and system context to determine which dimensions of system change to evaluate. (Blog 3)
In Part 1 of this blog we discussed lesson 1.
In this blog, we focus on lesson 2 learned in the evaluation process.
It’s tempting to focus primarily on the apparent magnitude of changes in a system. But a programme’s contribution to observed changes is just as important in generating useful insights on a programme’s strategy.
When many puzzle pieces are in place, impressive gains in system performance may quickly follow a programme’s contribution to improving one of those pieces. But taking into account the trajectory of change, we found it useful to consider whether a programme’s contribution might have addressed a problem that would have been solved by market actors relatively quickly anyway.
In contrast, when many puzzle pieces are missing, only modest gains in system performance may follow a programme’s contribution to getting a key piece in place. Nevertheless, getting that piece in place is essential to future system performance. In addition, the programme’s contribution may have been significant, considering there were few other pieces to support its introduction.
This doesn’t mean programmes should pursue unrealistic expectations, but it does mean that we, as a field, should recognise early contributions to long-term system change.
Harvest II worked in three tree crop subsectors - cashew, mango, and longan. These crops were mostly for export and faced a challenging environment. Many puzzle pieces necessary to directly serve foreign buyers - such as lab services, key technical skills, supportive government processes, and affordable energy - were missing. The challenging environment was exacerbated by COVID-19 border closures and adverse weather associated with climate change.
Harvest II implemented a range of interventions to build export readiness and enable firms to serve premium local markets. One of these was working with firms and their supplying farmers to obtain certifications necessary for these markets, such as organic certification, Cambodia Good Agricultural Practices (CamGAP), and Hazards Analysis and Critical Control Points (HACCP). While there has not yet been a substantial uptick in direct exports, Harvest II’s efforts to increase certifications and encourage expansion of the standards market are helping firms start to serve premium local markets and laying a critical foundation for future export growth.
We evaluated the programme’s contribution to putting in place or improving pieces of the system puzzle, irrespective of the magnitude of performance change in the system. Some changes that were modest in scope were nevertheless fundamental to the future performance of the sector. By making a big contribution to these, Harvest II has put a piece in place that can be expected to contribute to significant system change in the future.
To evaluate the contribution of Harvest II to specific system changes, we triangulated findings from the ‘helicopter’ perspective that looked at key trends and why they occurred with a theory-based, ‘intervention’ perspective that followed the expected causal chain from interventions through to system changes.
We found that this approach yielded useful insights into how, and to what extent, the programme contributed to observed changes relative to other factors such as government activities, international markets, infrastructure development, and other programmes.
This approach is similar to the one outlined in the recent Practitioners' guidance to assessing systems change, but places more emphasis on triangulating information on causes for observed changes, rather than substantiating contribution claims.
Like the authors of the Practitioners guidance, we found operationalising this approach benefited from multiple perspectives.
We started the evaluation with several two-hour workshops with the Harvest II programme staff to understand in detail the team’s vision for a functioning market system in each crop, and how the programme’s interaction with market system stakeholders evolved over the life of the programme.
With that foundation, we conducted interviews with: firms that the programme had interacted with and those they had not; farmers directly and indirectly impacted; associations; and relevant government agencies. We used interviews with key informants to get another perspective and also to fill gaps in our understanding. An analysis of these varying perspectives informed our evaluation of both the extent of changes and the programme’s contribution to them.
- See Part 1 of this blog, focused on evaluating how the ‘puzzle pieces’ of the system are, or are not, fitting together. And Part 3 on starting from the programme strategy and system context to determine which dimensions of system change to evaluate.
This blog was originally posted in November 2023 on Marketlinks