Start from the programme strategy and system context to determine which dimensions of system change to evaluate.

In this three-part blog series, we share lessons on the process of evaluating the Cambodia Feed the Future Harvest II programme focused on what systemic changes the programme made progress on in the horticulture sector (check out the full report: Assessing systems change in Cambodian agriculture).

The lessons are:

  1. Evaluate how the ‘puzzle pieces’ of the system are, or are not, fitting together. (Blog 1)
  2. Recognise that a big contribution to a small system change may be more important than a small contribution to a big system change. (Blog 2)
  3. Start from the programme strategy and system context to determine which dimensions of system change to evaluate.

In Part 3 of this blog, we focus on lesson 3 learned in the evaluation process.

We wanted to be thorough, so we reviewed key literature on dimensions of system change, such as USAID Market systems resilience: a framework for measurement. We chose 22 dimensions that we thought were relevant, such as buyer-supplier coordination, value chain efficiency, information flows, networks, gender equity, and environmental stewardship. We aimed to evaluate each dimension in terms of scale, sustainability, and depth of change. It was both way too many (22!!) and not specific enough!

We realised during the field research that some dimensions were much more important than others in the context of the crop systems and the uniqueness of each crop with respect to the dimension of changes. In this context, there was substantial overlap among some of the dimensions we had chosen, while others were too general. We had visualised what the puzzle pieces looked like before we had enough information.

In hindsight, it makes more sense to start with the programme’s strategy to understand the vision for a functioning market system in each crop to choose which dimensions of system change to evaluate. Then, it helps to refine that list based on early field research. In other words, each system puzzle is unique, so use literature as a menu, not a mandate.

Example:
In our evaluation plan, we separated agricultural policies and public-private coordination. However, interviews showed that we had to consider these together as public-private coordination was a part of the agricultural policy development process that the programme worked on. At the same time, we had to consider more than just policies. Other levels of the business enabling environment (BEE) were arguably more important, such as the regulations and procedures for exporting. (For more on levels of BEE, see Enhancing the Use of Evidence and Results Measurement in Business Environment Reform Programming).

Reflections

We defined the dimensions of system change to evaluate after reviewing the programme literature and workshops with programme staff but before fieldwork started. Looking back, it would have been more effective to delay the definition of dimensions until some interviews with market actors and key informants had been done.

It also would have been more practical to limit the number of dimensions to roughly 6 -10, depending on how broadly they were defined. In the end, our analysis focused on 13 dimensions, combining some from our original list and making others more specific. Limiting the number of dimensions can help to keep the evaluation focused on robustly investigating the most important aspects of system change according to the programme strategy and the system context.

We also found it was important to explore and pinpoint the most important levels of change for each dimension. In the BEE example, it was helpful to pinpoint missing puzzle pieces at the level of regulations and procedures, not only at the level of policies. This hunt for the ‘key puzzle pieces’ in each relevant dimension further hones the explanation of how, why, and to what extent system changes are happening.

Effectively assessing system changes, both during and after a programme, can help donors and practitioners to better design and adaptively manage programmes so that they contribute to transformational change with widespread and positive impacts.

The assessment itself is fascinating - but tricky! Let’s keep the conversation going on the best ways to practically do it.

  • See Part 1 of this blog, focused on evaluating how the ‘puzzle pieces’ of the system are, or are not, fitting together. And Part 2 on recognising that a big contribution to a small system change may be more important than a small contribution to a big system change

This blog was originally posted in November 2023 on Marketlinks

Add your comment

Sign up or log in to comment and contribute.

Sign up