Module 4: Assessing wider system change

Principles to consider in assessing wider system change

The following principles, developed by USAID’s Complexity Aware Monitoring initiative, are useful to bear in mind in designing a monitoring framework that goes beyond intervention performance to look at change in the wider market system.

Observe without focusing on confirming a hypothesis

It is important to keep an open mind. Focusing on proving a hypothesis right (such as one specified in a results chain) will shape perceptions of how trends are interpreted or analysed. It is important to remember for instance, that programme interventions are not necessarily the dominant causal factor in a process of change, and that change may also take a different shape to that which is predicted in a results chain. It is therefore also important to take into account unintended outcomes and other influencing factors.

Consider relationships, perspectives, and boundaries

An insight from systems theory, which can be usefully applied to monitoring in a market systems context, is to focus on relationships, perspectives and boundaries.

Relationships
Any system contains a range of different elements or components, so the ways in which these relate to each other is key in defining the nature of the system. While the question of relationships is central to the field of monitoring and evaluation, the use of such tools that embody relationships, such as results chains or theories of change, is typically static. Key questions to consider in assessing relationships within the market system include:

• What is the nature of the interrelationships? (strong, weak, fast, slow, conflicted, collaborative etc.)
• What patterns emerge from these relationships over time, and with consequences for whom?
• What are the ways in which these relationships change over time?

In a market systems programme, monitoring might therefore need to consider relationships between programme partners, other leading organisations in the market, suppliers, customers etc., how these are changing, and what this means for the implementation of interventions, or the programme as a whole.

Perspectives
A single perspective approach is certain to miss how different market actors relate to each other, and the reasons that they act in a particular way. While a donor and programme team consider that they are implementing an ‘economic development’ intervention, beneficiaries might primarily be motivated for other reasons. Failing to understand this would undermine the reasons for success (or otherwise) of an intervention, and illustrates the importance of engaging with multiple perspectives.

Boundaries
The programme team must be conscious about judgements on what is included within the frame of reference, and what it to be excluded. To extend the previous example, where it is considered that an intervention is about ‘economic development’, monitoring efforts might focus primarily on business issues such as levels of investment, market share, or product development. However, if discussions with market actors or other stakeholders illustrate that other issues are important to the success of the programme, it is important to broaden the boundaries of what is monitored to cover these.

Synchronise monitoring with the pace of change

It is also important for monitoring to adjust to any changes in the pace of implementation, and changes in the broader market system. In highly dynamic contexts, this can mean that monitoring needs to focus on collecting data that is as real-time as possible. To be able to do this, a monitoring system also needs to assess the pace of change. The use of different types of indicators (leading, coincident or lagging) can help do this.

Feedback from this type of open monitoring can also inform intervention performance monitoring. For instance, by providing information about causal pathways this will also support the programme team’s reflections about the assumptions behind the theory of change and the results chains for individual interventions.

Complexity-aware monitoring

Methods and tools that can be used by programmes in complex contexts.

Synchronizing monitoring with the pace of change in complexity: a complexity-aware monitoring principle

Guide on monitoring in complex situations and how to make sense of the data.

Attending to interrelationships, perspectives, and boundaries: a complexity-aware monitoring principle

How systemic monitoring can enhance your performance monitoring system.