Module 9. Designs, tools and techniques for monitoring

Techniques for understanding market systems change

Different programmes have experimented with a range of techniques to understand how the market systems they are working in are changing, and the kinds of effects that their interventions are producing. Three promising approaches which may be of use for programmes are reviewed here. 

Social network analysis

Social network analysis assesses social relations, interactions and connections between people, organisations and other networks using quantitative data obtained through surveys. The method is not restricted to social networks in a narrow sense but can be used to analyse any network of connections. For example, the Feed the Future Agricultural Input project in Uganda used Social Network Analysis to analyse transactional data between sellers, wholesalers and retailers of agricultural inputs. The networks are visualised in network maps with nodes (the elements of the network) and ties or links (the connections). Part of the utility of network analysis lies in its ability to quantify the structure of networks and therefore to identify changes in relational patterns.

In market systems approaches, social network analysis can be a powerful tool to monitor changes in the relationships among different market actors in a value chain or sector. It can also be used to map market transactions, including frequency of sales and satisfaction with a deal. The key tools used in social network analysis include:

• Objective: the objective of the analysis is identified
• Identification: the network of people, groups or teams is identified, and the relationships between them determined either by observation or through a survey. 
• Analysis: the information collected is analysed with mapping tools to identify changing patterns in the network. This can be undertaken through participatory discussions or analysis of data collected through surveys. 

The Feed the Future project has made use of this technique to gauge systemic change in behaviour and business practices among different market actors in the agricultural inputs sector in Uganda, and to explore the impacts of interventions on smallholder farmers.

In this particular case, network mapping was used to describe patterns in how agricultural input wholesalers interact over time. The Feed the Future programme team were specifically interested in identifying changing patterns of connection, trust, satisfaction, investment in relationships, and frequency of interaction in the sector. The approach proved particularly useful in alerting the programme to the fact that the changes they were expecting to see in the market system were not in fact materialising. 

Team members who used the approach noted that it proved to be very useful in understanding the dynamics of relationships in the market, and was not difficult to put into practice. Challenges include managing the significant amounts of data, and the fact that it was difficult to present the results visually.

Most Significant Change

Most Significant Change is a participatory technique in which stakeholders and staff collectively decide which changes are significant. In this technique, personal stories and narratives are collected from stakeholders at ground level. These stories are analysed first by the stakeholders themselves and then by staff and other stakeholders, to decide which changes are most significant, and why. 

Most Significant Change can provide an effective evaluation tool to help assessing what beneficiaries and other participants think the real successes of a programme are. It may therefore be particularly helpful in evaluating systemic initiatives, because of its potential to identify issues which actors from different parts of the market system agree are important. 

It can be conducted multiple times, at different stages of the project. The stages for using the technique include the following:

• Practitioners decide what type of stories they want to collect, and from whom
• Individual stories are collected in text, audio or video format
• Stakeholders and staff analyse the stories and collectively decide which are the most significant and relevant
• Those stories identified as most significant are then shared with other participants, who also discuss which changes are the most significant and should be recorded.

The Most Significant Change technique does have some limitations: 

• It is a long-term process that takes time for stories to be collected, recorded and shared
• The people involved in story collection and analysis need to have good facilitation skills, and need to ensure that participants remain interested
• The technique tends to be biased towards stories of positive change, to the detriment of stories about lack of change or unintended negative outcomes. To mitigate this, negative stories could be purposely selected in parallel.

Using systemic M&E tools in Feed the Future Uganda: Network Mapping

Using Network Mapping to analyse relationships and patterns of interaction.

SenseMaker

SenseMaker is a proprietary software programme that allows large numbers of personal stories and other narrative fragments to be collected and analysed in a quantitative way. It aims to avoid researchers introducing their own biases into the analysis by asking respondents to interpret or ‘signify’ their own stories by answering a carefully crafted set of questions, called a signification framework. The signification framework allows respondents to interpret shared experiences and also provides quantitative data for statistical analysis. The results of this analysis can then be presented back to respondents or other stakeholders for further discussion of the conclusions. 

SenseMaker has been used by various programmes using Market Systems approaches, including PRIME in Ethiopia, Katalyst in Bangladesh and also the Feed the Future project in Uganda. SenseMaker may be useful for market systems programmes to assess people’s attitudes, beliefs and knowledge. Feed the Future used it to assess how company owners’ mental perceptions of the way their businesses worked for instance

As proprietary software, SenseMaker comes with a financial cost. Other costs associated with its use include the time it takes to learn how to use the tool and to analyse the data it produces, and also the investment in time for data collectors to be trained and supervised, which is necessary to ensure they use the tool consistently. 

Using systemic M&E tools in Feed the Future Uganda: SenseMaker

Recommendations on using the tool to monitor systemic change.