Peer-to-peer workshop reveals growing interest in SenseMaker®

Is SenseMaker® a suitable tool to measure systemic change in complex systems? Read this workshop round-up to find out.

I spent the last two days at a peer-to-peer learning workshop on using SenseMaker® in inclusive business and market systems development. The workshop brought together practitioners from all over the world who are using the software as part of their monitoring efforts. It was two days of lively discussions on what works and what does not when using SenseMaker® to measure change. BEAM Exchange supported this event, organised by the Belgian NGO and SenseMaker® pioneer VECO.

SenseMaker® has gained some prominence as a tool to measure systemic change in market systems and inclusive business and a small community of practice is beginning to form. A number of programmes using market systems approaches are using the tool, including Katalyst in Bangladesh, PRIME in Ethiopia and Feed the Future in Uganda. FTF Uganda’s monitoring framework, including their use of SenseMaker®, was featured in a recent webinar supported by BEAM. PRIME and FTF Uganda were present and shared their experiences at the workshop.

Many participants said they used SenseMaker® because it added a new dimension to monitoring. One participant described the software as a tool that helps go beyond measuring what, ‘we know we know.’ 

Adding less tangible and unpredictable aspects to the mix of monitoring data, it brings a more nuanced understanding of the complex changes programmes are effecting, she added, helping to 'rehumanise' data by 'bringing the voices of the beneficiaries back into the room.'

A participant from Colombia presented how they use SenseMaker® to collect stories from coffee farmers, including ones growing for big companies and those growing to sell directly to speciality roasters in Europe. Against expectations, they were partly surprised to discover that farmers were generally ok with selling to big companies – contesting prior assumptions that they wouldn't be.  

But it was also recognised during the workshop that the software is not an easy tool to use. In particular, there is no training to teach you how to develop a perfect SenseMaker® study. All participants agreed that a lot of learning by doing is required. One participant remarked that designing a SenseMaker® framework 'is as much an art as it is a science'.

Framework design

On the first day of the workshop, discussions focused on what we need to get right before starting to design a data collection framework. There was broad agreement that we need to become better at anticipating the use of the data before jumping into design. 

There was some discussion on who are and who should be the end users of SenseMaker® data. For some it was clear that the data needs to be fed back to the people who contributed it in order to allow them to make informed decisions. For others, data collected through the software is a powerful way to inform programme monitoring and report to donors. 

There was broad agreement that SenseMaker® data should also be used as a basis for programme design. But also that practitioners should think about the analysis of the data before designing a framework. In particular, it is important to understand who is going to analyse the data. If it will be a participatory process, which was seen as ideal by all, this has an influence on framework design.

While there are multiple routes to a SenseMaker® framework, the two cases that were presented both combined a literature study with a workshop. The literature review is used to draw out the most important theoretical concepts and models relevant for the given research question. The workshop then adds the experience of the people involved in important influencing factors.

VECO presented its Inclusive Business Scan, a standardised research framework powered by SenseMaker® that assesses value chains and reveals crucial insights on the inclusion of smallholders in those chains. It provides valuable information for outcome and impact measurement and assists chain actors and supporters to gain insights into people’s perceptions, to understand dominant and deviant patterns and to guide future actions and interventions.

Belgian commodity trading company and impact investor, Durabilis presented how they used VECO’s Inclusive Business Scan to assess the inclusiveness of their operations in countries like Senegal, Malawi and Burkina Faso.

Collecting data: what's good practice?

The collection of SenseMaker® data was described by one participant as the easiest phase. Not all agreed, though. Getting people to share good stories is not easy. But in general, collecting data for SenseMaker® faces similar challenges to traditional survey tools: the training of enumerators, quality of data collected by enumerators, quality control, access to people, etc.

Participants agreed that to collect SenseMaker® data, a clear sampling framework is needed. While the software is often inaccurately defined as a qualitative research tool, it does generate quantitative data. In order to assess this data with statistical confidence, sampling needs to follow statistical sampling rules. In particular, that stratification of the sample into different groups requires a sampling framework that includes sample sizes big enough to make valid claims for the different strata.

A representative from FTF Uganda shared how they organised the collection of stories from agricultural input wholesalers using own project staff. The VECO representative from Mesoamerica contrasted this experience with their use of students from within the communities to collect data. Advantages and disadvantages of both approaches were discussed.

Data analysis

The second day of the workshop was to a large extent focused on data analysis as this poses particular challenges. In particular, there is still a lack of codified processes and good practices around data collection, while analysis does require a certain amount of experience in order to pick up nuances and intangible aspects in the data. 

As SenseMaker® is often used to analyse change in complex systems, the interaction of the main target audiences with the data is central for the use of the data it generates, in contrast to isolated data analysis by an expert followed by publication of the expert’s interpretation of the findings.

A Cognitive Edge representative shared his experiences with organising feedback sessions with different user groups. He stressed the importance of not revealing the data too early but instead first extracting the assumptions and hypotheses that are present in the room. This will allow it later to clearly see a shift from these earlier hypotheses to potential new ones after discussing the data.

The community of practice around SenseMaker® is only emerging, but is enthusiastic about the possibilities the approach opens up. The growing interest in the market development community to test the tool is not without risk because application is not without challenges. Participants of the workshop agreed that applying it without the proper training and experience could lead to frustration and potential dismissal of the tool, which would be a loss for the monitoring and evaluation community.

 Marcus Jenal leads BEAM's M&E activities. As an independent consultant and Mesopartner Associate, Marcus has contributed to SEEP's Systemic M&E initiative and co-authored papers on the topic.