Our third webinar on systemic M&E focused on the challenges and potential of SenseMaker.
The third and final webinar in the series on using systemic M&E in Feed the Future Uganda focused on SenseMaker®. This M&E approach combines an innovative research methodology with patented software to collect and analyse large quantities of narratives in order to understand complex change. It brings together insights from anthropology, complexity and cognitive sciences.
The format of the webinar was a conversation between the panellists Leanne Rasmussen and Eric Derks, and myself, with Lucho Osorio asking questions from the audience. First, we talked about FTF Uganda’s experience with SenseMaker. The intention of using SenseMaker in the project is to capture attitudes and perceptions of market actors. The project wants to investigate behaviour changes such as the adoption of new business practices to find out what perceptions and attitudes are driving them. The assumption is that if observed behaviour change can be linked to changes in mentality and attitudes of the market actors this would indicate more deeply rooted and sustainable change.
The project collected SenseMaker data from markets actors that are supported by the project - the same people that were already included in the transactional network analysis. They asked people to tell a story about an interaction with a supplier or a seller of agricultural inputs they made business with. After telling the story, the respondents answered a number of questions on their story using signifiers that were developed by the project team in advance (an example of a signifier used by the project is shown in Figure 1). Through the signifiers, the respondents can interpret their own stories – rather than the stories being collected and interpreted by the project team. The team used the two basic trading paradigms they work with as the basis for developing the signifiers – the current paradigm reflects the use of trading practices based on short-term profit versus their hypothesis about an improved paradigm where market actors are using a more customer-oriented and long-term business strategy.
<cms-plugin alt="Image - sensemaker_blog.png " title="Image - sensemaker_blog.png" id="14947"></cms-plugin>
In contrast to traditional surveys, the SenseMaker approach implements a number of insights from cognitive science on how people answer questions in surveys. For example, when people are asked direct questions about mentality or attitudes, they usually answer with what they think you want to hear. SenseMaker frameworks are designed to not give away the ‘right answer’ but rather stimulate people to think about what the specific event in their story means for them.
Designing the signifiers is not easy; in fact, it was one of the biggest challenges Leanne and Eric mentioned in connection with SenseMaker. Also getting people to understand how to use the signifiers to interpret their stories was challenging for the team. As the tool was new to the team it took them a few months of drafting, testing and revising to get the signifiers right.
Now, after having used SenseMaker to collect data over two agricultural seasons, the team is starting to think that focusing on an evaluative research question (i.e. are market actors adopting new business practices?) might not be using SenseMaker to its full potential. Could they also use SenseMaker as a powerful approach to project monitoring? Rather than testing a hypothesis, this would enable the project to openly scan for change. In this way, they could detect early signs of change they might not have expected or change that takes a different form than predicted. For Eric, the use of SenseMaker as a tool to continuously explore different patterns of change rather than to prove the project’s theory of change could lead to information that allows for better adaptation of the project’s activities. In this way SenseMaker would allow for much improved short-term decision making but also for better informed strategic shifts in the project.
One practical question from the webinar audience was about the sample size needed to get meaningful data out of a SenseMaker study. Leanne explained that their team used half of the same respondents as for the transactional network analysis for stories using SenseMaker. This led to between 200-260 stories for each season. To be able to identify patterns in SenseMaker data in a meaningful way one should aim to have at least 100 stories for each stratum in the sample. But in principle one can apply ‘traditional’ statistical sampling calculations in order to get statistically significant data. SenseMaker is, however, not searching for an average answer to a question but rather for patterns within the answer. This means the more stories collected, the higher the chance is of finding interesting patterns such as the odd outlier or interesting case that does things differently – and potentially more successfully – than all the others. This can, for example, be done by continuously capturing stories rather than having neat collection cycles.
Find out more about the webinar series on systemic M&E.
Marcus Jenal leads BEAM's M&E activities. As an independent consultant and Mesopartner Associate, Marcus has contributed to SEEP's Systemic M&E initiative and co-authored two papers on the topic.
Add your comment
Sign up or log in to comment and contribute.
Sign up