The schism in systemic change measurement

There are two schools of thought in the MSD community as to how to go about assessing systemic change. These are rooted in two different ways of seeing systems.

One school focuses on the system as a complex entity, where the whole is greater than the sum of its parts. Here intangible emergent properties of the system are a key part of the measurement of systemic change.

The second school also recognises that systems are complex and adaptive, but believes that the important parts of the system can be reduced to the actions, actors and resources that, in aggregate, form the system. And hence the belief here is that systems are no more than the sum of these parts. To this school, most of what is important about systemic change can be observed in the aggregated changes in actors’ behaviour, and the resource impacts of these behaviour changes.

This reductionist approach does not ignore many of the things we might think important. For instance, a reductionist will likely also consider that emergence, resilience, adaptation, adaptive capacity, relationships, norms, innovation, and so on are important elements of systemic change. But a reductionist would argue that these can and must be defined in terms of actions, actors, and resources in order that they be readily understood and measured. Similarly, reductionism does not take a view either way as to whether MSD programmes should seek shallow ‘technical’ or deeper ‘transformative’ change (as Matt Ripley puts it), only that whatever the parts of the system we seek to change, they should be defined in terms of actions, actors and resources.

There are important consequences of the way systems are viewed as to what we think constitutes progress in how to go about measuring systemic change. If systems can be reduced to actions, actors and resources, this is helpful for measurement, because measurement necessarily operates at the level of actors. If, however, this misses the central systemic impacts of MSD programmes, which are only observable holistically, then time would be better spent trying to understand and define the intangible emergent impacts of systemic change programmes so we can better measure them.

Here I might lay my cards on the table – though you may have caught a glimpse already. I’m in the second, reductionist school. If impacts of systemic change programmes are not manifest in some change in what people are doing, and the resource consequences of those behaviour changes, then until someone can tell me exactly what it is we’re missing then I really don’t think it matters. And I think our time is better spent trying to improve our approach to measuring the tangible systemic changes we can readily understand than trying to define and measure some property of systemic change so intangible we struggle to even describe it.

This is important because measurement of systemic change in MSD programmes is often quite mediocre and inconsistent. I don’t think what is holding back improvement in measurement is an insufficiently holistic understanding of systems. Rather, what is holding us back is the lack of clarity in the process and methodology of systemic change measurement. As such I believe we need increased effort in setting out a clear representation of systems and systemic change that those funding, implementing and evaluating programmes can easily understand, and that connects meaningfully to the work programmes are actually doing on the ground.

A new 3sd Research paper provides a step-by-step process to guide implementers’ and evaluators’ efforts to assess the tangible elements of systemic change. There are Six Steps, three that provide a System Snapshot that can be repeated in order to asses how system composition and performance change over time. And three more steps that analyse System Dynamics, supporting understanding as to why the system has changed and how this impacts sustainability.

• Read the paper: The antidote to systemic change frameworks