Measuring how long a programme's impact lasts is crucial to understanding its sustainability.
How often do you hear the word 'sustainability'? In development we almost all recognise sustainability as a critical component of success. Yet privately, whether we work as a donor, implementer or consultant, most of us pay less attention to sustainability than we would like. Why is that, and how can we fix it?
The first reason why sustainability is valued more in principle than practice is incentives.
Development programmes’ performance is judged more by how many people the programme impacts and by how much, than by how long that benefit will last.
For example, we recently analysed a random sample of 30 development programme logframes from 2017-20191. Only five explicitly mentioned post-programme impacts, and only 40 per cent had any form of sustainability indicator as a top-level objective. Just under half (47 per cent) had no sustainability indicator at all. Since logframe indicators and targets are the main way we assess programme performance, this is alarming. What gets measured in the logframe is what most implementers feel most pressure to deliver2.
This is more a symptom of the problem than its root cause though.
Why isn’t there more pressure to deliver on sustainability?
Crucially, sustainability is both hard to measure and hard to predict. Sustainability of a programme’s impact cannot - by definition - be measured within the life of a programme. By the time it can be measured, reports have been written, staff have disbanded, budgets are closed and nothing can be done to change a poor result.
Consequently, neither donors nor implementers have a sufficiently strong incentive to overcome the logistical, technical and financial challenges of returning several years after programme completion to measure whether impact has lasted.
What can be done about measuring sustainability?
It would be easier to increase attention to sustainability if we could accurately predict it.
Fortunately, development agencies already have a reasonable idea of which factors increase the probability of development impact lasting. Springfield, and others, have even outlined suggested indicators of sustainability which programmes can use while interventions are live, to predict how likely benefits are to last3.
Such proxies for sustainability are a great start, but they have their limitations. So far, there is not enough evidence to convince sceptics of their utility4. Even Market Systems Development programmes, which are explicitly geared towards sustainability, struggle here.
What we need is strong evidence of the characteristics that we can observe and measure now that consistently correlate with impact lasting long-term. If we could base our findings on rigorous empirical research we would be a lot closer to developing more widely accepted sustainability indicators.
We already have an idea what this research could look like. What if we analysed a sample of development programmes, all of which ended at a similar time, and measured how much of their impact had lasted? If our sample programmes were sufficiently well documented - and most are - we could then compare characteristics of their intervention strategies. This would allow us to begin to identify differences between the sustainable and unsustainable interventions that are measurable during the life of the programme.
Armed with this information the possibilities look exciting. If you know which factors correlate with lasting impact, you can look for them when monitoring and measuring a programme’s results5. This could be done ‘live’, using data to identify weaknesses and improve interventions during the life of the programme.
Ultimately the results of this research could reduce the pressure to focus on short-term results by providing a measure that correlates with lasting impact.
Help with our research
With such ambitious objectives, we appreciate the importance of getting the research right.
If your organisation funds or implements high quality applied research and you are interested in partnering with us - or if you simply have ideas on how this research could be done well - Springfield would like to hear from you.
Contact Rachel Shah at The Springfield Centre
1We found two bilateral and one multilateral donor that make logframes available online and drew a random sample of 10 logframes from each. Programmes or projects were limited to those that were ‘active’ at some point between 1st January 2017 and 1st June 2019. Although the sample size is too small to be representative, randomisation was used to reduce bias.
2To check this, we asked a range of programme managers and advisers what they feel most pressure to achieve. With few exceptions, they cited greater pressure to create impact during the programme than to create impact likely to continue after it ends.
3For example, sustainability indicators associated with Adopt-Adapt-Expand-Respond. We have also been influenced by a recent paper from AIP-Rural (PRISMA): Shahi, Nasution and Tomecko (2018) Understanding, Promoting and Measuring Behaviour Change in Partners.
4Those logframes that did use a sustainability indicator mostly focused on measures of increased institutional capacity. Some used changed perceptions or independent co-investment from local players as proxies for sustainability. These logically indicate increased likelihood of sustainability, but to date empirical evidence on the correlation between such proxies and actual sustainability is lacking.
5We recognise that correlation does not equal causation. However, such research provides a stronger evidence base than we currently have, as well as a foundation for future research into causality