Measuring impact, managing failure

Clara Garcia Parra of Helvetas interviews Susanne Thiard-Laforet of the Austrian Development Agency to discuss 'impact' from the donor perspective and to share thoughts on learning from failure.

Susanne Thiard-Laforet, an economist and anthropologist, has been working for the Austrian Development Agency, the operational unit of Austrian Development Cooperation, since 2010. 

Susanne is not only the sole in-house expert for Private Sector Development at the Austrian Development Agency (ADA) since 2017, she is also a co-chair of the International Working Group on Market System Development within the Donor Committee for Enterprise Development (DCED).

The temptation to bombard her with questions was strong - but the fact that we had never met in person held me back. Fortunately, Susanne is extremely open and we immediately get talking about one of the toughest topics in international development: the understanding and measurement of impact and how we use failure to learn, adapt and change.

What we mean when we talk about impact

Without much ado, we start with tackling the million-dollar question: what does impact look like?

According to Susanne, achieving impact means 'reaching out to as many poor people as possible, working in a way that is as sustainable as possible by taking care of the effects on society and the environment, as well as taking into consideration the local political circumstances. We cannot do more and should not do less than that'.

But this task cannot ignore the fact that we're always dependent on people: 'In our own teams, in our organisations, political surroundings and, of course, our partner countries - we always depend on people. And different people are shaping the work of such different institutions, public, private… no wonder we have our difficulties with finding a straightforward answer to this question!'

This ‘human factor’ comes to the fore repeatedly during the interview. Susanne links it with one of the major challenges that development programmes face: the fact that we are accountable to different people, often with diverging interests. In that sense it isn't only about defining impact, it's also about telling the impact story in a way that responds to the needs and requirements of various stakeholders. 

But before we get into how best to communicate impact we discuss the issue of designing programmes that are likely to generate it. Does Susanne think donors have the necessary tools to assess potential impact before giving out a mandate?

She immediately answers 'When I think of my own position, working for the national donor organisation of a small but rich country, I'm at my desk in my office - and then I receive project and programme documents where I’m asked for comments, corrections and, ultimately, for my approval. Quite often it’s hard to assess the practicability of a proposal. Mainly because I'm so far away from the actual implementation site and because I’ve never met the implementers!'

Luckily, there are tools out there that can help navigate the uncertainty around how to maximize chances of success.

Susanne mentions the BEAM Exchange Competency Framework, which provides guidance on the complementary skills a team must have to deliver impactful programmes. She explains, 'If you don’t find the right combination of people, the right teams, the right combination of competences in the team, it'll be difficult.' Not surprisingly, she also brings up the issue of time and prioritisation: 'There are so many things to consider before you start thinking about what useful activities you could do to reach the goals in a programme. If we invested more in decent and better preparation of a specific development initiative, it could also lead to bigger impact.' 

Understanding and managing failure

The conversation moves on to how we manage and learn from failure in development programmes. A very important topic that the developmental world has grappled with in recent years.

What role can donors and the DCED play in sharing knowledge among projects to help them incorporate learnings from past failures into their programming?

Susanne first answers in her capacity of private sector development expert at ADA: 'In Kosovo we used to have small projects that worked with farmer groups, women trying to improve their economic standing and their agency. And we had a certain degree of success. But we didn’t reach enough scale and sustainability.'

Things changed quickly, though. She says: 'The teams responsible for the Kosovo budget at headquarters and in the coordination office sat together and decided to try and increase scale based on what we learned from the smaller projects. We formulated a call for proposals, all together it took one and a half years to prepare. It has taken a lot of internal convincing. But at the end, this'll now become the first Market Systems Development (MSD) programme that we'll handle at the ADA. As an organisation we were able to learn from the past 10 years.'

Putting on her DCED hat, Susanne adds, 'At ADA I’m the only person in charge of the private sector development portfolio. This can be very convenient. However, there's no one I can ask for a second opinion when it comes to questions or decisions. That’s why I like the DCED working group as there's people there in the same position as  me. It’s very comfortable to talk about difficulties, and sometimes failures. that we all experience. It’s a safe space, a protected room for free exchange of ideas and for getting over failures, problems and  challenges.' This idea of creating safe spaces where people can interact and exchange both advice and ideas is a key element of being able to incorporate intelligent failures into management decisions to improve development work. A key element of this is shedding the blame-game culture and building constructive dialogue.

Susanne and I agree that if the work we do was easy or straightforward, we would not be having this conversation. This leads us to the topic of organisational culture. It canh often be rigid in development organisations and might  discourage learning and adaptation. Add to that the friction between often centralised decision-making structures on the donors’ side and projects that are set up for very short periods, it is no wonder that building a culture conducive to learning is hard.

When I ask Susanne for some tips on managing failures, she doesn't hesitate. She says, 'It’s very human to make mistakes or having to face challenges – regardless of whether they’re self-caused or external. To be brave enough to talk about failure, to try not to blame somebody else or to hope that nobody notices it is not something everyone can do. We must talk about the problem in order to avoid it – or, if it isn't avoidable, we should learn how to live with failure.'

Market Systems Development (MSD): a failure-friendly framework

In Susanne’s opinion, there's one development approach that particularly supports managing failure: the MSD approach: 'It’s very analytical, but also tries to be pragmatic. At a certain point in time, teams need to realise that they have analysed enough, that it’s time to implement the first set of activities and see if it has the intended impact. And if it doesn’t have that impact, it doesn’t automatically mean that it was the wrong thing to do. The core of development work is to do the things that other people and institutions don’t even dare to touch because they’re so complicated to solve.'

I couldn't agree more. But I’m sure some readers will be thinking about their targets and the pressure they sometimes face for having to deliver according to the log-frame. So then, are we using the right tools to measure change in a way that allows us to capture unintended effects?

To gauge Susanne’s thoughts on this, I present a hypothetical situation where a project underachieves its targeted impact but unexpectedly fosters the creation of a new, inclusive industry that has the potential to grow and develop beyond the lifecycle of the programme. Susanne clarifies my query: 'What you mean is whether that would be perceived as a failure or a success, right? I don’t know. But I think we as donors should ask ourselves this question every now and then. If there's a person in charge of the programme that can sell the success of creating a new market, it could still be recognised as a success in that organisation. If you have somebody there who works in a very administrative way, it will be dealt with more as a failure than as a success. It all boils down to the ‘human factor’. It’s always a question of who's interpreting the results and how.'

Some philosophical musings

We’re almost at the end of our chat, and I ask: should we invest to create spaces where failure can be shared?

Susanne has a unique perspective on the development industry, which she understands as a market system by itself. In her opinion, the systemic change that needs to happen is precisely that of putting 'more time and money into learning, into reflecting on achievements as well as on failures.' This is what initiatives such as the DCED working group she co-chairs are trying to address by way of creating a sense of community among practitioners and empowering them to believe that change is possible. However, she adds, 'We shouldn’t lose sight of what we’re trying to achieve by overemphasising analysis over trying to achieve results.'

She explains that there are many platforms such as the DCED or the BEAM Exchange where one can find competency frameworks and other tools useful for managing failure. 'But' she says, 'you often need somebody to explain to you how to use these tools. So far there are more such services provided to implementing organisations than to donor organisations.' She concludes by saying, 'You can become very philosophical when it comes to our work.'  Again, I couldn’t agree more.

Read more on Impact and Learnings from Failure

This blog was first published on www.helvetas.org