Monitoring and results measurement (MRM) is a crucial aspect of any functional programme. This is especially true with regard to market systems approaches, where there is need for up-to-date information that allows adapting programme strategy in reasonable time. However, dedicating time and resources to MRM can be a challenge in the context of limited team resource and small budgets. This how-to note outlines one programme’s approach to 'rightsizing' MRM in this context.
The challenge
Balancing rigour with feasibility is a key challenge for market systems programmes, which often operate with small, technical teams and relatively limited MRM budgets. Combined with donor demands related to 'value for money', programmes are often challenged to provide evidence of results with a limited ability to do so. This requires MRM teams, and programmes more generally, to find innovative solutions that adapt the design of MRM systems to fit the size and capacities of the programme, i.e. 'right-sizing'. Right-sizing is not, however, about shifting resources away from MRM.
RisiAlbania’s approach
Due to budgetary constraints, RisiAlbania only has one dedicated MRM person, who is on the programme 80 percent of the time and can at most spend a third of that expressly dedicated to MRM. Such a low resource allocation to MRM can be partially explained by a combination of under estimating the task at hand by both the implementer and the donor, as well as from the development of broad sector strategies that further stretched relevant resources that could have been put towards MRM. Given that the programme has a wide breadth in terms of, for example, working in four distinctive sectors focused on growth and intermediation, it was imperative to devise a means of utilising time and financial resources effectively, early on.
The team agreed that they would not be able to achieve every aspect of DCED Standard compliance, especially some of the more rigorous standards related to programme attribution. Instead, the team would strive to align itself with the Standard. Risi considered five aspects to do so:
- Developing manageable results chains
- Prioritising where to invest
- Collecting data
- Measuring indirect impacts
- Determining attribution
Developing results chains
As many staff had not been involved in market systems programmes before, results chains were a relatively new concept. The idea was to introduce complexity gradually, and progressively build the MRM system over time, rather than start immediately with a fairly robust system. This approach fitted well with limited time resources and allowed RisiAlbania to progressively build the capacities of staff. The team started with an overall intervention logic, which would be developed into early results chains that were not fully fleshed out. As they gathered further information on each area, results chains were then developed into further detail. This way, results chains were progressively built over time.
One of the challenges with this approach, however, is that results chains are not really being used as a planning tool by RisiAlbania. Instead, they represent a reasonable logic of how change will happen over time. As a result, there have been cases where intervention teams had gaps in information, leading to moments where they have had to ask, "Did we miss a crucial piece of information along the way?" In the end, these issues have sorted themselves out but they could have been addressed earlier by better referring back to the results chains as blueprints for work plans.
Fleshing-out results chains: example of a tourism product development intervention
RisiAlbania’s intervention focuses on establishing and institutionalising a tourism award, RisiTuristike, that will act as a sustainable mechanism to stimulate tourism-related product development in the country. The first results chain showed only the functioning of the award as the systemic change sought, with its general impact (more products, more tourists staying longer and more income and employment) and general areas of activities. As the intervention progressed, the results chain was further developed to better illustrate the strategy of the intervention, divided into three distinct phases - 1) development and testing of the concept, 2) identification of best partners and hand over, and 3) institutionalisation - as well as specific programme activities. This latter results chain better illustrates the different paths to employment taken by beneficiaries as well as indirect impacts that can be expected, which helped in defining how, and when, different indicators would be measured.
Prioritising where to invest
After having results chains developed for interventions, it was important to prioritise where to invest. In all of Risi's discussion about 'right-sizing', a guiding principle has been how useful a particular indicator/data set is for 1) steering, 2) learning and 3) reporting/accountability. This has helped the programme decide where compromises needed to be made, from deciding not to measure a particular indicator to simplifying the data collection process. For example, some indicators are crucial for understanding whether systemic change is occurring or not, and they are of course measured rigorously. Other results/indicators potentially make attribution easier, but are actually not very useful for either steering an intervention or reporting. In such cases, the programme may decide not to measure them and use other means or proxies to bridge the results chains logic instead.
This principle has also helped prioritise which investments (in terms of time and finances) are made in MRM, in particular with regard to measuring higher-level impact. Prior to having measured results, a programme in general has a sense of whether an intervention has had a large scale impact compared to other interventions, and this can help guide decisions as to the depth and rigorousness of measurement. For example, in agro-processing, Risi determined that the scale of impact would be limited. Therefore, the programme has decided to 'compromise' on data collection. Risi’s media work, alternately, was showing great potential for both scale and depth of impact, and so the programme decided to allocate important resources to an in-depth case study.
Deep dive into understanding attribution: RisiAlbania’s media work
The success of this media work had led, for example, to the decision to carry out a thorough and independent case study of RisiAlbania’s media intervention. The team needed a clear picture of how its media work was having an impact on decision-making of the audience beyond whether or not they were tuning into the programmes RisiAlbania was supporting.
Through the utilisation of extensive surveys and investigative journalism techniques, the team was able to draw out detailed information as to how the programme impacted on the lives of specific young people listening. For example, there was one documented case where after listening to a supported radio programme, a woman opened a training centre for the hospitality industry and within months had almost 50 graduates, each of whom had found a job or had been promoted. The case study was expensive and could not be repeated for all interventions, but it gave much needed clarity as to the efficacy and impact of the media intervention.
Collecting data
It is also necessary to delegate responsibility for key tasks associated with data collection. As with most market systems programmes, all intervention managers have a role to play in most aspects of monitoring, from developing results chains to data collection. This initially proved to be a challenge on RisiAlbania, given that managers were new to market systems approaches and were used to working in contexts where MRM functions existed in an independent team.
Shifting the MRM burden to everyone required heavy coaching early on with staff and training focused on building MRM systems and data collection. Given that RisiAlbania has very limited resources to collect data, they rely heavily on partners to collect information beyond their usual business records to determine if change is happening over time. Commissioning surveys has been limited to strategic interventions and cases where partners would not have the capacity to collect data or may bias the collection. The project admits that this is not as rigorous of a process as they would like. As such, they try to diminish bias through cross-checking exercises and regular field visits, as well as adhering to strict internal control and financial monitoring systems.
Drawing on partner data in agro-processing
In a promising intervention with banks in order to improve access to finance for agroprocessors, RisiAlbania will have to rely exclusively on data provided by the banks themselves. The impossibility of accessing client lists (confidentiality) as well the prohibitive cost of a large, random survey, makes this a necessity.
Measuring indirect impacts
Programmes should look to reliable multipliers where available, as these can be useful in estimating (indirect) impact level results where more rigorous alternatives are not realistic. This was helpful to Risi in ascertaining spill-over benefits in its tourism intervention.
In other interventions, however, data has been scarce and proven to be less reliable. In these cases, Risi has not really been able to work with multipliers. For example, for RisiAlbania’s work in tourism, government data in Albania is not considered a valid source for determining if there is an influx of new visitors. It is important not to think of multipliers as a means of simply inflating programme impact, but rather one tool within an overall suite that offers insight into potential impact. Multipliers should be reliable and rigorously tested.
Multipliers in tourism
For a tourism intervention seeking to stimulate product development, measuring direct impact is fairly straightforward. RisiAlbania has worked to encourage tourism businesses to develop new products such as biking and bird watching tours, neither of which had existed in Albania beforehand. This makes it easier to measure impacts such as new jobs created, number of tourists purchasing these packages, and new revenue as a result. Tourism is a multi-faceted sector, however, with spill-over benefits. When tourists come to Albania on a biking tour, they will often leave their hotel and roam the streets purchasing souvenirs, food, beverages, etc. The more tourists who do this, the more revenue increases which, in turn, creates further employment. These types of indirect benefits are not easy to measure. To do this, RisiAlbania relies on an economic multiplier based on reliable data from the World Travel and Tourism Council, which makes the argument that for every direct job created in the tourism sector, 2.5 other jobs are created through spill-over. Risi hopes however to be able to triangulate this data with an independent study in phase 2 of the project, once the intervention has matured.
Determining attribution
In many cases, Risi relies on the triangulation of business surveys as a means of approximating attribution. For example, for its agro-processing work, RisiAlbania has focused on the provision of new services such as marketing and certification. If they wanted to utilise a counter-factual, then they would have to incorporate a series of additional activities without the requisite human and financial resources to do so. Instead, they surveyed both businesses and training participants.
Challenges of attribution in agro-processing
When a partner marketing consulting company trained future service providers, RisiAlbania would not only need to interview all the young people attending as well as the businesses that directly benefited from the marketing services but also young people that fit similar demographics and a host of other businesses working in the same industry in order to determine ‘difference in differences’. Unable to carry out those additional surveys, RisiAlbania decided to undertake a business survey with their partners as well as one for training participants. Included in both surveys are opinion-based questions that for example ask participants what they would be doing differently if they had not participated in the training. Admittedly not an airtight, scientific approach to measurement, this nonetheless gives the team a lens into how participants (as well as organisers) have interpreted and reacted to the training.
Going forward
The process of implementing an adequate MRM system in a programme where so many things were new, from the use of a market systems approach to a DCED-based MRM system, has been an important learning journey for all involved. While in this phase the programme has resorted to the adjustments mentioned in this document, it is clear that the second phase will benefit from additional resources, which would have been highly valuable throughout. Rightsizing should be about compromising where necessary and not about diluting resources from MRM deliberately. Regardless, the programme and others in the region are continuing to explore innovative ways to tackle the ‘right-sized’ MRM challenge by looking at all the different aspects of the MRM system: from HR and knowledge management to more user-friendly tools and incremental development principles.
Published: January 2017