Kenya Markets Trust (KMT) works in partnership with the private sector and government to transform how markets work so they become more inclusive and competitive. KMT’s focus on markets recognises that they are the main mechanism through which wealth is created and growth occurs. KMT’s long term goal is to deliver large scale, sustainable change in selected markets which benefits all market players, including poor producers and consumers. To support this goal, KMT receives funding from DFID, the Gatsby Charitable Foundation (GCF) and the Embassy of the Netherlands under the six-year Kenya Market Assistance Programme (MAP).
In contrast to many others, MAP’s evaluation is not fully implemented by a third 'independent' party, but managed 'in house' in collaboration with external partners. It is a longitudinal evaluation, running alongside the programme’s implementation. Activities for this theory-based evaluation started in 2013 with the collection of baseline data for the initial portfolio of sectors. The evaluation will continue until 2017.
Background
Kenya is the fifth largest country in sub-Saharan Africa with a gross domestic product of $55 billion and a population of 44 million people. More than 70 per cent of Kenyans live in rural areas. Rural households rely on agriculture for most of their income and subsistence consumption, making them especially vulnerable to climate and price shocks. Poverty rates are highest (more than 70 per cent) in the arid and semi-arid regions in the north and north-east where inadequate adaptation to low annual rainfall, reflected in low agricultural productivity, contributes to acute poverty, food insecurity and even armed conflict.
KMT’s ambition is to transform markets so all market actors benefit, including big business, SMEs, micro entrepreneurs, civil society and government. Without vibrant and strong businesses, an appropriate role for government and an engaged civil society, markets will continue to underperform and stagnate. Ultimately the programme's main target group consists of those poor producers and micro businesses which are unable to participate in markets effectively, or to realise the full extent of their economic potential. MAP applies a Making Markets Work for the Poor (M4P) approach to designing, implementing and measuring its interventions. MAP's theory of change is summarised below. Currently, MAP works in five agriculture sectors – inputs, seed, dairy, livestock and aquaculture, and one basic service sector – water.
Evaluation purpose
The impact evaluation of MAP serves three main purposes:
- To assess the programme's impact and provide a means of accountability to MAP funders and other stakeholders in terms of results and goals. By engaging an external firm, the evaluation aims to provide an unbiased view of the programme's successes and failures.
- To add to the existing evidence base on market-focused agricultural development programmes. In so doing, provide important lessons for the design of similar projects in the future.
- To contribute to the process of adaptive management within MAP. The baseline and mid-line data will provide an important source of information which MAP can draw upon as it adapts and expands its approaches over the programme's life.
Methodology and data collection
The evaluation targets the dairy, water and agriculture inputs sectors. Data is collected at baseline, mid-line and end-line for the sectors' indicators of 'increased access and growth' and 'impact level'. It will also capture the overall market systems level changes within KMT's partner and a number of non-partner market actors.
The individual sector evaluations employ a mixed methods approach with an important emphasis on qualitative data collection and analysis to supplement the quantitative analysis. Quantitative surveys are done with a longitudinal sample, i.e. the same people are contacted for all three stages of data collection. Qualitative research comprises focus group discussions and one-on-one interviews with farmers/consumers at each of the programme implementation sites for each sector, and include both participating and control group farmers as relevant.
In addition, stakeholder-mapping at the start and end of projects in the dairy and input sectors in one case area will add further understanding of changes in the market system in terms of the relationships between actors, structures, dynamics and processes.
The evaluation of the dairy sector employs a difference-in-difference methodology with propensity score matching. The approach compares outcomes for MAP beneficiaries to those of a comparison (or 'control') population of dairy farmers that did not participate in the programme. Data will be collected using surveys of dairy farmers both before and after MAP implementation. Focus groups and key informant interviews will provide an important means of explaining the underlying reasons behind quantitative findings, as well as exploring anomalous or unexpected results.
The evaluation of the water sector focuses on outcomes for water users in the areas in which MAP is active. In this case, many of the outcomes of interest can be safely attributed to the programme without the need for a counterfactual as many outcomes of interest are only affected by interventions. For example, changes in water quality and time spent collecting water in project areas was only affected by programme interventions as opposed to other factors. Consequently, a before-after comparison without a control group is used. Previous research on links between water quality and health outcomes are used to estimate impact.
For KMT's inputs sector interventions, identifying a suitable comparison group would be problematic due to the fact that private sector partners self-select into the MAP programme, and moreover farmers self-select to be customers of particular companies. Consequently, at the market-system level, the evaluation will compare the activities of input firms before and after activities take place. Similarly, at the target group and impact level, a longitudinal quantitative survey will compare the situation before and after with a sample of customers/farmers. This will be combined with detailed longitudinal case studies of a specified number of customers/farmers of agro-dealers/retailers/soil testers. This more detailed understanding of how change happens partly compensates for the lack of a control group and will help KMT determine what impacts the interventions have had.
The evaluation also includes an Economic Rate of Return (ERR) cost-benefit model in order to provide evidence of the programme's overall impact and cost-effectiveness. The model incorporates data from the individual sector assessments, KMT monitoring data, and other sources in order to account for a wide range of benefits due to the programme that might otherwise be difficult to incorporate into an evaluation.
Coordination
The evaluation is closely linked to the KMT Knowledge & Result Measurement System (KRS), which is based on the DCED Standard for result measurement. While the different sector teams are responsible for monitoring indicators at the intervention and market system change levels, the responsibility to monitor changes higher up the programme’s theory of change lies with the Knowledge and Results Team (KRT).
The KRT collect and collate data from field monitoring and estimate results up to impact level. Using data from the evaluation, the KRT triangulates these estimates and verifies the assumptions. TNS Global in partnership with NORC at the University of Chicago provides independent external evaluation services, and is responsible for the evaluation methodology, research tools, sampling strategy, data collection, data analysis and reporting. DFID Kenya provides feedback on evaluation outputs. The evaluation design is based on the principle of honest inquiry, to optimise internal learning for programme course corrections.
Insights and lessons learned in evaluation design
In designing and implementing an evaluation of MAP, KMT has identified the following:
- Timing and sequencing of different evaluation stages is important. Evaluation activities need to be in sync with the progress of the programme. For instance, a large sector-wide baseline collected after the elaboration of specific intervention results chains and indicators of change risks the collection of a significant amount of irrelevant data. Intervention specific baselines are preferable for this reason.
- The focus of data collection changes as intervention strategy evolves. In KMP's water sector the primary indicator at the impact level was initially the number of households consuming clean and safe water. However, as a result of changes in strategy, the programme began to track changes in income resulting from economic activities which were prompted by the increased availability of water. In such cases, retrospective baselines may be considered by the evaluation during the mid-line data collection
- It can be difficult to find research firms that are familiar with and understand the market systems approach. Programmes designed to facilitate changes in market systems are complex and build on particular theoretical underpinnings which differ significantly from traditional approaches to development intervention design. Even though some social research firms understand the inherent concept of behaviour change and sustainability, not all understand the theory of systemic change and how market systems programmes deliver impact. KMT went through a rigorous induction session with the selected research firm to explain the approach and how the theory of change works.
- A culture of honest enquiry is fundamental to adaptive programming. Within KMT, sector teams, the Knowledge and Results Team and management worked to create a system which accepts failures and learns from them rather than treating them negatively. An evaluation process should not only help the programme to prove its results but help improve its strategy for greater impact.
This evaluation was developed by Khaled Khan and Ryan Masila at KMT, in collaboration with BEAM Exchange. If you'd like to know more about this evaluation email Khaled Khan.