Propcom Mai-karfi is a DFID-funded programme that uses a market systems approach. This evaluation synopsis reviews the independent longitudinal evaluation of the Propcom Mai-karfi programme conducted by WYG and Upper Quartile using a theory-based approach.
Longitudinal evaluations are implemented alongside the actual programme they evaluate, covering longer time frames compared with traditional mid-term or ex-post evaluations. The evaluation started in 2013 and will run until 2018. As of July 2015, the first phase of this evaluation is being implemented.
Background
Building on the positive experience of its predecessor programme Propcom 1, Propcom Mai-karfi aims to stimulate sustainable, pro-poor growth in selected rural markets and improve the position of 650,000 poor men and women within these market systems. Propcom Mai-karfi aims to work in a minimum of eight rural markets, both agricultural and non-agricultural, in northern Nigeria.
The evaluation team supports the development of the programme and market-level theory of change where possible with the aim of ensuring it provides a suitable framework for the evaluation.
Propcom Mai-karfi evaluation aims:
- To add to the body of knowledge regarding M4P programmes (learning)
- To determine to what extent the programme has achieved its objectives (accountability).
To this end, the evaluation aims to both verify the programme’s claims of increased incomes and resulting decrease in poverty, and the extent of pro-poor systemic change the programme has achieved. The evaluation team developed a set of questions to assess systemic change, which are informed by the Donor Committee for Enterprise Development’s (DCED) definition.
Methodology
The evaluation combines a mixed methods approach for data collection, including a quasi-experimental quantitative study, using successive independent random sampling for beneficiaries and non-beneficiaries.
This means that each wave of survey involves a different, albeit representative, sample of farmers. This allows for comparisons over time, enhances the internal validity of the evaluation, and surveys are less vulnerable to respondent attrition (a common issue with longitudinal survey designs). The disadvantage is that the survey does not allow for the evaluation to trace the development of individual farmers. However, given the survey's primary purpose — to determine the impact in two selected programme markets — this is a lesser concern.
Statistical power analysis was used to calculate the minimum sample size required to detect reasonably small differences between beneficiary and non-beneficiary farmers. The sample size and distribution will be revisited when the evaluation team designs a detailed survey strategy. This will depend on the market type and who the target beneficiaries are.
Based on the data collected, the evaluation will use contribution analysis to come to robust conclusions about how Propcom Mai-karfi did or did not contribute to observed outcomes. Insights will be used to revise and substantiate, or invalidate, the programme’s theory of change.
Propcom Mai-karfi is a large and complex programme, so data collection and analysis are focused on specific sectors and interventions. Initially, the evaluation team gained a good understanding of all of the programme markets and interventions. Subsequently, robust criteria to select markets and interventions for evaluation, based on the evaluation objectives, were developed.
One important objective is to add to the body of knowledge on the market systems approach. To this end, key selection criteria including: the intervention’s adherence to market systems principles, degrees of innovation, potential scale and impact, and potential for gaining learning and insights about the market systems approach were developed. Propcom Mai-karfi adapts its interventions and sector focus to learn about what works and what does not. Practical considerations such as monitoring data availability and quality, the likelihood that interventions will continue, as well as the cross-cutting issue of gender, were also key selection criteria.
To enable the programme to react to change, the evaluation budget provides for three ‘close-out’ evaluations should the programme withdraw from a sector, and two 'interim evaluations' after significant market events. These events are external to the programme but have an influence over the market. They may be part of the natural cycle of the market (for instance, a harvest), or an event external to the market that impacts on the market and the programme market intervention (for instance, a policy on increasing or decreasing subsidies).The additional close-out and interim studies will focus on targeted data collection for learning purposes rather than full-scale quantitative surveys. If these additional studies are not needed, the budget for this will be re-directed for other data collection activities.
Data collection
Methods for the evaluation include:
- Quantitative surveys in two markets at two different points with 3,800 farmers/small rural enterprises (1,600 in the baseline and 2,200 in the final survey), designed to explore the impact of market interventions in depth at beneficiary level.
- Focus group discussions providing qualitative data on farmers’ diverse experiences of the services, their relevance, benefits and unintended consequences. This adds explanatory power to the quantitative data. Women-only groups, facilitated by women researchers, will explore the impact on women.
- Case studies allow the evaluation to provide a deeper understanding of how Propcom Mai-karfi brings about change. Innovations and failures will be considered as selection criteria.
- Key informant interviews (KIIs) focus on the design and implementation of interventions as well as the extent to which they have contributed to systemic change. KIIs will include programme staff, local partners and enterprises, village promoters and distributors, relevant policy makers, agricultural and climate specialists and local officials.
Coordination with the programme
The evaluation team works with the implementing partner, taking care to maintain independence. Cooperation includes coordination of timings for activities, efforts to consult stakeholders, communities and local government officials, and sharing of logistical and field support where feasible. In addition, it has been agreed through discussions with the programme and DFID that both the evaluator and the implementing partner will:
- Communicate proactively to ensure effective joint planning and ongoing survey operations
- Use, where appropriate, the programme’s internal monitoring systems for the collection of data
- Respect the core evaluation questions agreed by DFID
- Respect the independence of the evaluation, which may require the evaluation team to produce its own instruments or collect data independently
- Respect the need for timely data collection and analysis to inform interventions
- Be aware of the potential for research fatigue among communities and respondents
- Recognise the implications of proposed changes on field and data management systems, including survey design, manuals and training materials, data entry templates, cleaning script and the code for generation of indicators and other relevant data points.
While data will be shared between the parties, analysis of the data will be conducted independently. The monitoring data used by the evaluation team is selected in tandem with the programme’s Monitoring and Results Measurement (MRM) team, which underwent a mock audit for compliance with the DCED Standard for results measurement in December 2013. A full DCED audit is planned once the programme has finalised MRM plans and strategy.
As the evaluation progresses, the evaluation team will feed back findings to the programme related to the theory of change to enable programmatic learning.
Challenges and lesson learned
During the design of the evaluation, the team faced several challenges from which they have learned the following lessons:
- Flexible evaluation design relies on regular communication about programme interventions and the timings of monitoring activities. This evaluation date collection complement monitoring data collection, for instance after significant market events or when interventions close.
- It is vital to have a detailed theory of change in place, owned by the programme and including consideration of underlying assumptions and contextual influences which may affect it. This may require the evaluator to work with the programme to develop the theory of change to a level of detail and rigour appropriate for the evaluation.
This evaluation was developed by Michelle Moffatt (WYG/Upper Quartile) in collaboration with BEAM Exchange. To learn more about this evaluation please email Michelle Moffatt.