INOVAGRO II is a SDC-funded programme that uses a market systems approach. This evaluation synopses reviews the independent longitudinal evaluation of INOVAGRO II.
The programme focuses on the functioning of market systems and on the capacity and technology use of farmers with the main aim of decreasing rural poverty. The programme is funded by the Swiss Agency for Development and Cooperation (SDC) and implemented in Northern Mozambique. SDC engaged the International Food Policy Research Institute (IFPRI) to conduct a rigorous and unbiased impact evaluation of selected aspects of the project. The impact evaluation focuses on three value chains ‒ soya beans, pigeon peas and maize ‒ and will include two districts in Northern Mozambique where INOVAGRO II had yet to establish a presence.
The evaluation is longitudinal and is being rolled out alongside the intervention. Quantitative data is collected at two stages: at the baseline survey in 2015, before the start of INOVAGRO II activities, and at the endline survey in 2017, after approximately 24 months of the project. The analysis for the impact evaluation is carried out after completion of this endline survey. As of January 2016, the baseline data had been collected and the implementation of the programme was well under way.
Background
Despite a decade of sustained growth in Mozambique, triggered mainly by the extractive industries, growth has failed to translate into significant poverty reduction. In particular, small-scale farmers have not benefited, due to key constraints in agriculture (e.g. access to finance, inputs, know-how) and the lack of business linkages.
Building on the experiences of the first phase of INOVAGRO, INOVAGRO II seeks to target 15,000 small-scale farmers and expand activities to new regions in Northern Mozambique ‒ including the two districts focused on in the impact evaluation. INOVAGRO II aims to reduce poverty by improving linkages between markets and small-scale farmers. By empowering farmers to better engage with the private sector and improve farming practices, the programme hopes to sustain pro-poor growth that can outlive the intervention. The theory of change of INOVAGRO II is presented in the diagram below.
Coordination with the programme
An implementing agency with its on-the-ground knowledge and know-how is an indispensable contributor to the understanding of the channels of impact that the programme may have. The evaluation team, in turn, has the technical expertise needed to carry out a scientifically sound evaluation that captures any potential programme impacts. Collaboration and coordination between the two is necessary to carry out an impact evaluation of this nature. The extent and terms of this collaboration was formally agreed on through a mutually agreed framework. This agreement stated that the evaluator and the implementing partner must:
- Communicate methodologies, practices and scope of any intervention
- Share relevant information and data
- Coordinate to ensure that the selected target areas and non-target areas are credibly comparable
- Consistently coordinate on the design elements necessary for the research to proceed
- Communicate in a timely manner regarding any changes made to implementation focus or design
- Coordinate primary data collection and intervention efforts as well as communication with stakeholders.
The main objectives of the evaluation are to:
- Contribute to the body of knowledge of the M4P approach
- Assess whether and how market system development benefits the rural poor in Mozambique and how this could potentially influence the design of future agricultural policies
- Determine the impact of INOVAGRO II on key outcome objectives of the M4P approach, in the study’s empirical context.
Methodology and data collection
The evaluation uses a mixed methods approach for data collection and analysis. It focuses on two districts in Northern Mozambique where INOVAGRO II had not worked before. The sample consists of a total of 8 treatment communities and 8 control communities (4 of each group in each district). The study communities were selected by the evaluation team after consultation with the implementing agency. Quantitative data is collected at baseline and endline through a comprehensive household level survey. Approximately 2,000 households from treatment and control areas are included in the survey. The same households are interviewed at both occasions to be able to trace the development of individual farmers. In addition to the quantitative data collection, complementary qualitative data has been obtained by interviewing stakeholders, and another round of such interviews will take place in the second half of 2016, i.e. between baseline and endline surveys, with respondents such as agro-dealers and other private sector actors that have partnered with the implementation team.
The quantitative analysis follows a quasi-experimental approach including a difference-in-difference analysis. This approach compares the average development over time of treatment communities to the average development over time of the control communities. By mimicking an experimental research design, the difference-in-differences method is arguably one of the most robust quasi-experimental models for evaluations with a non-random intervention design.
As INOVAGRO employs a flexible approach to implementation, the evaluation needs to deal with the potential differences between the intervention design and INOVAGRO’s actual implementation. In order to do so, intention-to-treat data is used as an instrument for treatment in the analysis. Here, 'intention-to-treat' refers to the treatment assignment: communities and farm households that were targeted to be part of INOVAGRO II. This may differ from those that ended up actually being treated, for a range of reasons including willingness of farmers to participate, changes in some specifics of programme implementation by the programme staff, etc. The instrumental variable technique is used here to compensate for the fact that there is a difference between the people INOVAGRO intended to treat and the people actually treated.
As to the purpose of contributing to the knowledge of the M4P approach, the questionnaire design places heavy focus on farmers’ participation in market systems and access to market information as well as productivity and farmer practice.
Based on the challenges faced thus far, the evaluation team has learned the following lessons:
- Collaboration between the implementing agency and evaluation team is crucial to understanding the theory of change and designing data collection and analytical tools. It is important to establish clear and formal agreements between the two teams to avoid misunderstandings regarding objectives and division of labour, particularly if the implementation partner has relatively little experience with impact evaluation research
- It is crucial to come to a clear understanding and agreement between the two teams about the implications of the empirical methodology for aspects of the intervention roll-out. As important is the need for the implementing agency to have incentives and a stake in adhering to the agreed-upon intervention design, without which it might be difficult to commit to the design.. This in turn can adversely affect the empirical evaluation method
- Timing of data collection and other field work must be well coordinated with the programme
- Continuous communication between the implementing agency and the evaluation team during the entire process of the impact evaluation is vital. Even after the baseline data is collected, communication is needed to be able to capture any potential changes made to the implementation schedule.
The instrumental variable technique is employed when there is a concern that the explanatory variable of interest ‒ here, treatment by INOVAGRO ‒ is endogenous. For example, those that are treated (as opposed to those originally planned to be targeted) may end up being the better-off, more productive, larger farmers. In this case, endogeneity may bias results upwards. There are similar scenarios in which the effects may be biased downwards. The use of instrumental variables (IV) involves using a variable as an instrument for the potentially endogenous explanatory variable, such that the IV is expected to be correlated with the explanatory variable, but is not expected to affect the outcome of interest (such as for example market participation). In this case, one can expect treatment assignment to be correlated with treatment, but not to have an independent effect on outcomes.
The authors of this evaluation are Tewodaj Mogues, Alvina Erman, and Hosaena Ghebru (IFPRI). For more information about the programme or evaluation please contact Tewodaj Mogues.