Market Development in Northern Ghana (MADE) is a DFID-funded programme that uses a market systems approach. This evaluation synopsis reviews the independent longitudinal evaluation of the Ghana MADE programme conducted by WYG and Upper Quartile using a theory-based approach.
Longitudinal evaluations are implemented alongside the actual programme they evaluate, covering longer time frames compared with traditional mid-term or ex-post evaluations. The Ghana MADE evaluation started in 2013 and will run until 2017. As of July 2015, WYG and Upper Quartile had completed the design of the evaluation and were about to begin implementation.
Background
Ghana suffers from economic disparity between its northern regions and the more industrialised south. The north is much poorer and its markets are ‘thin’ – characterised by few buyers and sellers, sparse transactions and volatile prices. In thin markets, information about the demand for products and services is hard to verify, and it is tricky for programmes to find businesses to work with.
Ghana MADE aims to help reduce regional disparities by improving the incomes and resilience of poor farmers and small-scale rural entrepreneurs in the Northern Savannah region. In order to do this, it aims to achieve substantial, transformative and lasting change in six agricultural value chains. A summary of the programme’s theory of change is set out in the image below. Crucially, because interventions address the functioning of the overall market system, as opposed to individual farm-level issues, it is hoped that benefits will extend beyond the programme's duration.
The objectives of the independent evaluation are to:
- Assess whether the M4P approach used in the context of ‘thin markets’, such as the Northern Savannah, can deliver benefits for poor producers and entrepreneurs
- Contribute to the evidence base about what works in Northern Ghana
- Inform international debate about the value of the M4P approach, and thereby feed into future policy and funding decisions related to tackling poverty reduction and achieving the Millennium Development Goals (MDGs)
- Inform DFID and the government of Ghana about sustainable approaches to developing market systems and which interventions have the greatest impact.
Methodology
The analytical framework used in this evaluation is based on contribution analysis combined with a mixed methods approach to data collection. The analysis thereby focuses on key elements of the programme theory, rather than tracking the whole programme logic. These key elements, or ‘keystone nodes’ as they are termed by the evaluation, are identified as being critical to the success of the intervention.
The nodes are selected from the results chains for each programme intervention following specific criteria. Certain nodes are common to a number of Ghana MADE target markets, and therefore offer opportunities for learning across sectors. The keystone nodes are the focus of the primary research efforts as they contain the most critical and learning-rich steps within each results chain. The keystone nodes will be tracked throughout MADE programme implementation.
Based on the tracking of the keystone nodes, the evaluation will synthesise findings from the individual market sectors at key evaluation moments to test and revise the overall programme-level theory of change. At this step, the contribution of the programme to observed outcomes will be assessed.
Data collection
Quantitative data will be collected through surveys conducted by the MADE programme’s own monitoring function. In addition, the evaluation team will conduct qualitative surveys, Focus Group Discussions (FGDs), Key Informant Interviews (KIIs) and case studies. This will provide data on:
- farmers’ and service providers’ experiences in relation to the keystone nodes
- the relevance, benefits, and unintended consequences of interventions at critical points of the theory of change (the keystone nodes)
- observed best practice
- the extent to which interventions have contributed to systemic and lasting change.
Coordination with the programme
In longitudinal evaluations, as opposed to a traditional evaluation approach, the evaluation team is working much closer with the programme team as programme and evaluation are implemented in parallel. This has implications for how the two teams communicate and how responsibilities are assigned between the programme and the evaluator. The specific arrangements for this evaluation are captured in a ‘principles of work agreement.’
To better understand this complex programme, WYG and Upper Quartile have been closely involved in the development of the programme’s theory of change, helping Ghana MADE feed useful findings from the evaluation back into the programme.
To ensure that the data collected by the monitoring team is robust, the evaluation team will review the monitoring plan and tools. An audit of the MADE monitoring system following the DCED Standard for Results Measurement will also be conducted. This is intended to increase confidence in the reliability and accuracy of MADE's monitoring data and research. The evaluation team will independently analyse the data.
Challenges and lessons learned
During the design of the evaluation, the evaluation team faced several challenges from which they have learnt the following lessons:
- When evaluating a large and complex programme, it is important to have clarity on evaluation objectives and focus on what specific aspects should be evaluated and what specific questions should be answered. Given the evolving nature of M4P programmes, this is even more important.
- The relationship between the evaluator and the programme should be guided by the evaluation objectives. In Ghana MADE's case, the focus is on learning rather than accountability so the evaluation team established a collaborative relationship, with the programme acting as a ‘critical friend’ and focusing on learning and feedback. This approach ties in with the adaptive nature of M4P programmes where evaluation findings shape ongoing programme design.
- The timing of data collection should be carefully considered and will depend on market dynamics, the timing of programme activities, and other factors such as seasonal variables. Given the adaptive nature of M4P programmes, it is important to plan what to do if the programme pulls out of activities or a whole market from which evaluators have already collected data. For the Ghana MADE evaluation, close-out assessments were planned should activities relating to the selected keystone nodes be closed down. This helps provide feedback on why they were not successful.
- In order to apply theory-based evaluation approaches, it is vital to have a good theory of change in place. This should be owned by the programme and include an appropriate degree of detail (including consideration of underlying assumptions and contextual influences). A theory of change workshop, facilitated by the evaluator at the start of the programme and attended by key programme staff, is useful for developing the theory of change and ensuring the evaluator and programme are working from the same page.
This evaluation was developed by Michelle Moffatt (WYG/Upper Quartile) in collaboration with BEAM Exchange. To learn more about this evaluation please email Michelle Moffat.