Tags: Attribution, self-selection, propensity score matching, baselines, quasi-experimental designs
This is one of a series of stories that complement the BEAM Monitoring Guidance. It offers a practical example of how a market development programme has solved a typical monitoring or evaluation challenge.
Growth and Employment in States – Wholesale and Retail Sector (GEMS4) is a market development project in Nigeria, funded by DFID.
The challenge
In 2014, GEMS4 conducted baseline studies for three pilot interventions in micro-retail, good handling practices and mobile money. Prior to starting the pilots, it was difficult to identify groups that would not benefit from the interventions and thus serve as a comparison group, because beneficiaries were self-selecting. For instance, it was difficult to determine in advance which retailer in a market location would take up the use of mobile money for payment transactions.
Initial response to the challenge
To address this, GEMS4 adopted an ex-ante quasi-experimental approach: propensity score matching. This required having large sample sizes to allow for retrospective identification of who in the market had been exposed to the interventions and who had not; identifying key observable characteristics to explain uptake of interventions; and matching beneficiaries to similar respondents who had been exposed to, but did not take up the intervention.
However, because of the adaptive nature of M4P approaches, the pilot interventions evolved and there were subsequent changes to the pilot locations, partners and beneficiaries ‒ rendering the data collected for propensity score matching largely obsolete.
Solution
In response to the experience with the initial baselines and considering the increase in the number of initiatives being piloted by the programme, GEMS4 replaced the initial propensity score matching method. The programme set out to achieve the most efficient design that would represent the best value for money. Consequently, the approach changed as follows:
- Timely baselines: GEMS4 carries out baselines as late as possible, rather than as soon as possible, but always before changes have taken place. This ensures that the beneficiaries of the interventions have been properly identified, and impact on their outcomes and incomes can be properly assessed.
- Counterfactual: Since it is difficult to know at the outset who would be participating in some interventions, the programme resort to using an ex-post comparison group design for the attribution strategy. After the intervention, when they know who has participated in the intervention, they conduct retrospective baselines for this self-selected treatment group. This survey also identifies key characteristics of the treatment group that enables construction of a comparison group. However, they do not match each beneficiary in the treatment group using statistical probabilities (as you would with propensity score matching). Rather, they identify a group of actors for comparison that are similar, based on key characteristics – e.g. for a farming cluster: crops grown, varieties grown, markets they sell to.
- Portfolio approach: The programme is working on eight initiatives, and this number may increase or decrease depending on the success/failure of pilots and addition of new initiatives. Therefore, the programme is adopting a portfolio approach whereby only selected interventions (for instance, interventions with the largest projected impact) are evaluated using rigorous impact assessments with large sample sizes, while others will be assessed using smaller sample sizes. However, for all interventions, holistic results measurement frameworks – results chains, indicators, projections and measurement plans – are put in place to monitor and measure results.
To learn more, see the BEAM Monitoring Guidance on attributing results to programme interventions.