Constructing a comparison group in agri-finance
Tags: Attribution, self-selection bias, microfinance
This is one of a series of stories that complement the BEAM Monitoring Guidance. It offers a practical example of how a market development programme has solved a typical monitoring or evaluation challenge.
The Strengthening African Rural Smallholders (STARS) programme is run by ICCO Cooperation and ICCO Terrafina Microfinance, in partnership with The MasterCard Foundation. The programme started in 2016 and works in Burkina Faso, Ethiopia, Rwanda and Senegal. The programme aims to facilitate increased access to financial markets and agricultural services for smallholder and subsistence farmers. STARS plans to partner with local microfinance institutions (MFIs) to assist them in providing smallholder and subsistence farmers with bundled packages of agri-loans and extension services.
The challenge
As of early 2016, the STARS programme is about to launch their agri-credit pilots, and have come up against the following challenge in designing their attribution strategy.
STARS partners will be offering a new form of agricultural credit, yet there will likely be a group of farmers that come in at the beginning (early adopters) followed by farmers coming in at a later stage (late adopters), or farmers not taking the new loan at all (risk-aversive subsistence farmers in more remote areas for example). Any group of farmers not using the new loan will vary in key characteristics to the treatment group (like risk appetite and entrepreneurial tendency), resulting in self-selection bias. The farmers that opt out, therefore, cannot be used as a comparison group. One alternative could be to select a comparison group from an area where the new services will not be offered, but a different location will have different crops, soil, climate, infrastructure, or access to markets. We will therefore not pursue this option.
A two-step solution
Step 1
During the pilot phase for the new credit products, STARS will select a convenience sample of a few groups of farmers, for example farmer associations, in an area where the new loan product will be introduced and to whom the product will be offered. All farmers will have access to the loan, but over time, STARS will use MFI client lists to split this group into three categories: people opting into the new loan at the beginning, people opting in later, and people consistently opting out. With a purposive selection of farmers from each category, STARS will conduct annual or bi-annual Focus Group Discussions (FGDs) to look into:
- reasons for opting in or opting out (barriers and catalysts)
- farmer characteristics
- product satisfaction
- unexpected benefits
- negative effects
- gender aspects.
Next to FGDs, we will also do a standard quantitative household survey of the entire farmer group (again annual or bi-annual). These surveys will focus on farmer demographics and household assets, access to services and programme impacts.
The results from the FGDs and surveys will give detailed insights into the reasons for opting in and opting out, as well as product satisfaction information which, from a marketing perspective, will be useful for STARS’ MFI partners. It will also yield insight into the characteristics of those opting in or out of the new loan (farmer profiles). And, importantly, it will yield insights into how impact differs between the three groups.
Step 2
After the credit pilot, during the regular product roll out phase, STARS will conduct pre and post-intervention surveys with a statistically representative sample of loan users from MFI client lists. These household surveys will focus on farmer demographics and characteristics, access to services and programme impacts, and, based on results from the FGDs, will include additional survey questions on key farmer characteristics and demographics that allow us to assign farmers to specific farmer profiles.
Using the information from step 1, the measured impacts in step 2 can be corrected for average differences among farmer types, including farmers opting out. This will allow us to circumvent the challenge of self-selection bias that would occur in a regular comparison group approach and still make claims regarding programme attribution.
Whilst this method can be challenged based on unknown internal and external validity, extrapolating from an in-depth pilot study makes a convincing case for net attributable effects. Results will additionally be triangulated with an external impact evaluation. But, more importantly, this approach will allow STARS to learn from its implementation, and help STARS partners to further tailor their credit products to different client types.
Do you have anything to add? Please contact the author.
To learn more, see the BEAM Monitoring Guidance on attributing results to programme interventions.