Using the difference in difference method for impact assessments
Tags: Attribution, quasi-experimental methods, regression analysis, media
This is one of a series of stories that complement the BEAM Monitoring Guidance. It offers a practical example of how a market development programme has solved a typical monitoring or evaluation challenge.
Katalyst share their experience applying the difference in difference method to measure the attribution of their media sector interventions on farmers.
Katalyst aims to increase the income of poor farmers in Bangladesh by improving the market system in which they operate. Media is one sector that Katalyst works in. For poor farmers, quality agricultural information is delivered through key information channels, such as television, radio, and newspapers. Katalyst's TV intervention provided training to staff working on the four most prominent TV channels in Bangladesh to improve the information offered to poor farmers.
The challenge of using the difference in difference method for impact assessments
Our attribution strategy for measuring the impact of the TV intervention was to use the difference in difference method (DiD). We first conducted a service provider level assessment of the TV channel programme staff to check whether they had incorporated elements from Katalyst's training into their agricultural programmes. Once this was validated, we conducted farmer level assessments, where we interviewed a sample of 240 farmers who had viewed the agricultural programmes. We compared their profits before they had applied the agricultural information gained from watching the TV programme and afterwards. These were our treatment farmers. We then did the same interview with a sample of 160 farmers who had not viewed the agro-programmes. These were our comparison group farmers. The comparison group farmers had similar characteristics to the treatment group farmers. We then compared the before and after profit difference of the treatment farmers and the comparison farmers to measure the attribution of the TV intervention.
There were, however, two major problems with this impact assessment:
- Since the TV programme reaches farmers across sectors ‒ including vegetable, fish, and maize ‒ a 240 sample size was not statistically significant to be representative of the income increase of the total diverse population of around 900,000 who were watching the agro-programmes.
- The DiD method did not decisively conclude that the income increase that the treatment farmers were enjoying was due only to the improved agricultural programmes of the Katalyst-intervened TV channels. There could have been other factors influencing the changes in profit compared to the previous year that the DiD method did not account for.
Our solution
To address the criticisms raised on the first media impact assessment, we devised a new attribution strategy and more appropriate sampling plan. This time, the assessment was for a similar intervention in the media sector: BTV (Bangladesh television). This intervention aimed to improve the agricultural programme to benefit poor target farmers, but this time through the state-owned TV channel of Bangladesh. This intervention trained the agricultural news and programme staff of BTV. Like the previous attempt, it involved a service provider level assessment, followed by a farmer level assessment. However, there were new elements added to the methodology which addressed the criticisms raised on the previous assessment.
Before the service provider level assessment of the BTV programme staff, we hired a local media expert to conduct a qualitative content analysis to compare old episodes of the BTV agricultural programme (before the Katalyst intervention) with new episodes of the agro-programme that were aired after the intervention. We then designed the service provider level questionnaire where, in addition to other questions, we asked BTV staff if they had introduced the particular changes into their agro-programme that were raised in the content analysis. We also probed them specifically about whether these changes were due to the Katalyst training.
We then conducted a farmer level assessment, but this time with a bigger sample of 600 treatment and 400 comparison group farmers. For both the treatment and comparison groups, we conducted the interviews only on farmers who had introduced an innovation in their farming practice in the last year and asked them the source of this innovation. Besides the innovation-related questions, we also asked both the farmer groups various other questions regarding their farming practices. We also collected the usual before and after profit data to conduct a cost-benefit analysis.
We have just completed the data collection and are in the process of conducting the data analysis of the second assessment. However, this time, the data analysis will involve regression analysis in order to estimate the effect of various external factors ‒ such as characteristics of farmers, their sources of innovation etc. ‒ on the increase in farmers' income. Regression analysis will take into account most of the possible factors that can affect the income increase of both treatment and comparison group farmers. Therefore, by ruling out all the other possible factors, we hope to be able to determine whether any income increase can be solely attributed to Katalyst's BTV intervention.
Do you have anything to add or ask a question? Please comment below or contact the author.
To learn more, see the BEAM Monitoring Guidance on attributing results to programme interventions.