Poorest States Inclusive Growth Programme (PSIG) is a £65 million programme funded by DFID. Oxford Policy Management (OPM), in partnership with EDA Rural Systems, is conducting a multi-year mixed-method evaluation of the financial inclusion component of PSIG. The evaluation design uses a theory-based approach incorporating composite methodologies to test outcomes and impacts at the household, institution and sector level.
Background
This evaluation will help improve understanding of the impact of DFID's financial sector deepening initiatives in India. The evaluation will run from 2014-18, alongside the actual programme, which started in 2012 and will run until 2018. As of September 2015, the evaluation is under implementation with the inception design phase and the first round of data collection is complete.
Access to financial services is seen as a critical factor in economic development, but there are a number of issues to be resolved, including the role of the microfinance sector and the potential for banks to reach low income customers, with mobile technology and through local agents.
Microfinance institutions provide a viable option but there are challenges in depth of outreach (serving poorer people, operating in poorer areas); in product offering (moving beyond a standard credit product); and in ensuring client protection and responsible practices particularly in serving customers who are poor, low income, and often women.
The PSIG programme aims to enhance the income and employment opportunities of poor women and men by enabling them to participate and benefit from India’s wider economic growth. It seeks to improve the incomes, and reduce the vulnerability, of poor people and small producers, by expanding their access to finance and markets. PSIG has two separate, interlinked components: financial inclusion and impact investing. The described evaluation is only looking at the financial inclusion component. Under this, DFID is providing up to £30 million to support financial inclusion and women’s empowerment across the four states of Bihar, Madhya Pradesh, Orissa and Uttar Pradesh. It aims to widen the scope of available financial services beyond credit to include a full range of accessible financial services that support the needs of the poorest. It will do this by working with stakeholders at a number of levels:
- At the sector level, working with thinktanks and microfinance associations to advocate for policy change
- At the institutional level, providing technical support and capacity building to banks, microfinance institutions and other intermediary organisations to support the development of responsible, sustainable, pro-poor financial services
- At the client level, facilitating the delivery of financial services, financial literacy training and new products, with a focus on gender empowerment.
Methodology
OPM and EDA have designed and are implementing a multi-year impact evaluation based around the programme’s theory of change, incorporating indicators for outcomes and impacts at the client (household), institutional and sector levels. For each level, the evaluation design includes a composite methodology that uses both quantitative and qualitative tools.
At the institutional and sector levels, the team will use a combination of primary and secondary research approaches – including a literature review, process analysis and interviews with selected institutions and government stakeholders – to identify programme outcomes and impacts. An in-depth analysis of a sample of the financial institutions supported by the programme will be carried out midline. This will look at the particular activities supported by PSIG, the ability to extend product design for diversified products that are amenable to the needs of the poor, and changes incorporated with regards to responsible finance. A specific focus of the evaluation will be to look at the introduction of responsible finance mechanisms and whether these are fit for purpose, and having a tangible impact. This will be done through tracking not only official policies, but also looking at the branch level realities of these organisations.
At the client level, the design includes a longitudinal panel survey of nearly 4,000 households. Qualitative methods including key informant interviews and focus group discussions were used to finalise the indicators of interest for the quantitative baseline, and will continue to be used in the triangulation of results. The quantitative information provides a picture of the client base at the initial stages of the evaluation, and will be compared to findings from an end-line survey to be carried out in 2017-18. Sophisticated statistical analysis will be used to determine the effect of the programme. This includes pre/post comparisons, comparing across different models of financial institutions, different financial products and exposure variation.
Main activities, including the evaluation, started in 2013-14. The baseline was undertaken in 2015; the concluding survey will be taken in 2018. The midline research in 2016 will focus on qualitative themes.
Data collection
The evaluation draws on secondary sources that provide context, as well as programme documents, financial service provider data and reports and primary research interviews.
The first round of quantitative household data collection, covering nearly 4,000 households has been completed. The sample was drawn on the basis of clients of twelve purposively selected financial institutions operating in four selected districts within each state. The households were then randomly selected from client lists within these districts.
The household level questionnaire collected extensive primary data on household socio-economic characteristics like size, asset ownership, consumption and investment; livelihood characteristics like employment profile, and occupation, and usage and availability of basic services; as well as more nuanced empowerment indicators such as participation in decision making. It has extensive modules focusing on financial services including credit, savings, insurance and pensions. The first round provided a baseline of indicators which will be tracked over the programme duration to help measure its effect.
Coordination with the programme
For the inception report there was very close coordination between the evaluation team and the programme team with detailed presentations and feedback on the evaluation questions and the proposed methodology. The programme team appointed a technical advisory committee (TAC) with evaluation specialists from DIFD, along with microfinance stakeholder experts from India (who are also members of the national thinktank set up for the programme). The evaluation team has regular interactions with both DFID India, which is the primary funder, and Small Industries Development Bank of India, which is responsible for programme implementation.
The programme team was directly involved in sampling decisions (particularly the selection of 12 sample FSPs – out of 28 total PSIG partners). The TAC was also involved in the review of household/community level tools. Since the start of the baseline field work, the team has linked in every few months to check progress.
Regular engagement has involved feeding into the design of the evaluation and tools, feedback on reports and timing of activities. This has also allowed for regular sharing of data from PSIG's internal management information system which feeds into the findings of the evaluation. Findings from the baseline report and interactions between the implementing team and the evaluation team, are expected to feed into the programme's continuing implementation strategy.
Challenges and lessons
At this baseline stage, the main challenges so far have been:
- Managing the expectations of the programme team
- Keeping track of the various reports generated at different levels of the programme
- Following up with FSPs for organisational and client data, and the quality of data provided
- Designing the household questionnaire to capture the use of diverse financial services (from programme FSPs and other FSPs)
- Working in a fast changing and dynamic policy environment.
The programme team wanted an impact assessment that was 'rigorous and uses internationally recognised methods.' It was important to establish a credible counterfactual. At the same time, the options for effective randomisation were limited due to the programme’s limited ability to influence FSP outreach decisions. Beneficiaries were self-selecting into services and could not be actively allocated to treatment or control groups. At the same time, the evaluation design was subject to resource limits, which most importantly limited field sample size and with it statistical power necessary for disaggregation.
The sample design had to be pragmatic, within the given resources. It allowed for different treatments (in terms of model and products) and different contexts. We dropped the idea of a 'pure' counterfactual (someone with no access to any financial services.) During the pilot it was not possible to identify this type of control – since those without financial services were likely to be significantly different and above or below the criteria for access. As part of the impact analysis, we will include questions on self-perception of change and reasons for change at end-line.
At different levels of the programme there are many reports that get generated at different times, with no specific calendar or coordination. We have to identify the different people responsible and follow up regularly to keep up to date, and request programme documents as they appear.
FSP data is critical to analyse baseline parameters and to provide the sample frame for the household data. Broad FSP data is maintained by the programme on a quarterly basis, and an MIS is likely to be put in place for more detailed data capture. Until that time, obtaining FSP data requires direct follow up. FSPs are mostly very cooperative, when staff have the time. The evaluation team provides forms that match the FSP’s management information systems, which are mostly computer based and allow data to be extracted quite easily. For smaller institutions, information quality is a challenge.
Working with financial institutions that focus on being commercially viable gives the evaluation team a more limited scope to influence operational and implementation details. This has meant a need for close communication and collaboration at the institution level, as well as relationship building, to ensure a feeling of ownership and interest in the evaluation.
At the household level, people now have access to a range of different financial service providers – both partners of PSIG, and other non-partners. In India, even in the northern states, clients may borrow from 2 or 3 MFIs. And are beginning to use different options for savings – through banking agents, or through MFIs as banking correspondents. For effective data collection, some parts of the questionnaire are quite complex, in asking about different sources and uses of credit, alongside the use of other kinds of financial services.
Up to the end of the programme, we anticipate additional challenges, in particular those related to analysing how different pieces of the programme link together to contribute to change – both at the sector level (policy and regulation) and FSP level (pro-poor outreach, responsible and sustainable practices) – given the dynamic environment and number of players involved.
The financial inclusion landscape is rapidly changing and evolving in India. For example, the availability of financial services grew exponentially in the period between inception and baseline with the introduction of the Pradhan Mantri Jan Dhan Yojana (PMJDY), which claims to bring universal access to finance in India. The introduction of this scheme resulted in the opening of over 180 million bank accounts within the first few months of implementation. Changes in the broader policy context and in turn, the programme, have meant that the evaluation has been designed to be flexible and needs to communicate closely and regularly with the implementation team.
This synopsis has been developed by Vinayak Uppal, Oxford Policy Management and Frances Sinha, EDA Rural Systems, in collaboration with BEAM Exchange.