Module 9. Designs, tools and techniques for monitoring

Data collection methods

Surveys

Surveys take a sample of opinions from a wider population, normally using a carefully structured questionnaire or a looser interview topic guide. They are useful for gauging how businesses, households, individual producers or wage labourers respond to a market systems intervention, and therefore for assessing an intervention’s effects. 

A survey based on a questionnaire typically includes five main stages:

• Preparation: The questionnaire is developed to provide answers for key research questions. There are many options for asking questions, including choosing from lists, ranking options, or responding to open-ended questions. Questionnaires take time to get right, and should therefore be piloted with a small group before being implemented for the whole sample. The preparation stage should also plan how data will be processed and analysed
• Sampling: The population to be surveyed is identified and defined. The population may also be stratified depending on the question and the need for disaggregated results (e.g. for a certain proportion of respondents to be men and women, or from particular age groups). Sampling methods and sample sizes are also defined at this stage
• Data collection: Data is then collected from individuals, households or enterprises. Options for data collection depend on the particular research questions, the size and characteristics of the population and the resources available. These include face-to-face questionnaires, mobile questionnaires, email questionnaires, or internet questionnaires
• Data analysis: Depending on the questionnaire, and the purpose of the survey, the method of data analysis can vary. Quantitative data is typically analysed with statistical methods to identify effects. Other data (obtained for instance in open-response questions) is usually analysed qualitatively. There are however some techniques and software (for example NVivo) which use quantitative methods to analyse qualitative data.
• Data use and reporting: after analysis, the data needs to be accessible or reported to the people who need the data for decision-making. These could be people within the programme, donor representatives, beneficiaries, or the wider public. The way the results are reported should be adapted to the intended audience.

One of the basic distinctions in surveys is between those that use closed and open-ended questionnaires. In a closed-response questionnaire respondents chose from the same pre-defined responses (i.e. a standard quantitative survey). 

• They are appropriate when researchers have a good idea of what they are looking for, and can be used to determine the scale of the effect
• They can be used in a number of different ways, including providing the basic tool for experimental survey methods (randomised control trials or quasi-experimental methods) which allow observed changes to be attributed to an intervention (see module 5 [link here to “Using a counterfactual” section])

In an open-ended questionnaire respondents are asked the same questions, but have the freedom to respond as they see fit. They are typically used during interviews (of which there are various kinds, as discussed next. 

Structured, semi-structured, unstructured and key informant interviews

Structured, unstructured and key informant interviews are typically used for qualitative surveys. They are particularly helpful in identifying a wider range of effects from an intervention, some of which the researcher may not know about in advance.

Market systems interventions will certainly produce some unexpected effects, and as a result programmes will rely heavily on interviews with beneficiaries, stakeholders and other market actors to understand what is happening. 

In a structured interview, all the questions (including the exact wording and order) are determined beforehand and the interviewer does not alter them. In effect this is a form of closed-response survey that is carried out in person, either face to face or over the telephone. Structured interviews are useful where a larger number of interviewers and interviewees are involved and where the data being collected needs to be uniform. 

In contrast, semi-structured interviews are useful for exploring issues of interest in more depth and typically include both closed and open-ended questions. As a data collection method it is also more personal and can be helpful for identifying individual reactions towards an intervention and how it is working. 

Unstructured interviews do not make use of strict interview guides, but work with an interviewer keeping the overall goal of the interview in mind while asking open-ended questions. This approach could be appropriate where front–line staff are in contact with market actors on a regular basis, and want to gauge their reactions to how an intervention is performing, or what is happening in the wider market.

Key informant interviews are semi- or unstructured interviews with people who have in-depth knowledge about the issue being studied. In a market systems programme these could be individuals in an enterprise or other market actor with detailed knowledge of an industry or sector, or other observers who understand the local political economy. Talking to key informants can be useful in particular:

• When qualitative and descriptive information is needed to help make a decision
• When it is important to gain an understanding of the perspectives, behaviour and motivations of customers and partners in order to explain shortcomings or success
• When generating recommendations is important
• To help interpret quantitative data obtained by other methods, or to decide which issues are relevant before designing a quantitative study.

Any key informant interviews will provide only a partial perspective on an issue. Findings should therefore be viewed as indicative, and triangulated with information from other sources. 

Focus group discussions

Focus groups are group interviews where information is gathered from participants with similar characteristics or interests. As a small number of participants are selected to take part, a focus group is not a random sample and is not a substitute for a large scale survey. It cannot, for instance, provide reliable conclusions about what a large population (e.g. customers for a particular product or service) think. A focus group can however be very useful to explore in detail issues identified by a broader survey, or for uncovering the early signs of change resulting from an intervention. 

Focus groups need to have a good facilitator who can identify key points, keep the discussion moving, and not allow the group to be dominated by one or two people. Steps for preparing, conducting and analysing data from a focus group include the following:

• The beneficiaries or the relevant group of people is identified. Usually, the size is quite small (up to ten people) and the people are carefully recruited
• The facilitators identify the themes and ideas that are to be explored in the group discussion. Some questions and pointers are also identified at this point
• The focus group takes place, with the discussion recorded in text, audio or video formats
• The discussion is then analysed and key points noted.

Case studies

Case studies can be used to examine a specific situation in an intervention and determine the different factors that influenced the outcome. They typically use a mix of qualitative and quantitative information. Case studies may be valuable to allow deeper investigation of specific issues identified in other monitoring activities. They can also illustrate statistical findings for audiences that respond better to narrative explanations of results, and to share learning within a programme or its partners.

Different types and uses of case studies include the following:

• Illustrative: these are descriptive in nature and provide in-depth examples to explain other information such as quantitative data
• Exploratory: these case studies are also descriptive but provide information to help develop hypotheses for future investigation
• Critical instance: these examine a single instance of unique interest, or serve as a critical test of an assertion about a programme
• Programme implementation: these explore the implementation, reception and success of programmes, often at several sites
• Cumulative: data from numerous case studies is used to answer or monitoring or evaluation question.

Recording observations - field diaries and activity logs

Front line programme staff are in continuous dialogue with market actors and hear their everyday stories, and are therefore well placed to observe changes on the ground. By helping to develop a narrative about what is happening in the field, such observations can provide very useful information for monitoring purposes, and support ongoing efforts by the programme to learn and adapt where necessary. 

Field diaries can be used to maintain detailed notes about events of interest. Writing lots of information down can be cumbersome, so programme teams might experiment with other methods, such as using the video or audio functions of smart phones to record observations.

Activity logs are similar to field diaries, but provide more concise records of particular events or situations in the form of brief comments or simple facts noted down when observing a situation. Maintaining a log in chronological order also allows a narrative of events to be established and analysed at a later date. The potential drawback is that the observer is unable to recollect exactly what happened at a later date, making it advisable to reflect on the information noted down relatively soon after collection.

An easy way to review staff observations, whether collected in a diary or log, is to hold regular reflection meetings where team members discuss these. This can typically be carried out as part of a regular team meeting or after-action review (Module 6). It is however important that observations, as collections of subjective views, are taken as indications of what is happening, and reviewed and triangulated against other sources of information.