Evidence methodology

The BEAM Evidence Map and database compiles and presents robust evidence on the impact and effectiveness of market system approaches.

The intention is to provide a broad overview of the existing evidence, rather than answer a particular research question.

The Evidence Map and database have two objectives:

• To facilitate informed judgement and evidence-based decision making in market systems approaches by providing user-friendly access to evidence.
• To guide effective use of research funds and enhance the potential for future primary research and evidence synthesis by identifying key 'gaps' in the available evidence, with the expectation that future research will fill these gaps thus increasing its relevance and value for money.

Audience

The evidence map and database primarily targets policy advisors, programme designers and decision-makers that are looking for evidence to assist in:

• advocating the use of market systems approaches in development, and
• the process of designing market systems programmes.

Definition of evidence

Evidence is defined in a broad sense as the findings from research. This includes the results of programmes using a market systems approach that are collected by projects, donors or other researchers using robust and transparent measurement and analysis practices.

Evidence documents differ from knowledge products/resources, which can be broadly defined as records of learning and experience. Evidence documents, therefore, are judged against higher quality standards than knowledge products/resources. 

Scope

The evidence map includes documents in English language only, published after 2000, and containing evidence of results from a wide variety of programmes and initiatives that used a market systems approach. The structure of the Evidence Map is based on the strategic results framework as described in the M4P Operational Guide and illustrated in the diagram below.

Inclusion criteria

For a resource to be classified as ‘evidence’ and included in the evidence database and on the evidence map, it must fulfil a number of inclusion and minimum quality criteria as defined in the table below. The document also needs to meet the BEAM Resource Library quality criteria.

Table 1: Inclusion criteria

Criteria	Description
Relevance: The document is aligned with the objective of the BEAM evidence base	The document contains evidence of results from programmes using a market systems approach. Documents may include evidence of results from programmes which are designed using a market systems approach only for one component of the programme. Some documents contain evidence of results from multiple programmes using a market systems approach. In particular, the document should illuminate the connection between market system interventions and the intended or unintended results. It is not essential for results to be measured by an independent party or against a counterfactual for the document to be included in the evidence base. The database does not include theoretical or conceptual studies which focus on the construction of new theories rather than generating or synthesising empirical data. The database also does not include knowledge products, such as guidance, think pieces, blogs, etc.
Currency: The document has been produced no earlier than 2000	The start date for evidence documents included in the database is 2000 because this is when the original framework document for making markets work better for the poor (M4P) was developed.
Accessibility: The document is publicly accessible or publication on the BEAM website has been approved by the owner of the copyright	All documents are published or publicly available. If not publicly available, BEAM Exchange must have the written consent of the organisation or programme/project to publish it in its evidence database.
Language: English language documents only	Only English documents are included in the evidence database at present as the BEAM Exchange team does not currently have the capacity to review and assess documents in other languages.
Secondary criteria	Description
Transparency: The document is transparent about the data collection and analysis methodology used to measure results	All documents included describe the methodology used to collect and analyse data, and the sample frame used to select data sources (including size and composition) to measure results. Documents based on secondary sources must all describe the methodology to select, assess and compile these sources. Programme documents which self-report results and have successfully passed a DCED audit are rated as partially achieving the criteria. The rationale is that if DCED audited, the programme has been certified as using good measurement techniques, even if the exact methodology is not shared in the document.
Credibility: The data collection methods generate a credible dataset, and analysis methods generate credible results.	All documents included describe a methodology that applies robust measurement and analysis practices that are generally accepted to represent best-fit for the study design to generate data and study results.
Cogency: The report presents a convincing argument	All documents included deliver a plausible, coherent and convincing argument (from design, through data collection, analysis to conclusions) to explain results achieved.

Management process to source and assess inclusion of evidence documents

The process to collate and assess evidence documents has been adapted from the stages of ODI's how to do a rigorous, evidence-focused literature review in international development and is informed by the 3ie evidence gap map approach. Some of the stages were modified to account for the fact that BEAM's goal is to fill an evidence database and provide a broad overview of the existing evidence, rather than answer a particular research question.

The BEAM team developed a 6-step management process which is detailed below. The management process was reviewed and updated in December 2016. The original evidence inclusion protocol was updated to include an evidence quality grading system (see below).

The evidence map population process is continued periodically throughout the lifetime of BEAM, and therefore after the initial map population, new evidence may be added from time to time. 

6-step process

Step 1: Setting objective and scope
BEAM set the objective of the map and database; to compile an evidence database that provides a broad overview of the existing evidence.

Step 2: Setting the inclusion criteria
BEAM set primary inclusion criteria to narrow down relevant documents for review, then a more detailed set of secondary criteria to assess the quality of documents in a consistent manner. 

Step 3: Developing a strategy for populating the evidence database
BEAM worked in two phases: initial population of the evidence map on the BEAM website (Phase 1); and secondary population of the evidence map (Phase 2). 

Phase 1
1. Internet search based on pre-defined search strings, both in relevant databases but also using popular search engines.
2. 'Eye-ball' elimination of some documents coming up as a result of the search string. This is used particularly when it is very clear a document does not meet the relevance criterion.

Following this initial population of the evidence map, additional evidence is sourced via connecting with BEAM’s network:

Phase 2
3. Crowd-sourcing using BEAM’s networks and community
4. Snowball searching for documents through key informants and contacts in implementing organisations

Step 4: Retrieval
The retrieval happens in two phases following the above-mentioned strategy: 

1. An initial first effort aimed to capture as many evidence documents as possible that are currently published. 
2. After the first effort, the database is periodically updated with newly published evidence documents by the BEAM team, and the community are asked to contribute new documents.

Step 5: Screening
During the screening phase, the collated evidence documents are assessed relative to the defined inclusion criteria.

There are two rounds in the screening process. The first screening is against the primary inclusion criteria, assessing titles and abstracts. All documents must pass all primary inclusion criteria to be included in the evidence base. The second screening is done by assessing the full text of the document, using the secondary inclusion criteria. 

Step 6: Evidence characterisation
At this stage, all evidence documents will be categorised according to the criteria defined in table 2. 

Table 2: Characterisation tags

Criteria	Description
1. Results level	This category will be used to locate the document on the evidence map. Possible tags: impact, systems qualities, systemic change, interventions
2. Type of document	This is an open category where tags will be developed as documents are added. (The number of tags should be as small as possible, i.e. the categories are broad.) Possible tags: project monitoring report, internal project review, donor review, external review, impact evaluation, case study
3. Method	Possible tags: experimental quasi-experimental, before/after, observational/qualitative
4. Data source	Possible tags: monitoring data, primary surveys, secondary data
5. Sector	This is an open category capturing the sector a particular evidence document reports about. This category will also be used to locate the document on the evidence map. Possible tags: agriculture, manufacturing, water supply, education
6. Intervention type	This is an open category that will specify the type of intervention the evidence document looks at. Possible tags: improvement of input supply, improvement of value chain coordination, improved marketing of products, improved product quality

Evidence quality 'grading'

In 2017, BEAM Exchange updated the original evidence inclusion protocol to include an additional evidence quality grading system, which assesses all evidence documents as either 'high confidence' (represented by orange 'bubbles') or 'low confidence' (represented by yellow 'bubbles'). All resources in the map meet the primary inclusion criteria. ‘Low Confidence’ resources partially meet the secondary criteria. 'High confidence' resources fully meet all the criteria. This grading system was introduced to align with best practice on evidence mapping. 

• Contribute your evidence to the map.